|
Zhen Xu, Sergio Escalera, Adrien Pavao, Magali Richard, Wei-Wei Tu, Quanming Yao, et al. (2022). Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform. PATTERNS - Patterns, 3(7), 100543.
Abstract: Obtaining a standardized benchmark of computational methods is a major issue in data-science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here, we introduce Codabench, a meta-benchmark platform that is open sourced and community driven for benchmarking algorithms or software agents versus datasets or tasks. A public instance of Codabench is open to everyone free of charge and allows benchmark organizers to fairly compare submissions under the same setting (software, hardware, data, algorithms), with custom protocols and data formats. Codabench has unique features facilitating easy organization of flexible and reproducible benchmarks, such as the possibility of reusing templates of benchmarks and supplying compute resources on demand. Codabench has been used internally and externally on various applications, receiving more than 130 users and 2,500 submissions. As illustrative use cases, we introduce four diverse benchmarks covering graph machine learning, cancer heterogeneity, clinical diagnosis, and reinforcement learning.
Keywords: Machine learning; data science; benchmark platform; reproducibility; competitions
|
|
|
Jose Antonio Rodriguez, Florent Perronnin, Gemma Sanchez, & Josep Llados. (2008). Unsupervised writer style adaptation for handwritten word spotting. In Pattern Recognition. 19th International Conference on, IBM Best Student Paper Award..
|
|
|
David Masip, & Jordi Vitria. (2006). Boosted discriminant projections for nearest neighbor classification. Pattern Recognition, 39(2): 164–170.
|
|
|
Yong Xu, Jing-Yu Yang, & Zhong Jin. (2004). A novel method for Fisher discriminant analysis. Pattern Recognition, 37(2):381–384 (IF: 2.176).
|
|
|
Jian Yang, Zhong Jin, Jing-Yu Yang, David Zhang, & Alejandro F. Frangi. (2004). Essence of kernel Fisher discriminant: KPCA plus LDA. Pattern Recognition, 37(10): 2097–2100 (IF: 2.176).
|
|
|
I. King, & Zhong Jin. (2003). Integrated Probability Function and Its Application to Content-Based Image Retrieval By Relevance Feedback. Pattern Recognition, 36(9): 2177–2186 (IF: 1.611).
|
|
|
X. Jing, David Zhang, & Zhong Jin. (2003). Improvements on the uncorrelated optimal discriminant vectors. Pattern Recognition, 36(8): 1921–1923 (IF: 1.611).
|
|
|
M. Bressan, David Guillamet, & Jordi Vitria. (2003). Using an ICA Representation of Local Color Histograms for Object Recognition. Pattern Recognition, 36(3):691–701 (IF: 1.611).
|
|
|
Yong Xu, Jing-Yu Yang, & Zhong Jin. (2003). Theory analysis on FSLDA and ULDA. Pattern Recognition, 36(12): 3031–3033 (IF: 1.611).
|
|
|
X. Jing, David Zhang, & Zhong Jin. (2003). Improved algorithm and generalized theory. Pattern Recognition, 36(11): 2593–2602 (IF: 1.611).
|
|
|
Jordi Vitria, & J. Llacer. (1996). Reconstructing 3D light microscopic images using the EM algorithm. Pattern Recognition Letters, 1491–1498.
|
|
|
A. Pujol, Jordi Vitria, Felipe Lumbreras, & Juan J. Villanueva. (2001). Topological principal component analysis for face encoding and recognition. PRL - Pattern Recognition Letters, 22(6-7), 769–776.
|
|
|
Gemma Sanchez, Josep Llados, & K. Tombre. (2002). A mean string algorithm to compute the average among a set of 2D shapes. PRL - Pattern Recognition Letters, 23(1-3), 203–214.
|
|
|
A. Martinez, & Jordi Vitria. (2000). Learning mixture models using a genetic version of the EM algorithm. PRL - Pattern Recognition Letters, 21(8), 759–769.
|
|
|
M. Bressan, & Jordi Vitria. (2003). Nonparametric Discriminant Analysis and Nearest Neighbor Classification. PRL - Pattern Recognition Letters, 24(15), 2743–2749.
|
|