TY  - JOUR
AU  - Zhengying Liu
AU  - Adrien Pavao
AU  - Zhen Xu
AU  - Sergio Escalera
AU  - Fabio Ferreira
AU  - Isabelle Guyon
AU  - Sirui Hong
AU  - Frank Hutter
AU  - Rongrong Ji
AU  - Julio C. S. Jacques Junior
AU  - Ge Li
AU  - Marius Lindauer
AU  - Zhipeng Luo
AU  - Meysam Madadi
AU  - Thomas Nierhoff
AU  - Kangning Niu
AU  - Chunguang Pan
AU  - Danny Stoll
AU  - Sebastien Treguer
AU  - Jin Wang
AU  - Peng Wang
AU  - Chenglin Wu
AU  - Youcheng Xiong
AU  - Arber Zela
AU  - Yang Zhang
PY  - 2021//
TI  - Winning Solutions and Post-Challenge Analyses of the ChaLearn AutoDL Challenge 2019
T2  - TPAMI
JO  - IEEE Transactions on Pattern Analysis and Machine Intelligence
SP  - 3108
EP  - 3125
VL  - 43
IS  - 9
N2  - This paper reports the results and post-challenge analyses of ChaLearn's AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a “meta-learner”, “data ingestor”, “model selector”, “model/learner”, and “evaluator”. This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark (http://autodl.chalearn.org), the open-sourced code of the winners, and a free “AutoDL self-service.”
UR  - https://ieeexplore.ieee.org/abstract/document/9415128
UR  - http://dx.doi.org/10.1109/TPAMI.2021.3075372
N1  - HUPBA; no proj
ID  - Zhengying Liu2021
ER  -