Following the success of the AutoDL 2019-2020 challenge series, including an official competition of NeurIPS'19, and the first Meta-Learning challenge at AAAI'21, we organized MetaDL, a few-shot-learning competition challenging DL methods for meta-learning at NeurIPS'21 on December 8th, 2021.

Schedule:

• 11:25 - 11:45: MetaDL: A few-shot learning competition (pre-recorded on NeurIPS platform, by Adrian El Baz)

• 12:00 - 12:30: Keynote: Frank Hutter, University of Freiburg (live, Zoom room)

DL 2.0: How Meta-Learning May Power the Next Generation of Deep Learning

Deep Learning (DL) has been incredibly successful, due to its ability to automatically acquire useful representations from raw data by a joint optimization process of all layers. However, current DL practice still requires substantial manual efforts to define the right neural architecture and training hyperparameters to optimally learn these representations for the data at hand. The next logical step is to jointly optimize these components as well, based on a meta-level of learning and optimization. In this talk, I will discuss several advances towards this goal, focusing on (1) joint optimization of several meta-choices in the DL pipeline, (2) efficiency of this meta-optimization, and (3) optimization of uncertainty estimates and robustness to data shift.

• 12:30 - 13:00: Reveal of the MetaDL winners and datasets (live, Zoom room)

• 13:00 - 13:30: Invited talk by Team Meta Delta, Tsinghua University (live, Zoom room)

MetaDelta++: Improve Generalization of Few-shot System Through Multi-Scale Pretrained Models and Improved Training Strategies

Meta-learning aims at learning quickly on novel tasks with limited data by transferring generic experience learned from previous tasks. Naturally, few-shot learning has been one of the most popular applications for meta-learning. Recently, an ensembled few-shot system MetaDelta is proposed to boost the performance, which won first place in the AAAI 2021 MetaDL challenge with leading performance. However, the generalization ability of MetaDelta is still limited by the homogeneous model setting and weak pretraining and fine-tuning strategies, hindering MetaDelta from being applied to more diverse scenarios and problems. We further boost the performance and generalization ability of MetaDelta by leveraging pre-trained models at multi-scale and improved training strategies, including semi-weakly supervised pretraining, data augmentation, separated learning rate at each layer, lazier BN statistics update, and better decoder design. Our system MetaDelta++ substantially boosts the performance and generalization abilities by a large margin and stands the 1st place in phase 1 of the NeurIPS 2021 MetaDL system with a large margin compared to MetaDelta and other teams.

• 13:30 - 14:30: Tutorial on Meta-learning, by Isabelle Guyon, Zhengying Liu, Felix Mohr and Jan N. van RIjn (live, Zoom room)

In this slot, we will reflect on some latest developments in Meta-learning. We will present several frameworks that capture the relation between various research directions in meta-learning and AutoML. More specifically, we will reflect on the role of meta-learning in the broader context of machine learning, and on the role of learning curves in AutoML.

• 14:30 - 14:45: Wrap up session, discussion on follow-up competition (live, Zoom room)

Shared resources:

• Meta Padawan Video

New Benchmark for Meta-Learning

We have worked hard to come up with several new meta-learning 128x128 image datasets. We feel that the meta-learning community can greatly benefit from a new benchmark, and plan to publish these datasets open source after the challenge. It is our goal to establish this as a new benchmark for the meta-learning community. Participating in this challenge will give you a head start at getting to work with these datasets.

The challenge is with code submission and will be run on the Codalab platform with generous donations of cloud units from Microsoft and Google. The challenge winners will receive prizes donated by ChaLearn, if they agree to open-source their code. However, there is no such requirement to enter the challenge. The top ranking participants will be invited to co-author a paper on the challenge results, planned to be published in the PMLR, the proceedings track of the Journal of Machine Learning Research.

The competition has a focus on fast solutions. Per submission, the user will get 2 hours of calculations on each meta-dataset. Presumably, this competition favors first-order solutions (such as FO-MAML, LSTM meta-learner) over second-order solutions (such as MAML, TURTLE).

About Meta Learning

For a comprehensive overview of Meta-learning, we refer to the following resources:

• Metalearning: Applications to Automated Machine Learning and Data Mining (2nd edition, Open Access, by Brazdil et al. (2022))

• Meta-learning: A survey (Book chapter, by Vanschoren (2019))

• Meta-learning in neural networks: A survey (Arxiv, Hospedales et al. (2020))

• A Survey of Deep Meta-Learning (Artificial Intelligence Review, Huisman et al. (2021))

Organization

This challenge would not have been possible without the help of many people.

• Adrian El Baz (Université Paris-Saclay, France)

• Isabelle Guyon (Université Paris-Saclay; INRIA, France and ChaLearn, USA)

• Jennifer He (4Paradigm, China)

• Mike Huisman (Leiden University, the Netherlands)

• Zhengying Liu (Université Paris-Saclay, France)

• Yui Man Lui (University of Central Missouri)

• Felix Mohr (Universidad de la Sabana)

• Jan N. van Rijn (Leiden University, Netherlands)

• Sebastien Treguer (Université Paris-Saclay, France)

• Wei-Wei Tu (4Paradigm, China)

• Jun Wan (Chinese Academy of Sciences, China)

• Lisheng Sun (Université Paris-Saclay, France)

• Haozhe Sun (Université Paris-Saclay; LISN, France)

• Phan Anh Vu (Université Paris-Saclay; LISN, France)

• Ihsan Ullah (Université Paris-Saclay; LISN, France)

• Joaquin Vanschoren (Eindhoven University, the Netherlands)

• Jun Wan (Chinese Academy of Science, China)

• Benjia Zhou (Chinese Academy of Sciences, China)

The challenge is running on the Codalab platform, administered by Université Paris-Saclay and maintained by CKCollab LLC, with primary developers:

• Tyler Thomas (CKCollab, USA)

• Loic Sarrazin (Artelys, France)

• Anne-Catherine Letournel (UPSaclay, France)

• Adrien Pavao (UPSaclay, France)

ChaLearn is the challenge organization coordinator. Microsoft and Google are the primary sponsors of the challenge. 4Paradigm donated prizes, datasets, and contributed to the protocol, baselines methods and beta-testing. Other institutions of the co-organizers provided in-kind contributions, including datasets, data formatting, baseline methods, and beta-testing.

Contact the organizers.