MetaLearn Challenges


MetaDL @ NeurIPS'21

We are grateful to Microsoft and Google for generous cloud unit donations and to 4Paradigm for donating prizes. This project is also supported by ChaLearn and HUMANIA chair of AI grant ANR-19-CHIA-00222 of the Agence Nationale de la Recherche (France). Researchers and students from Université Paris-Saclay, Universiteit Leiden, TU Eindhoven, and 4Paradigm have contributed. The challenge is hosted by Codalab (Université Paris-Saclay).

Following the success of the AutoDL 2019-2020 challenge series, including an official competition of NeurIPS'19, and the first Meta-Learning challenge at AAAI'21, we are organizing MetaDL, a few-shot-learning competition challenging DL methods for meta-learning at NeurIPS'21.

2nd MetaDL Competition Workshop at NeurIPS

In coordination with NeurIPS 2021, we will host the 2nd MetaDL Competition Workshop on December 8th. Times are in GMX (London timezone). Exact Zoom link will be shared. Tentative schedule.

  • 11:15 - 11:45: MetaDL: A few-shot learning competition (pre-recorded on NeurIPS platform, by Adrian El Baz)

  • 12:00 - 12:30: Reveal of the MetaDL winners and datasets (live, Zoom room)

  • 12:30 - 13:00: Keynote: Frank Hutter, University of Freiburg (live, Zoom room)

DL 2.0: How Meta-Learning May Power the Next Generation of Deep Learning

Deep Learning (DL) has been incredibly successful, due to its ability to automatically acquire useful representations from raw data by a joint optimization process of all layers. However, current DL practice still requires substantial manual efforts to define the right neural architecture and training hyperparameters to optimally learn these representations for the data at hand. The next logical step is to jointly optimize these components as well, based on a meta-level of learning and optimization. In this talk, I will discuss several advances towards this goal, focusing on (1) joint optimization of several meta-choices in the DL pipeline, (2) efficiency of this meta-optimization, and (3) optimization of uncertainty estimates and robustness to data shift.

  • 13:00 - 13:30: Invited talk by Team Meta Delta, Tsinghua University (live, Zoom room)

MetaDelta++: Improve Generalization of Few-shot System Through Multi-Scale Pretrained Models and Improved Training Strategies

Meta-learning aims at learning quickly on novel tasks with limited data by transferring generic experience learned from previous tasks. Naturally, few-shot learning has been one of the most popular applications for meta-learning. Recently, an ensembled few-shot system MetaDelta is proposed to boost the performance, which won first place in the AAAI 2021 MetaDL challenge with leading performance. However, the generalization ability of MetaDelta is still limited by the homogeneous model setting and weak pretraining and fine-tuning strategies, hindering MetaDelta from being applied to more diverse scenarios and problems. We further boost the performance and generalization ability of MetaDelta by leveraging pre-trained models at multi-scale and improved training strategies, including semi-weakly supervised pretraining, data augmentation, separated learning rate at each layer, lazier BN statistics update, and better decoder design. Our system MetaDelta++ substantially boosts the performance and generalization abilities by a large margin and stands the 1st place in phase 1 of the NeurIPS 2021 MetaDL system with a large margin compared to MetaDelta and other teams.

  • 13:30 - 14:30: Tutorial on Meta-learning, by Isabelle Guyon, Zhengying Liu, Felix Mohr and Jan N. van RIjn (live, Zoom room)

In this slot, we will reflect on some latest developments in Meta-learning. We will present several frameworks that capture the relation between various research directions in meta-learning and AutoML. More specifically, we will reflect on the role of meta-learning in the broader context of machine learning, and on the role of learning curves in AutoML.

  • 14:30 - 15:00: Wrap up session, discussion on follow-up competition (live, Zoom room)


Meta-learning competition at NeurIPS 2021

$1000 of prizes in each track!

Provisional schedule:

    • Aug 2, 2021: Track 1 opens, few-shot-learning from 128x128 color images from 5 different domains.

    • Aug 20, 2021: Challenge bootcamp where newcomers to the field can learn how to get started, with expert coaching.

    • Sep 2, 2021: Track 2 opens, few-shot-learning from tabular data, from 5 different domains. (exact timeline TBD)

    • Oct 2, 2021: End of challenge track 1.

    • Dec 6-14: NeurIPS conference.

The top ranking participants will be invited to co-author a group paper on the results and analysis of the challenge, to be published in PMLR.

New Benchmark for Meta-Learning

We have worked hard to come up with several new meta-learning 128x128 image datasets. We feel that the meta-learning community can greatly benefit from a new benchmark, and plan to publish these datasets open source after the challenge. It is our goal to establish this as a new benchmark for the meta-learning community. Participating in this challenge will give you a head start at getting to work with these datasets.

The challenge is with code submission and will be run on the Codalab platform with generous donations of cloud units from Microsoft and Google. The challenge winners will receive prizes donated by ChaLearn, if they agree to open-source their code. However, there is no such requirement to enter the challenge. The top ranking participants will be invited to co-author a paper on the challenge results, planned to be published in the PMLR, the proceedings track of the Journal of Machine Learning Research.

The competition has a focus on fast solutions. Per submission, the user will get 2 hours of calculations on each meta-dataset. Presumably, this competition favors first-order solutions (such as FO-MAML, LSTM meta-learner) over second-order solutions (such as MAML, TURTLE).


Join our bootcamp and learn about Few Shot Learning

Speakers: Mike Huisman, Felix Mohr, Jan N. van Rijn

Date: August 21st, 8am - 12am (Bogota Timezone)


About Meta Learning

For a comprehensive overview of Meta-learning, we refer to the following resources:

Congratulations to the AAAI 2021 MetaDL winners

Organization

This challenge would not have been possible without the help of many people.

  • Adrian El Baz (Université Paris-Saclay, France)

  • Isabelle Guyon (Université Paris-Saclay; INRIA, France and ChaLearn, USA)

  • Jennifer He (4Paradigm, China)

  • Mike Huisman (Leiden University, the Netherlands)

  • Zhengying Liu (Université Paris-Saclay, France)

  • Yui Man Lui (University of Central Missouri)

  • Felix Mohr (Universidad de la Sabana)

  • Jan N. van Rijn (Leiden University, Netherlands)

  • Sebastien Treguer (Université Paris-Saclay, France)

  • Wei-Wei Tu (4Paradigm, China)

  • Jun Wan (Chinese Academy of Sciences, China)

  • Lisheng Sun (Université Paris-Saclay, France)

  • Haozhe Sun (Université Paris-Saclay; LISN, France)

  • Phan Anh Vu (Université Paris-Saclay; LISN, France)

  • Ihsan Ullah (Université Paris-Saclay; LISN, France)

  • Joaquin Vanschoren (Eindhoven University, the Netherlands)

  • Jun Wan (Chinese Academy of Science, China)

  • Benjia Zhou (Chinese Academy of Sciences, China)

The challenge is running on the Codalab platform, administered by Université Paris-Saclay and maintained by CKCollab LLC, with primary developers:

  • Tyler Thomas (CKCollab, USA)

  • Loic Sarrazin (Artelys, France)

  • Anne-Catherine Letournel (UPSaclay, France)

  • Adrien Pavao (UPSaclay, France)

ChaLearn is the challenge organization coordinator. Microsoft and Google are the primary sponsors of the challenge. 4Paradigm donated prizes, datasets, and contributed to the protocol, baselines methods and beta-testing. Other institutions of the co-organizers provided in-kind contributions, including datasets, data formatting, baseline methods, and beta-testing.

Contact the organizers.

ChaLearn (USA)

University Paris-Saclay (France)

Codalab, UPSaclay (France)

Leiden University (the Netherlands)

Universidad de la Sabana (Colombia)

4Paradigm (China)