Machine learning has solved with success many mono-task problems, but at the expense of long wasteful training times. Meta-learning promises to leverage the experience gained on previous tasks to train models faster, with fewer examples, and possibly better performance. Approaches include learning from algorithm evaluations, from task properties (or meta-features), and from prior models.
Following the AutoDL 2019-2020 challenge series and past meta-learning challenges and benchmarks we have organized, including MetaDL@NeurIPS'21 (read our PAPER) we are organizing three competitions in 2022:
1st Round of Meta-learning from learning curves (ENDED will be presented at WCCI 2022).
2nd Round of Meta-learning from learning curves (ENDED, accepted to AutoML-Conf 2022).
Cross-domain meta-learning (ENDED, accepted to NeurIPS'22).
Workshop (Meta-)Knowledge Transfer/Communication in Different Systems. Friday 23rd of Sept. 2022, Grenoble, France, associated with ECML/PKDD 2022
Contact us, if you want to join the organizing team.
€1000 in prizes!
May 16, 2022: Public phase starts, starting kit available.
May 23, 2022: Feedback phase starts, you can now make submissions.
July 4, 2022: Final phase starts. No new submission: your last submission of the previous phase is tested on test data.
July 11, 2022: Final phase ends.
July 25, 2022: Results revealed at AutoML-conf.
Meta-Learning from Learning Curves challenge (1st round, terminated)
€4000 in prizes!
Congratulations to the winners [Technical Report]:
The rows with yellow backgrounds indicate that not all team members fulfill the league requirements and therefore have less priority for prizes.
Jun 15, 2022: Public phase starts, the starting kit and 10 public datasets will be released for the participants to start familiarizing themselves with the problem and test their solutions locally.
Jul 1, 2022: Feedback phase starts, the CodaLab platform will start receiving submissions from the participants to provide them instant feedback of their solutions on 10 hidden datasets.
Sep 1, 2022: Final phase starts, the last submission of each participant from the feedback phase is blind-tested on 10 new hidden datasets to rank the participants and select the winners.
Oct 1 - 15, 2022: Notification of winners.
The top ranking participants will be invited to co-author a group paper on the results and analysis of the challenge
We have extended the Meta-Album Benchmark (read our PREPRINT) we started putting together last year. It now consists of 30 datasets from 10 domains. They are all image classification datasets, uniformly formatted as 128x128 RGB images, carefully resized with anti-aliasing, cropped manually, and annotated with various meta-data, including super-classes.
During the feedback and final phases the submissions of the participants will be evaluated on 10 datasets from 10 domains. The few-shot learning problems are often referred as N-way K-shots problem. This name refers to episode configuration at meta-test time. The number of ways N denotes the number of classes in an episode that represents an image classification task. The number of shots K denotes the number of examples per class in the support (training) set. In this challenge, we focus on the any-way any-shot setting: Tasks at meta-test time are image classification problems with a number of classes varying from 2 to 20, with 1 to 20 labeled example per class. Thus, at meta-test time the participants' code might be tested in the following way:
Test task 1: 5-way 1-shot task from dataset 3.
Test task 2: 3-way 15-shots task from dataset 7.
Test task 3: 12-way 4-shots task from dataset 1.
The ranking will be made by averaging performances over all meta-testing tasks in the final test phase, and taking the worst of 3 randomly seeded runs. Performance will be measured with balanced accuracy, normalized between 0 (random guess) and 1 (perfect prediction).
To encourage diverse participation, we will donate prizes in 5 leagues:
Free-style league: Submit a solution obeying basic challenge rules (pre-trained models allowed).
Meta-learning league: Submit a solution that meta-learns from scratch (no pre-training allowed).
New-in-ML league: Be a participant who has less than 10 ML publications, none of which ever accepted to the main track of a major conference.
Women league: Special league to encourage women, since they rarely enter challenges.
Participant of a rarely represented country: Be a participant of a group that is not in the top 10 most represented countries of Kaggle challenge participants.
In each league: 1st prize=400€, 2nd prize=250€, 3rd prize=150€. Entering multiple leagues is permitted. In addition, the top ranking participants will receive a certificate, and be invited to co-author a post-challenge analysis paper with the organizers, published in the NeurIPS proceedings of the competition track. Travel awards will also be distributed depending upon merit and needs to top ranking participants who want to attend NeurIPS in person, depending on availability.
First challenge round (NeurIPS'21) including the code of the winners.
Paper on the first round, explaining how they won (El Baz et al, 2022).
Paper on Meta-Album and baseline results for THIS challenge (Ullah et al, 2022).
We are grateful to Microsoft and Google for generous cloud unit donations and to 4Paradigm for donating prizes. This project is also supported by ChaLearn and HUMANIA chair of AI grant ANR-19-CHIA-00222 of the Agence Nationale de la Recherche (France), and the EU TAILOR consortium. Researchers and students from Université Paris-Saclay, Universiteit Leiden, TU Eindhoven, Universidad de la Sabana and 4Paradigm have contributed. The challenge is hosted by Codalab (Université Paris-Saclay).