Embedded Files
Machine learning has solved with success many mono-task problems, but at the expense of long wasteful training times. Meta-learning promises to leverage the experience gained on previous tasks to train models faster, with fewer examples, and possibly better performance. Approaches include learning from algorithm evaluations, from task properties (or meta-features), and from prior models.
Following the AutoDL 2019-2020 challenge series and past meta-learning challenges and benchmarks we have organized, including MetaDL@NeurIPS'21 (read our PAPER) we are organizing three competitions in 2022:
1st Round of Meta-learning from learning curves (ENDED will be presented at WCCI 2022).
2nd Round of Meta-learning from learning curves (ENDED, accepted to AutoML-Conf 2022).
Cross-domain meta-learning (ENDED, accepted to NeurIPS'22).
Workshop (Meta-)Knowledge Transfer/Communication in Different Systems. Friday 23rd of Sept. 2022, Grenoble, France, associated with ECML/PKDD 2022
Upcoming presentations:
NeurIPS'22
Meta-Album (Thursday 1st of December): [PAPER][POSTER][WEBSITE]
Cross-Domain MetaDL Competition (Tuesday 6th of December): [PAPER][POSTER][WEBSITE][RECORDED SESSION]
Contact us, if you want to join the organizing team.
€1000 in prizes!
The main goal of this competition is to push the state-of-the-art in meta-learning from learning curves, an important sub-problem in meta-learning. A learning curve evaluates an algorithm's incremental performance improvements, as a function of training time, number of iterations, and/or number of examples. Analysis of past ML challenges revealed that top-ranking methods often involve switching between algorithms during training. We are interested in meta-learning strategies that leverage information on partially trained algorithms, hence reducing the cost of training them to convergence. Furthermore, we want to study the potential benefit of learned policies, as opposed to applying hand-crafted black-box optimization methods. We offer pre-computed learning curves as a function of time, to facilitate benchmarking. Meta-learners must “pay” a cost emulating computational time for revealing their next values. Hence, meta-learners are expected to learn the exploration-exploitation trade-offs between exploiting an already tried good candidate algorithm and exploring new candidate algorithms. The first round of this competition was previously organized for WCCI 2022, please see the results on our website. In this new round, we propose an enlarged and more challenging meta-dataset. Having participated in the first round is NOT a prerequisite. The winners of the first round have open-sourced their code.
Timeline
Timeline
May 16, 2022: Public phase starts, starting kit available.
May 23, 2022: Feedback phase starts, you can now make submissions.
July 4, 2022: Final phase starts. No new submission: your last submission of the previous phase is tested on test data.
July 11, 2022: Final phase ends.
July 25, 2022: Results revealed at AutoML-conf.
Data
Data
We created a meta-dataset from the 30 datasets of the AutoML challenge, by running algorithms with different hyperparameter, from which we obtained learning curves both for the validation sets and the test sets.
Protocol
Protocol
During a development phase, participants submit agents to be meta-trained and meta-tested on all data, except the test learning curves of each task. During a final test phase, a scoring program computes the agent’s performance on the test learning curves, based on pre-recorded agent suggestions. Furthermore, the ingestion program runs a hold-out procedure: in each split, we hold out 5/30 datasets for meta-testing, and use the rest for meta-training.
Evaluation
Evaluation
The agent is evaluated by the Area under the agents’ Learning Curve (ALC). The values will be averaged over all meta-test datasets and shown on the leaderboards. The final ranking will be made according to the average test ALC.
Meta-Learning from Learning Curves challenge (1st round, terminated)
Meta-Learning from Learning Curves challenge (1st round, terminated)
€4000 in prizes!
Congratulations to the winners [Technical Report]:
The rows with yellow backgrounds indicate that not all team members fulfill the league requirements and therefore have less priority for prizes.
This challenge comes with an amazing on-line tutorial guiding you step-by-step and a white paper describing the challenge and baseline results. It focuses on “cross-domain meta-learning” for few-shot image classification using a novel “any-way” and “any-shot” setting. The goal is to meta-learn a good model that can quickly learn tasks from a variety of domains, with any number of classes also called “ways” (within the range 2-20) and any number of training examples per class also called “shots” (within the range 1-20). We carve such tasks from various “mother datasets” selected from diverse domains, such as healthcare, ecology, biology, manufacturing, and others. By using mother datasets from these practical domains, we aim to maximize the humanitarian and societal impact. The competition is with code submission, fully blind-tested on the CodaLab challenge platform. A single (final) submission will be evaluated during the final phase, using ten datasets previously unused by the meta-learning community. After the competition is over, it will remain active to be used as a long-lasting benchmark resource for research in this field. The scientific and technical motivations of this challenge include scalability, robustness to domain changes, and generalization ability to tasks (a.k.a. episodes) in different regimes (any-way any-shot).
Timeline
Timeline
Jun 15, 2022: Public phase starts, the starting kit and 10 public datasets will be released for the participants to start familiarizing themselves with the problem and test their solutions locally.
Jul 1, 2022: Feedback phase starts, the CodaLab platform will start receiving submissions from the participants to provide them instant feedback of their solutions on 10 hidden datasets.
Sep 1, 2022: Final phase starts, the last submission of each participant from the feedback phase is blind-tested on 10 new hidden datasets to rank the participants and select the winners.
Oct 1 - 15, 2022: Notification of winners.
The top ranking participants will be invited to co-author a group paper on the results and analysis of the challenge
Data
Data
We have extended the Meta-Album Benchmark (read our PREPRINT) we started putting together last year. It now consists of 30 datasets from 10 domains. They are all image classification datasets, uniformly formatted as 128x128 RGB images, carefully resized with anti-aliasing, cropped manually, and annotated with various meta-data, including super-classes.
Protocol
Protocol
During the feedback and final phases the submissions of the participants will be evaluated on 10 datasets from 10 domains. The few-shot learning problems are often referred as N-way K-shots problem. This name refers to episode configuration at meta-test time. The number of ways N denotes the number of classes in an episode that represents an image classification task. The number of shots K denotes the number of examples per class in the support (training) set. In this challenge, we focus on the any-way any-shot setting: Tasks at meta-test time are image classification problems with a number of classes varying from 2 to 20, with 1 to 20 labeled example per class. Thus, at meta-test time the participants' code might be tested in the following way:
Test task 1: 5-way 1-shot task from dataset 3.
Test task 2: 3-way 15-shots task from dataset 7.
Test task 3: 12-way 4-shots task from dataset 1.
etc.
The ranking will be made by averaging performances over all meta-testing tasks in the final test phase, and taking the worst of 3 randomly seeded runs. Performance will be measured with balanced accuracy, normalized between 0 (random guess) and 1 (perfect prediction).
Prize distribution
Prize distribution
To encourage diverse participation, we will donate prizes in 5 leagues:
Free-style league: Submit a solution obeying basic challenge rules (pre-trained models allowed).
Meta-learning league: Submit a solution that meta-learns from scratch (no pre-training allowed).
New-in-ML league: Be a participant who has less than 10 ML publications, none of which ever accepted to the main track of a major conference.
Women league: Special league to encourage women, since they rarely enter challenges.
Participant of a rarely represented country: Be a participant of a group that is not in the top 10 most represented countries of Kaggle challenge participants.
In each league: 1st prize=400€, 2nd prize=250€, 3rd prize=150€. Entering multiple leagues is permitted. In addition, the top ranking participants will receive a certificate, and be invited to co-author a post-challenge analysis paper with the organizers, published in the NeurIPS proceedings of the competition track. Travel awards will also be distributed depending upon merit and needs to top ranking participants who want to attend NeurIPS in person, depending on availability.
Lean more
Lean more
First challenge round (NeurIPS'21) including the code of the winners.
Paper on the first round, explaining how they won (El Baz et al, 2022).
Paper on Meta-Album and baseline results for THIS challenge (Ullah et al, 2022).
Metalearning: Applications to Data Mining (Book, by Brazdil et al. (2008))
Meta-learning: A survey (Book chapter, by Vanschoren (2019))
Meta-learning in neural networks: A survey (Arxiv, Hospedales et al. (2020))
A Survey of Deep Meta-Learning (Artificial Intelligence Review, Huisman et al. (2021))
We are grateful to Microsoft and Google for generous cloud unit donations and to 4Paradigm for donating prizes. This project is also supported by ChaLearn and HUMANIA chair of AI grant ANR-19-CHIA-00222 of the Agence Nationale de la Recherche (France), and the EU TAILOR consortium. Researchers and students from Université Paris-Saclay, Universiteit Leiden, TU Eindhoven, Universidad de la Sabana and 4Paradigm have contributed. The challenge is hosted by Codalab (Université Paris-Saclay).
University Paris-Saclay (France)
Leiden University (the Netherlands)
Universidad de la Sabana (Colombia)
Report abuse