This challenge comes with an amazing on-line tutorial guiding you step-by-step and a white paper describing the challenge and baseline results. It focuses on “cross-domain meta-learning” for few-shot image classification using a novel “any-way” and “any-shot” setting. The goal is to meta-learn a good model that can quickly learn tasks from a variety of domains, with any number of classes also called “ways” (within the range 2-20) and any number of training examples per class also called “shots” (within the range 1-20). We carve such tasks from various “mother datasets” selected from diverse domains, such as healthcare, ecology, biology, manufacturing, and others. By using mother datasets from these practical domains, we aim to maximize the humanitarian and societal impact. The competition is with code submission, fully blind-tested on the CodaLab challenge platform. A single (final) submission will be evaluated during the final phase, using ten datasets previously unused by the meta-learning community. After the competition is over, it will remain active to be used as a long-lasting benchmark resource for research in this field. The scientific and technical motivations of this challenge include scalability, robustness to domain changes, and generalization ability to tasks (a.k.a. episodes) in different regimes (any-way any-shot).

Timeline

• Jun 15, 2022: Public phase starts, the starting kit and 10 public datasets will be released for the participants to start familiarizing themselves with the problem and test their solutions locally.

• Jul 1, 2022: Feedback phase starts, the CodaLab platform will start receiving submissions from the participants to provide them instant feedback of their solutions on 10 hidden datasets.

• Sep 1, 2022: Final phase starts, the last submission of each participant from the feedback phase is blind-tested on 10 new hidden datasets to rank the participants and select the winners.

• Oct 1 - 15, 2022: Notification of winners.

The top ranking participants will be invited to co-author a group paper on the results and analysis of the challenge

Data

We have extended the Meta-Album Benchmark (read our PREPRINT) we started putting together last year. It now consists of 30 datasets from 10 domains. They are all image classification datasets, uniformly formatted as 128x128 RGB images, carefully resized with anti-aliasing, cropped manually, and annotated with various meta-data, including super-classes.

Protocol

During the feedback and final phases the submissions of the participants will be evaluated on 10 datasets from 10 domains. The few-shot learning problems are often referred as N-way K-shots problem. This name refers to episode configuration at meta-test time. The number of ways N denotes the number of classes in an episode that represents an image classification task. The number of shots K denotes the number of examples per class in the support (training) set. In this challenge, we focus on the any-way any-shot setting: Tasks at meta-test time are image classification problems with a number of classes varying from 2 to 20, with 1 to 20 labeled example per class. Thus, at meta-test time the participants' code might be tested in the following way:

• Test task 1: 5-way 1-shot task from dataset 3.

• Test task 2: 3-way 15-shots task from dataset 7.

• Test task 3: 12-way 4-shots task from dataset 1.

• etc.

The ranking will be made by averaging performances over all meta-testing tasks in the final test phase, and taking the worst of 3 randomly seeded runs. Performance will be measured with balanced accuracy, normalized between 0 (random guess) and 1 (perfect prediction).

Prize distribution

To encourage diverse participation, we will donate prizes in 5 leagues:

• Free-style league: Submit a solution obeying basic challenge rules (pre-trained models allowed).

• Meta-learning league: Submit a solution that meta-learns from scratch (no pre-training allowed).

• New-in-ML league: Be a participant who has less than 10 ML publications, none of which ever accepted to the main track of a major conference.

• Women league: Special league to encourage women, since they rarely enter challenges.

• Participant of a rarely represented country: Be a participant of a group that is not in the top 10 most represented countries of Kaggle challenge participants.

In each league: 1st prize=400€, 2nd prize=250€, 3rd prize=150€. Entering multiple leagues is permitted. In addition, the top ranking participants will receive a certificate, and be invited to co-author a post-challenge analysis paper with the organizers, published in the NeurIPS proceedings of the competition track. Travel awards will also be distributed depending upon merit and needs to top ranking participants who want to attend NeurIPS in person, depending on availability.

Lean more

• First challenge round (NeurIPS'21) including the code of the winners.

• Paper on the first round, explaining how they won (El Baz et al, 2022).

• Paper on Meta-Album and baseline results for THIS challenge (Ullah et al, 2022).

• Metalearning: Applications to Data Mining (Book, by Brazdil et al. (2008))

• Meta-learning: A survey (Book chapter, by Vanschoren (2019))

• Meta-learning in neural networks: A survey (Arxiv, Hospedales et al. (2020))

• A Survey of Deep Meta-Learning (Artificial Intelligence Review, Huisman et al. (2021))