[Apologies for cross-posting]
Call for partipation
ODESIA CHALLENGE @ SEPLN 2024
Evaluation of NLP Systems in Spanish
https://leaderboard.odesia.uned.es/en/leaderboard/challenge
The members of the ODESIA project (Space for the Observation of the Development of Spanish in Artificial Intelligence) and the organizers of SEPLN 2024 are pleased to invite you to participate in the ODESIA Challenge @ SEPLN 2024. This competition aims to promote the development and evaluation of language technologies in Spanish using the evaluation platform and datasets provided by ODESIA.
Participants must create a system capable of solving 10 discriminative Natural Language Processing (NLP) tasks in Spanish belonging to the ODESIA Leaderboard (https://leaderboard.odesia.uned.es/en). The winning team will receive a cash prize of 3,000 euros, donated by the company Llorente y Cuenca Madrid, SL. (conditions apply, see below).
Tasks
The ODESIA-CORE benchmark consists of 10 discriminative tasks with public training datasets and private test datasets (not previously distributed by any means) created within the ODESIA initiative. The private nature of the test data guarantees the absence of contamination in the leaderboard results: no LLM should have seen the test set annotations in its pre-training phase.
Accepted Systems
All types of Natural Language Processing (NLP) systems that are applied uniformly to all tasks will be accepted. That is, each participation must be a single system that applies to all tasks, instead of different approaches for each task. A submission in which the solution for each task is constructed independently will not be acceptable. For illustrative purposes, systems with the following characteristics (the list is non-exhaustive) are acceptable:
- The system is an encoder-type LLM (or an ensemble of LLMs), to which a fine-tuning process is applied for each of the challenge tasks, using the training data provided in the participants’ package or from other sources as deemed appropriate by the participating team.
- The system uses one or more generative LLMs, combined with a uniform zero-shot, one-shot or few-shot prompting strategy.
- The system uses one or more generative LLMs combined with a retrieval-augmented generation strategy on the training dataset or other external sources.
- Any combination of the above methods, as long as it is applied uniformly to all datasets.
Registration and Participation
Teams will have to pre-register before they can participate. Each team will register a single account on the ODESIA Leaderboard evaluation platform using the form provided for this purpose (https://leaderboard.odesia.uned.es/en/leaderboard/submit). The organizers will provide a username and password on the ODESIA Leaderboard platform upon validation of the registration data.
Prize
A single prize of 3,000 euros -donated by Llorente y Cuenca Madrid, SL- will be awarded to the team that submits the system with the best global average performance in the ODESIA-CORE tasks for Spanish, and that outperforms the best current model in the Leaderboard (XLM-Roberta-Large - 0.5873). Please note that for the prize to be awarded, there must be a minimum of five teams submitting results; if this number is not met, the organization reserves the right to defer the challenge's deadline until this number is reached. Also, The winning team commits to present its solution (in-person or online) at the Award ceremony at SEPLN 2024 (25th September 2024, Valladolid - Spain).
Important dates
- Registration opens: 1st July 2024
- Registration closes: 30th July 2024
- Submission deadline: 14th September 2024
- Official results announced: 16-20th September 2024
- Award ceremony and presentation of results: 25th September 2024 - 5:30pm, at SEPLN 2024
For all deadlines the time is 23:59 AoE (Anywhere on Earth)
Organizing Committee
Alejandro Benito-Santos (co-chair, UNED)
Roser Morante (co-chair, UNED)
Julio Gonzalo (UNED)
Jorge Carrillo-de-Albornoz (UNED)
Laura Plaza (UNED)
Enrique Amigó (UNED)
Víctor Fresno (UNED)
Andrés Fernández (UNED)
Adrián Ghajari (UNED)
Guillermo Marco (UNED)
Eva Sánchez (UNED)
Miguel Lucas (LLYC)
Contact and Information
The full contest rules, along with instructions to participate, can be found at the ODESIA Leaderboard website (https://leaderboard.odesia.uned.es/en/leaderboard/challenge). For questions related to the challenge, please join our Discord server: #odesia-challenge-2024. You can also contact the challenge co-chairs, Alejandro Benito-Santos (al.benito [at] lsi.uned.es) and Roser Morante (r.morant [at] lsi.uned.es).