Organized by the 4th Workshop on Multilingual Representation Learning
In collocation with EMNLP 2024
With the advancement of language models accessing and processing tons of information in different formats and languages, it has become of great importance to be able to assess the capabilities to access and provide the right information useful to different audiences. Large language models (LLMs) continue to demonstrate outstanding performance in many applications that require competence in language understanding and generation, but this performance is especially prominent in English, where large amounts of public evaluation benchmarks for various downstream tasks are available and the extent to which language models can be reliably deployed in terms of different languages and domains are not still well established.
In this new shared task we provide new high-quality annotations in a selected set of data-scarce and typologically-diverse languages that can be used in evaluation of multilingual LLMs in information retrieval tasks; including Named Entity Recognition (NER) and Reading Comprehension (RC), on test sets curated from articles on Wikipedia. The main objective of the shared task is to assess and understand the multilingual characteristics of the inference capability of multilingual LLMs in understanding and generating language based on logical, factual or causal relationships between knowledge contained over long contexts of text, especially under low-resource settings.
Task and Evaluation
Our task provides a multi-task evaluation format that assesses reading comprehension capabilities of language models in terms of two subtasks: named entity recognition and question answering.
Named Entity Recognition (NER)
is a classification task that identifies phrases in a text that refer to entities or predefined categories (such as dates, person, organization and location names) and it is an important capability for information access systems that perform entity look-ups for knowledge verification, spell-checking or localization applications.
The objective of the system is to tag the named entities in a given text as person (PER), organization (ORG), location (LOC), and date (DATE) (Our tag set uses $$ as delimiter).
The objective of the system is to tag the named entities in a given text as person (PER), organization (ORG), location (LOC), and date (DATE) (Our tag set uses $$ as delimiter).
Question Answering (QA)
is an important capability that enables responding to natural language questions with answers found in text. Here we focus on the information-seeking scenario where questions can be asked without knowing the answer—it is the system’s job to locate a suitable answer passage (if any). The information-seeking question-answer pairs tend to exhibit less lexical and morphosyntactic overlap between the question and answer since they are written separately, which is a more suitable setting to evaluate typologically-diverse languages.
Here, the system is given a question, title, and a passage and the system must pick the right answer among a list of 4 different potential options.
We evaluate model performance in the generative task terms of the accuracy in the multi-choice answering task, and the F1 accuracy in the NER task. We obtain a final score by averaging the scores of QA and NER.
Data and Languages
Teams can use any resources relevant to the task.
The test sets for official evaluation will be released one week before the submission date, and will be in the following languages:
The test sets for official evaluation will be released one week before the submission date, and will be in the following languages:
- Turkish
- Uzbek
- Indonesian
- Alemannic
- Yoruba
- Igbo
Important Dates
- June 1, 2024: Release of validation data
- August 1, 2024: Release of test data
- August 20, 2024: Deadline to release external data and resources used in systems
- September 1, 2024: Deadline for submission of systems , and release of external data and resources used in systems.
- September 20, 2024: Release of rankings and results
- September 30, 2024: Deadline for submitting system description papers
All deadlines are 11.59 pm UTC -12h (“anywhere on Earth”)
Organizers
Duygu Ataman
David Ifeoluwa Adelani
Mammad Hajili
Francesco Tinner
Inder Khatri
Shared Task Prize
The winning team will receive an award of 500 USD and will be given a presentation during the workshop.