First Call for Papers for SUMEval 2025 at COLING 2025: Deadline 18th Oct 2024
Massively Multilingual Language Models (MMLMs), including representation models like mBERT, XLMR and XY-LENT support around 100 languages of the world. Additionally, generative models like GPT-4o, Phi3 and Llama 3.1 are getting attention from the NLP community and the public. However, most existing multilingual NLP benchmarks reflect a handful of cultures and languages. The languages present in evaluation benchmarks are usually high-resource and largely belong to the Indo-European language family. By extension, the cultures represented in evaluation benchmarks are also largely reflective of Western society.
This makes current evaluation unreliable and does not provide a full picture of the performance of MMLMs across the linguistic and cultural landscape. To that end, recent benchmarks such as MEGA and MEGAVERSE cover a larger variety of tasks, cultures, languages, and language families. While this shows progress towards multi-cultural and multi-lingual evaluation, there is still a lot of languages and cultures to cover. Alternative approaches to evaluating MMLMs, including performance prediction and Machine Translation of test data, have also been explored. The challenge to develop multicultural and multilingual benchmarks and optimize performance is even more pronounced in multimodal contexts. This workshop is an extension of the SumEval 2022 workshop, with a wide scope focusing on multicultural evaluation in addition to multilingual evaluation.
Workshop Topic and Content
--------------------------------------------------------------
SUMEval 2025 will be a full-day in-person-only event. We welcome contributions from both original unpublished work and non-archival submissions. Topics of interest include but are not limited to:
- Studies on scaling up multilingual and multicultural evaluation
- Human evaluation of multilingual and multicultural aspects of models
- Automated evaluation metrics for multilingual and multicultural evaluation
- Studies on fairness and other aspects of evaluation
- Studies on cultural representation in multilingual models
- Data sets, benchmarks or libraries for evaluating multi-lingual models
- Probing and analysis of multilingual models across cultures and languages
- Comparison of evaluation strategies for languages, cultures and domains
Archival Submissions
--------------------------------------------------------------
Archival submissions should be up to 8 pages long with unlimited pages for references and appendices. Archival submissions should constitute unpublished work in the topics of interest mentioned above.
We follow ARR submission guidelines. For more information about templates, guidelines, and instructions, see the ARR CFP guidelines. We encourage authors to include a broader impact and ethical concerns statement, following ARR Ethics Policy from the main conference.
All archival submissions will be double-blind peer-reviewed (with author names and affiliations removed) by the program committee and judged by their relevance to the workshop themes.
Non-Archival Submissions
--------------------------------------------------------------
We welcome non-archival submissions of papers that have been previously published as long as they align with the theme of the workshop. Non-archival submissions should not be anonymous and will be evaluated for how closely they match the theme of the workshop.
Important dates (All deadlines are 11:59 PM (Anywhere on Earth)
--------------------------------------------------------------
- Archival Submission: October 18th, 2024
- Non-archival Submission: November 25th, 2024
- Notification of Acceptance: December 3rd, 2024
- Camera-Ready Papers Due: December 10th, 2024
- Workshop Date: co-located with COLING 2025 in January (19th or 20th, exact date to be announced)
Please note that at least one of the authors of each accepted paper must register for the workshop and present the paper in person.
Read more:
https://sites.google.com/view/sumeval-2025