First Call for Shared Task Proposals for RANLP 2025
We invite proposals for tasks to be run as part of RANLP 2025 (Recent Advances in Natural Language Processing): https://ranlp.org/ranlp2025/.
RANLP is one of the most influential and competitive NLP conferences. RANLP 2025 will take place in September 2025 at the Black Sea city of Varna. For the first time in RANLP history, we are organising a shared task campaign as part of the main conference and inviting task organisers to submit their task proposals. Researchers and practitioners from all areas of Natural Language Processing and related communities are invited to submit task proposals.
For RANLP 2025, we welcome any task that can evaluate an automatic system for natural language processing. We especially encourage tasks for languages other than English, multi-lingual tasks, and tasks that develop novel applications of natural language processing.
We strongly encourage proposals based on already published datasets, as this can provide concrete examples and help minimise the challenges of organising the shared task. In the event of receiving many proposals, preference will be given to proposals based on already published datasets.
If you are unsure whether a task is suitable, please contact the shared task chairs to discuss your idea.
Task Selection
Task proposals will be reviewed by at least two reviewers, and the reviews will serve as the basis for acceptance decisions. Task proposals will be evaluated on:
· Novelty - Is the task based on a new problem that has not been explored much in the community? If similar tasks have been organised before, does this task cover new languages/ domains?
· Data – Is the data available and published already? Do annotations have meaningfully high inter-annotator agreements? Have all appropriate licenses for the use and re-use of the data been secured?
· Evaluation—Is the evaluation methodology sound? Is there an automated platform for the evaluation (e.g., CodaLab, Kaggle)?
Task Organisation
We specifically welcome task proposals from early career researchers. However, we strongly encourage tasks that have a diverse team of organisers as that will ease the task organisation. Apart from providing a dataset, task organisers are expected to:
1. Verify data quality in terms of annotator agreement.
2. Verify licenses for the data to allow its use in the competition.
3. Provide task participants with baseline systems.
4. Create a CodaLab or other similar evaluation platform for the task and manage automatic evaluation.
5. Promote the task within the target research community.
6. Manage and organise review process of participants' submissions of system description papers.
7. Write a task description paper to be included in RANLP proceedings.
8. Contribute to the tasks overview paper written by shared task chairs and other task organisers which will also be included in RANLP proceedings.
9. Register and present the shared task description paper at RANLP 2025 on either 11th or 12th September 2025 (the exact date will be confirmed later)
Important Dates
· Task proposals due - October 28, 2024
· Task selection notification – November 4, 2024
Recommended Timeline for the Tasks
· Sample data and task website ready - November 15, 2024
· Training data ready - December 15, 2024
· Evaluation data ready - March 1, 2025
· Evaluation starts – March 10, 2025
· Evaluation end - March 31, 2025 (latest date; task organisers may choose an earlier date)
· Paper submission due – April 20, 2025
· Notification to authors – May 16, 2025
· Task overview paper due – May 25, 2025
· Camera-ready due - May 31, 2025
· Shared task presentation co-located with RANLP 2025 – September 11 and September 12, 2025
Tasks that do not meet critical deadlines such as those for launching the task, setting up the CodaLab website, and uploading samples, training, and evaluation data may be cancelled at the discretion of the shared task chairs.
Submission Details
The task proposal should be a self-contained document of no longer than 2 pages (plus additional pages for references). All submissions must be in PDF format, following the RANLP 2023 template available at https://ranlp.org/ranlp2023/index.php/submissions/
Each proposal should contain the following:
· Overview
o Summary of the task – What is the goal of the task
o Expected number of participants and justification
· Data & Resources
o How the training/testing data will be produced. Discuss whether the dataset is already published
o Details of license, so that the data can be used by the research community
o How much data will be produced
o How data quality will be ensured and evaluated
o An example of what the data would look like
· Evaluation
o The evaluation methodology to be used, including clear evaluation criteria -
o The evaluation platform (i.e. CodaLab, Kaggle etc.)
· Task organisers
o Names, affiliations, email addresses
o brief description of relevant experience or expertise
The submissions should be done via START - https://softconf.com/ranlp25/papers/user/scmd.cgi?scmd=submitPaperCustom...
Proceedings
Tasks overview paper, task description papers and participant papers will be published as part of RANLP 2025 proceedings in ACLAnthology. Task organisers and participants are expected to attend RANLP 2025 on September 11 and September 12, 2025, and present their work in order to include it in the proceedings.
Shared Task Chairs
Dr Tharindu Ranasinghe, Lancaster University, UK
Dr Saad Ezzini, Lancaster University, UK
RANLP 2025 Chairs
Programme Committee Chair: Prof Dr Ruslan Mitkov, Lancaster University, UK
Organising Committee Chair: Prof Dr Galia Angelova, Bulgarian Academy of Sciences, Bulgaria