The “test suites” sub-task will be included for the sixth time in the
General MT Shared Task of the Conference on Machine Translation
(WMT24).
*OVERVIEW*
Test suites are custom extensions to the test sets of the General MT
Shared Task, constructed so that they can focus on concrete aspects of the
MT output. They consist of a source-side test-set and a customized
evaluation service. As opposed to the standard evaluation process which
produces generic quality scores, test suites often produce separate
fine-grained results for each phenomenon.
Since the usage of LLMs for translation is getting more popular, and we
are expecting more LLMs participations in WMT this year, the theme of this
year’s test suite sub-task is "Help us break LLMs", i.e. to reveal
weaknesses and serious flaws of LLMs when translating, hidden within the
overall high-quality generation.
*IMPORTANT DATES*
- 11th April: Test suite source texts may be submitted for a pre-run on
SoTA MT systems - 12th June: Test suite source texts must reach us
- 11th July: Translated test suites shipped back to test suites authors:
- TBC - August: Test suite description and analysis paper
- 12th-13th November: Conference
Potential participants are kindly requested to fill in this form
https://forms.office.com/e/e4JuMTSWFF
Further information can be found in the dedicated page of the WMT
website
http://www2.statmt.org/wmt24/testsuite-subtask.html