While large language models (LLMs) offer to become a viable alternative to traditional rule-based data-to-text (D2T) natural language generation (NLG), they still suffer from well-known neural model issues, such as lack of controllability and risk of producing harmful text. There are many potential solutions to this problem up for discussion.
The Practical D2T workshop at INLG 2024 aims to build a space for researchers to discuss and present innovative work on D2T systems using LLMs. Building upon the 2023 edition’s hackathon, Practical D2T 2024 opens up a broader range of activities, including a special track for neuro-symbolic D2T approaches and a shared task in D2T evaluation focused on semantic accuracy.
Workshop Topic and Content
---------------------------------------------------
Practical D2T 2024 will be a full-day in-person-only event. We welcome contributions from both original unpublished work and non-archival submissions, in the form of long (8 pages) or short (4 pages) papers, on topics including but not limited to:
- Design, implementation and evaluation of LLM-assisted D2T systems
- Cross-domain adaption of LLMs for D2T
- User perceptions and acceptance of LLM-generated text in D2T
- Bias, fairness and red-teaming issues in LLM-assisted D2T systems
- Leveraging LLMs for D2T in low-resource languages and domains
- Error analysis and debugging techniques for LLM-assisted D2T
- Human-in-the-loop approaches for improving LLM-assisted D2T
- Comparison between LLM-assisted D2T and traditional symbolic approaches
Special Track: Neuro-Symbolic D2T
---------------------------------------------------
Research is currently seeing a renewed interest in developing systems combining neural and symbolic approaches to improve explainability and reduce dependence on training data. Practical D2T 2024 will feature a special track on neuro-symbolic approaches to D2T. Submissions for papers in the special track follow the same requirements and procedure as the main workshop submissions.
Shared task: Improving Semantic Accuracy in LLM-assisted D2T
---------------------------------------------------
This year will feature a shared task on improving semantic accuracy of D2T systems. Participants will build an LLM-assisted D2T system to generate textual reports from various domains, such as weather forecasting, product descriptions or sports reports. We will provide testing data obtained from public APIs, to limit potential previous exposure to the used LLMs.
We encourage participants to focus on system robustness and objective evaluation, rather than metrics scores. Because of this, participants will receive an initial evaluation script, that they are encouraged to change/improve. All submitted system’s outputs will be evaluated against every submitted custom evaluation, and correlated with human ratings.
The system reaching the highest correlation with humans will be declared winner of the competition. Results and participants’ system descriptions will be featured in the workshop proceedings.
Important dates (all deadlines are 23:59 UTC-12)
---------------------------------------------------
- Evaluation script and data release for known domains (shared task) 24 June
- Regular paper submission (main & special track, archival & non-archival): 22 July
- Known domains system output submission & surprise domain data release: 29 July
- Surprise domain system outputs submission: 5 August
- System description submission (shared task): 12 August
- Notification of acceptance (main, special track and shared task): 19 August
- Camera-ready (main, special track and shared task): 28 August
- Workshop: 23/24 September (to be announced)