Grounding Language by Continuous Observation of Instruction Following

Ting Han, David Schlangen


Abstract
Grounded semantics is typically learnt from utterance-level meaning representations (e.g., successful database retrievals, denoted objects in images, moves in a game). We explore learning word and utterance meanings by continuous observation of the actions of an instruction follower (IF). While an instruction giver (IG) provided a verbal description of a configuration of objects, IF recreated it using a GUI. Aligning these GUI actions to sub-utterance chunks allows a simple maximum entropy model to associate them as chunk meaning better than just providing it with the utterance-final configuration. This shows that semantics useful for incremental (word-by-word) application, as required in natural dialogue, might also be better acquired from incremental settings.
Anthology ID:
E17-2079
Volume:
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Mirella Lapata, Phil Blunsom, Alexander Koller
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
491–496
Language:
URL:
https://aclanthology.org/E17-2079
DOI:
Bibkey:
Cite (ACL):
Ting Han and David Schlangen. 2017. Grounding Language by Continuous Observation of Instruction Following. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 491–496, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
Grounding Language by Continuous Observation of Instruction Following (Han & Schlangen, EACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/E17-2079.pdf