Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search

Jamie Kiros, William Chan, Geoffrey Hinton


Abstract
We introduce Picturebook, a large-scale lookup operation to ground language via ‘snapshots’ of our physical world accessed through image search. For each word in a vocabulary, we extract the top-k images from Google image search and feed the images through a convolutional network to extract a word embedding. We introduce a multimodal gating function to fuse our Picturebook embeddings with other word representations. We also introduce Inverse Picturebook, a mechanism to map a Picturebook embedding back into words. We experiment and report results across a wide range of tasks: word similarity, natural language inference, semantic relatedness, sentiment/topic classification, image-sentence ranking and machine translation. We also show that gate activations corresponding to Picturebook embeddings are highly correlated to human judgments of concreteness ratings.
Anthology ID:
P18-1085
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
922–933
Language:
URL:
https://aclanthology.org/P18-1085
DOI:
10.18653/v1/P18-1085
Bibkey:
Cite (ACL):
Jamie Kiros, William Chan, and Geoffrey Hinton. 2018. Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 922–933, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (Kiros et al., ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-1085.pdf
Note:
 P18-1085.Notes.pdf
Data
AG NewsMS COCOMultiNLISICKSNLI