Noun-Modifier Questions (State of the art)
Jump to navigation
Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
- multiple-choice noun-modifier questions
- 2,180 seven-choice questions for evaluating noun-modifier composition models
- questions derived from WordNet
- split into 680 questions for training (e.g., development, parameter tuning) and 1,500 questions for testing
- introduced in Turney (2012)
- see also: Similarity (State of the art)
Sample question
Stem: coin collector Choices: (a) numismatist (b) coin (c) collector (d) coinage (e) aggregator (f) reflectiveness (g) plasma Solution: (a) numismatist
Table of results
| Algorithm | Reference for algorithm | Reference for experiment | Correct - 7-choices | Correct - 14-choices |
|---|---|---|---|---|
| Vector addition | Landauer and Dumais (1997) | Turney (2012) | 50.1 | 22.5 |
| Element-wise multiplication | Mitchell and Lapata (2008) | Turney (2012) | 57.5 | 27.4 |
| Dual-Space model | Turney (2012) | Turney (2012) | 58.3 | 41.5 |
| SuperSim | Turney (2013) | Turney (2013) | 75.9 | 68.0 |
| Holistic model | Turney (2012) | Turney (2012) | 81.6 | NA |
References
Turney, P.D. (2012). Domain and function: A dual-space model of semantic relations and compositions, Journal of Artificial Intelligence Research (JAIR), 44, 533-585.
Turney, P.D. (2013), Distributional semantics beyond words: Supervised learning of analogy and paraphrase, Transactions of the Association for Computational Linguistics (TACL), 1, 353-366.