Difference between revisions of "Noun-Modifier Questions (State of the art)"
Jump to navigation
Jump to search
(Created page with "* [http://jair.org/media/3640/live-3640-6413-jair.txt multiple-choice noun-modifier questions] * 2,180 seven-choice questions for evaluating noun-modifier composition models *...") |
|||
| (3 intermediate revisions by the same user not shown) | |||
| Line 4: | Line 4: | ||
* split into 680 questions for training (e.g., development, parameter tuning) and 1,500 questions for testing | * split into 680 questions for training (e.g., development, parameter tuning) and 1,500 questions for testing | ||
* introduced in [http://jair.org/papers/paper3640.html Turney (2012)] | * introduced in [http://jair.org/papers/paper3640.html Turney (2012)] | ||
| + | * see also: [[Similarity (State of the art)]] | ||
== Sample question == | == Sample question == | ||
| Line 47: | Line 48: | ||
== Table of results == | == Table of results == | ||
| + | |||
| + | {| border="1" cellpadding="5" cellspacing="1" | ||
| + | |- | ||
| + | ! Algorithm | ||
| + | ! Reference for algorithm | ||
| + | ! Reference for experiment | ||
| + | ! Correct - 7-choices | ||
| + | ! Correct - 14-choices | ||
| + | |- | ||
| + | | Vector addition | ||
| + | | Landauer and Dumais (1997) | ||
| + | | Turney (2012) | ||
| + | | 50.1 | ||
| + | | 22.5 | ||
| + | |- | ||
| + | | Element-wise multiplication | ||
| + | | Mitchell and Lapata (2008) | ||
| + | | Turney (2012) | ||
| + | | 57.5 | ||
| + | | 27.4 | ||
| + | |- | ||
| + | | Dual-Space model | ||
| + | | Turney (2012) | ||
| + | | Turney (2012) | ||
| + | | 58.3 | ||
| + | | 41.5 | ||
| + | |- | ||
| + | | SuperSim | ||
| + | | Turney (2013) | ||
| + | | Turney (2013) | ||
| + | | 75.9 | ||
| + | | 68.0 | ||
| + | |- | ||
| + | | Holistic model | ||
| + | | Turney (2012) | ||
| + | | Turney (2012) | ||
| + | | 81.6 | ||
| + | | NA | ||
| + | |} | ||
== References == | == References == | ||
| + | |||
| + | Landauer, T.K., and Dumais, S.T. (1997). [http://lsa.colorado.edu/papers/plato/plato.annote.html A solution to Plato's problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge]. ''Psychological Review'', 104(2):211-240. | ||
| + | |||
| + | Mitchell, J., and Lapata, M. (2008). [http://www.newdesign.aclweb.org/anthology-new/P/P08/P08-1028.pdf Vector-based models of semantic composition]. In ''Proceedings of ACL-08: HLT'', pages 236-244, Columbus, Ohio. Association for Computational Linguistics. | ||
Turney, P.D. (2012). [http://jair.org/papers/paper3640.html Domain and function: A dual-space model of semantic relations and compositions], ''Journal of Artificial Intelligence Research (JAIR)'', 44, 533-585. | Turney, P.D. (2012). [http://jair.org/papers/paper3640.html Domain and function: A dual-space model of semantic relations and compositions], ''Journal of Artificial Intelligence Research (JAIR)'', 44, 533-585. | ||
| Line 56: | Line 100: | ||
[[Category:State of the art]] | [[Category:State of the art]] | ||
| + | [[Category:Similarity]] | ||
Latest revision as of 11:26, 28 June 2015
- multiple-choice noun-modifier questions
- 2,180 seven-choice questions for evaluating noun-modifier composition models
- questions derived from WordNet
- split into 680 questions for training (e.g., development, parameter tuning) and 1,500 questions for testing
- introduced in Turney (2012)
- see also: Similarity (State of the art)
Sample question
Stem: coin collector Choices: (a) numismatist (b) coin (c) collector (d) coinage (e) aggregator (f) reflectiveness (g) plasma Solution: (a) numismatist
Table of results
| Algorithm | Reference for algorithm | Reference for experiment | Correct - 7-choices | Correct - 14-choices |
|---|---|---|---|---|
| Vector addition | Landauer and Dumais (1997) | Turney (2012) | 50.1 | 22.5 |
| Element-wise multiplication | Mitchell and Lapata (2008) | Turney (2012) | 57.5 | 27.4 |
| Dual-Space model | Turney (2012) | Turney (2012) | 58.3 | 41.5 |
| SuperSim | Turney (2013) | Turney (2013) | 75.9 | 68.0 |
| Holistic model | Turney (2012) | Turney (2012) | 81.6 | NA |
References
Landauer, T.K., and Dumais, S.T. (1997). A solution to Plato's problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104(2):211-240.
Mitchell, J., and Lapata, M. (2008). Vector-based models of semantic composition. In Proceedings of ACL-08: HLT, pages 236-244, Columbus, Ohio. Association for Computational Linguistics.
Turney, P.D. (2012). Domain and function: A dual-space model of semantic relations and compositions, Journal of Artificial Intelligence Research (JAIR), 44, 533-585.
Turney, P.D. (2013), Distributional semantics beyond words: Supervised learning of analogy and paraphrase, Transactions of the Association for Computational Linguistics (TACL), 1, 353-366.