Difference between revisions of "Noun-Modifier Questions (State of the art)"

From ACL Wiki
Jump to: navigation, search
(Created page with "* [http://jair.org/media/3640/live-3640-6413-jair.txt multiple-choice noun-modifier questions] * 2,180 seven-choice questions for evaluating noun-modifier composition models *...")
 
Line 47: Line 47:
  
 
== Table of results ==  
 
== Table of results ==  
 +
 +
{| border="1" cellpadding="5" cellspacing="1"
 +
|-
 +
! Algorithm
 +
! Reference for algorithm
 +
! Reference for experiment
 +
! Correct - 7-choices
 +
! Correct - 14-choices
 +
|-
 +
| Vector addition
 +
| Landauer and Dumais (1997)
 +
| Turney (2012)
 +
| 50.1
 +
| 22.5
 +
|-
 +
| Element-wise multiplication
 +
|
 +
|
 +
| 57.5
 +
| 27.4
 +
|-
 +
| Dual-Space model
 +
|
 +
|
 +
| 58.3
 +
| 41.5
 +
|-
 +
| SuperSim
 +
|
 +
|
 +
| 75.9
 +
| 68.0
 +
|-
 +
| Holistic model
 +
|
 +
|
 +
| 81.6
 +
| NA
 +
|}
  
 
== References ==
 
== References ==

Revision as of 06:18, 28 April 2014

  • multiple-choice noun-modifier questions
  • 2,180 seven-choice questions for evaluating noun-modifier composition models
  • questions derived from WordNet
  • split into 680 questions for training (e.g., development, parameter tuning) and 1,500 questions for testing
  • introduced in Turney (2012)

Sample question

Stem: coin collector
Choices: (a) numismatist
(b) coin
(c) collector
(d) coinage
(e) aggregator
(f) reflectiveness
(g) plasma
Solution: (a) numismatist

Table of results

Algorithm Reference for algorithm Reference for experiment Correct - 7-choices Correct - 14-choices
Vector addition Landauer and Dumais (1997) Turney (2012) 50.1 22.5
Element-wise multiplication 57.5 27.4
Dual-Space model 58.3 41.5
SuperSim 75.9 68.0
Holistic model 81.6 NA

References

Turney, P.D. (2012). Domain and function: A dual-space model of semantic relations and compositions, Journal of Artificial Intelligence Research (JAIR), 44, 533-585.

Turney, P.D. (2013), Distributional semantics beyond words: Supervised learning of analogy and paraphrase, Transactions of the Association for Computational Linguistics (TACL), 1, 353-366.