SAT Analogy Questions (State of the art)

From ACL Wiki
Revision as of 04:57, 13 May 2007 by Pdturney (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search
  • SAT= Scholastic Aptitude Test
  • 374 multiple-choice analogy questions; 5 choices per question
  • SAT questions collected by Michael Littman, available from Peter Turney
  • introduced in Turney et al. (2003) as a way of evaluating algorithms for measuring relational similarity
  • Algorithm = name of algorithm
  • Reference = source for algorithm description and experimental results
  • Type = general type of algorithm: corpus-based, lexicon-based, hybrid
  • Correct = percent of 374 questions that given algorithm answered correctly
  • 95% confidence = confidence interval calculated using Binomial Exact Test
  • table rows sorted in order of increasing percent correct
  • VSM = Vector Space Model
  • LRA = Latent Relational Analysis


Algorithm Reference Type Correct 95% confidence
VSM Turney and Littman (2005) corpus-based 47.1%  ?
LRA Turney (2006) corpus-based 56.1% 51.0–61.2%


Turney, P.D., Littman, M.L., Bigham, J., and Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), Borovets, Bulgaria, pp. 482-489.

Turney, P.D., and Littman, M.L. (2005). Corpus-based learning of analogies and semantic relations. Machine Learning, 60 (1-3), 251-278.

Turney, P.D. (2006). Similarity of semantic relations. Computational Linguistics, 32 (3), 379-416.