SAT Analogy Questions (State of the art)

• SAT= Scholastic Aptitude Test
• 374 multiple-choice analogy questions; 5 choices per question
• SAT questions collected by Michael Littman, available from Peter Turney
• introduced in Turney et al. (2003) as a way of evaluating algorithms for measuring relational similarity
• Algorithm = name of algorithm
• Reference = source for algorithm description and experimental results
• Type = general type of algorithm: corpus-based, lexicon-based, hybrid
• Correct = percent of 374 questions that given algorithm answered correctly
• 95% confidence = confidence interval calculated using Binomial Exact Test
• table rows sorted in order of increasing percent correct
• VSM = Vector Space Model
• LRA = Latent Relational Analysis

Algorithm	Reference	Type	Correct	95% confidence
KNOW-BEST	Veale (2004)	lexicon-based	43.0%	38.0-48.2%
VSM	Turney and Littman (2005)	corpus-based	47.1%	42.2-52.5%
LRA	Turney (2006)	corpus-based	56.1%	51.0–61.2%

Turney, P.D., Littman, M.L., Bigham, J., and Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), Borovets, Bulgaria, pp. 482-489.

Turney, P.D., and Littman, M.L. (2005). Corpus-based learning of analogies and semantic relations. Machine Learning, 60 (1-3), 251-278.

Turney, P.D. (2006). Similarity of semantic relations. Computational Linguistics, 32 (3), 379-416.

Veale, T. (2004). WordNet sits the SAT: A knowledge-based approach to lexical analogy. Proceedings of the 16th European Conference on Artificial Intelligence (ECAI 2004), pp. 606–612, Valencia, Spain.