SAT Analogy Questions (State of the art)
Jump to navigation
Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
- SAT= Scholastic Aptitude Test
- 374 multiple-choice analogy questions; 5 choices per question
- SAT questions collected by Michael Littman, available from Peter Turney
- introduced in Turney et al. (2003) as a way of evaluating algorithms for measuring relational similarity
- Algorithm = name of algorithm
- Reference = source for algorithm description and experimental results
- Type = general type of algorithm: corpus-based, lexicon-based, hybrid
- Correct = percent of 374 questions that given algorithm answered correctly
- 95% confidence = confidence interval calculated using Binomial Exact Test
- table rows sorted in order of increasing percent correct
- VSM = Vector Space Model
- LRA = Latent Relational Analysis
| Algorithm | Reference | Type | Correct | 95% confidence |
|---|---|---|---|---|
| KNOW-BEST | Veale (2004) | lexicon-based | 43.0% | 38.0-48.2% |
| VSM | Turney and Littman (2005) | corpus-based | 47.1% | 42.2-52.5% |
| LRA | Turney (2006) | corpus-based | 56.1% | 51.0–61.2% |
Turney, P.D., Littman, M.L., Bigham, J., and Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), Borovets, Bulgaria, pp. 482-489.
Turney, P.D., and Littman, M.L. (2005). Corpus-based learning of analogies and semantic relations. Machine Learning, 60 (1-3), 251-278.
Turney, P.D. (2006). Similarity of semantic relations. Computational Linguistics, 32 (3), 379-416.
Veale, T. (2004). WordNet sits the SAT: A knowledge-based approach to lexical analogy. Proceedings of the 16th European Conference on Artificial Intelligence (ECAI 2004), pp. 606–612, Valencia, Spain.