Difference between revisions of "SAT Analogy Questions (State of the art)"
| Line 1: | Line 1: | ||
| − | * SAT= Scholastic Aptitude Test | + | * SAT = Scholastic Aptitude Test |
* 374 multiple-choice analogy questions; 5 choices per question | * 374 multiple-choice analogy questions; 5 choices per question | ||
* SAT questions collected by [http://www.cs.rutgers.edu/~mlittman/ Michael Littman], available from [http://www.apperceptual.com/ Peter Turney] | * SAT questions collected by [http://www.cs.rutgers.edu/~mlittman/ Michael Littman], available from [http://www.apperceptual.com/ Peter Turney] | ||
| Line 10: | Line 10: | ||
* '''95% confidence''' = confidence interval calculated using [http://home.clara.net/sisa/onemean.htm Binomial Exact Test] | * '''95% confidence''' = confidence interval calculated using [http://home.clara.net/sisa/onemean.htm Binomial Exact Test] | ||
* table rows sorted in order of increasing percent correct | * table rows sorted in order of increasing percent correct | ||
| + | * several WordNet-based similarity measures are implemented in [http://www.d.umn.edu/~tpederse/ Ted Pedersen]'s [http://www.d.umn.edu/~tpederse/similarity.html WordNet::Similarity] package | ||
| + | * KNOW-BEST = KNOWledge-Based Entertainment and Scholastic Testing | ||
* VSM = Vector Space Model | * VSM = Vector Space Model | ||
* LRA = Latent Relational Analysis | * LRA = Latent Relational Analysis | ||
| + | * PERT = Pertinence | ||
| Line 22: | Line 25: | ||
! Correct | ! Correct | ||
! 95% confidence | ! 95% confidence | ||
| + | |- | ||
| + | | JC | ||
| + | | Jiang and Conrath (1997) | ||
| + | | Turney (2006b) | ||
| + | | Hybrid | ||
| + | | 27.3% | ||
| + | | 23.1-32.4% | ||
|- | |- | ||
| HSO | | HSO | ||
| Line 61: | Line 71: | ||
Hirst, G., and St-Onge, D. (1998). [http://mirror.eacoss.org/documentation/ITLibrary/IRIS/Data/1997/Hirst/Lexical/1997-Hirst-Lexical.pdf Lexical chains as representation of context for the detection and correction of malapropisms]. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, 305-332. | Hirst, G., and St-Onge, D. (1998). [http://mirror.eacoss.org/documentation/ITLibrary/IRIS/Data/1997/Hirst/Lexical/1997-Hirst-Lexical.pdf Lexical chains as representation of context for the detection and correction of malapropisms]. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, 305-332. | ||
| + | |||
| + | Jiang, J.J., and Conrath, D.W. (1997). [http://wortschatz.uni-leipzig.de/~sbordag/aalw05/Referate/03_Assoziationen_BudanitskyResnik/Jiang_Conrath_97.pdf Semantic similarity based on corpus statistics and lexical taxonomy]. ''Proceedings of the International Conference on Research in Computational Linguistics'', Taiwan. | ||
Turney, P.D., Littman, M.L., Bigham, J., and Shnayder, V. (2003). [http://arxiv.org/abs/cs.CL/0309035 Combining independent modules to solve multiple-choice synonym and analogy problems]. ''Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03)'', Borovets, Bulgaria, pp. 482-489. | Turney, P.D., Littman, M.L., Bigham, J., and Shnayder, V. (2003). [http://arxiv.org/abs/cs.CL/0309035 Combining independent modules to solve multiple-choice synonym and analogy problems]. ''Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03)'', Borovets, Bulgaria, pp. 482-489. | ||
Revision as of 05:41, 13 May 2007
- SAT = Scholastic Aptitude Test
- 374 multiple-choice analogy questions; 5 choices per question
- SAT questions collected by Michael Littman, available from Peter Turney
- introduced in Turney et al. (2003) as a way of evaluating algorithms for measuring relational similarity
- Algorithm = name of algorithm
- Reference for algorithm = where to find out more about given algorithm
- Reference for experiment = where to find out more about evaluation of given algorithm with SAT questions
- Type = general type of algorithm: corpus-based, lexicon-based, hybrid
- Correct = percent of 374 questions that given algorithm answered correctly
- 95% confidence = confidence interval calculated using Binomial Exact Test
- table rows sorted in order of increasing percent correct
- several WordNet-based similarity measures are implemented in Ted Pedersen's WordNet::Similarity package
- KNOW-BEST = KNOWledge-Based Entertainment and Scholastic Testing
- VSM = Vector Space Model
- LRA = Latent Relational Analysis
- PERT = Pertinence
| Algorithm | Reference for algorithm | Reference for experiment | Type | Correct | 95% confidence |
|---|---|---|---|---|---|
| JC | Jiang and Conrath (1997) | Turney (2006b) | Hybrid | 27.3% | 23.1-32.4% |
| HSO | Hirst and St.-Onge (1998) | Turney (2006b) | Lexicon-based | 32.1% | 27.6-37.4% |
| KNOW-BEST | Veale (2004) | Veale (2004) | Lexicon-based | 43.0% | 38.0-48.2% |
| VSM | Turney and Littman (2005) | Turney and Littman (2005) | Corpus-based | 47.1% | 42.2-52.5% |
| PERT | Turney (2006a) | Turney (2006a) | Corpus-based | 53.5% | 48.5-58.9% |
| LRA | Turney (2006b) | Turney (2006b) | Corpus-based | 56.1% | 51.0–61.2% |
Hirst, G., and St-Onge, D. (1998). Lexical chains as representation of context for the detection and correction of malapropisms. In C. Fellbaum (ed.), WordNet: An Electronic Lexical Database. Cambridge: MIT Press, 305-332.
Jiang, J.J., and Conrath, D.W. (1997). Semantic similarity based on corpus statistics and lexical taxonomy. Proceedings of the International Conference on Research in Computational Linguistics, Taiwan.
Turney, P.D., Littman, M.L., Bigham, J., and Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), Borovets, Bulgaria, pp. 482-489.
Turney, P.D., and Littman, M.L. (2005). Corpus-based learning of analogies and semantic relations. Machine Learning, 60 (1-3), 251-278.
Turney, P.D. (2006a). Expressing implicit semantic relations without supervision. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics (Coling/ACL-06), Sydney, Australia, pp. 313-320.
Turney, P.D. (2006b). Similarity of semantic relations. Computational Linguistics, 32 (3), 379-416.
Veale, T. (2004). WordNet sits the SAT: A knowledge-based approach to lexical analogy. Proceedings of the 16th European Conference on Artificial Intelligence (ECAI 2004), pp. 606–612, Valencia, Spain.