WordSimilarity-353 Test Collection (State of the art)
- WordSimilarity-353 Test Collection
- contains two sets of English word pairs along with human-assigned similarity judgements
- first set (set1) contains 153 word pairs along with their similarity scores assigned by 13 subjects
- second set (set2) contains 200 word pairs with similarity assessed by 16 subjects
- WordSimilarity-353 dataset is available here
- performance is measured by Spearman's rank correlation coefficient
- introduced by Finkelstein et al. (2002)
- subsequently used by many other researchers
- Wikidata and Scholia
- see also: Similarity (State of the art)
Table of results
- Listed in order of increasing Spearman's rho.
| Algorithm | Reference for algorithm | Reference for reported results | Type | Spearman's rho | Pearson's r |
|---|---|---|---|---|---|
| L&C | Leacock and Chodorow (1998) | Hassan and Mihalcea (2011) | Knowledge-based | 0.302 | 0.356 |
| WNE | Jarmasz (2003) | Hassan and Mihalcea (2011) | Knowledge-based | 0.305 | 0.271 |
| J&C | Jiang and Conrath 1997 | Hassan and Mihalcea (2011) | Knowledge-based | 0.318 | 0.354 |
| L&C | Leacock and Chodorow (1998) | Hassan and Mihalcea (2011) | Knowledge-based | 0.348 | 0.341 |
| H&S | Hirst and St-Onge (1998) | Hassan and Mihalcea (2011) | Knowledge-based | 0.302 | 0.356 |
| Lin | Lin (1998) | Hassan and Mihalcea (2011) | Corpus-based | 0.348 | 0.357 |
| Resnik | Resnik (1995) | Hassan and Mihalcea (2011) | Knowledge-based | 0.353 | 0.365 |
| ROGET | Jarmasz (2003) | Hassan and Mihalcea (2011) | Knowledge-based | 0.415 | 0.536 |
| C&W | Collobert and Weston (2008) | Collobert and Weston (2008) | Corpus-based | 0.5 | N/A |
| WikiRelate | Strube and Ponzetto (2006) | Strube and Ponzetto (2006) | Corpus-based | N/A | 0.48 |
| Do19-corpus | Dobó (2019) | Dobó (2019) | Corpus-based | 0.579 | 0.577 |
| LSA | Landauer et al. (1997) | Hassan and Mihalcea (2011) | Corpus-based | 0.581 | 0.492 |
| LSA | Landauer et al. (1997) | Hassan and Mihalcea (2011) | Corpus-based | 0.581 | 0.563 |
| simVB+simWN | Finkelstein et al. (2002) | Finkelstein et al. (2002) | Hybrid | N/A | 0.55 |
| SSA | Hassan and Mihalcea (2011) | Hassan and Mihalcea (2011) | Knowledge-based | 0.622 | 0.629 |
| HSMN+csmRNN | Luong et al. (2013) | Luong et al. (2013) | Corpus-based | 0.65 | N/A |
| Pe14 | Pennington (2014) | Dobó (2019) | Corpus-based | 0.706 | 0.705 |
| Multi-prototype | Huang et al. (2012) | Huang et al. (2012) | Corpus-based | 0.71 | N/A |
| Multi-lingual SSA | Hassan et al. (2011) | Hassan et al. (2011) | Corpus-based | 0.713 | 0.674 |
| DSG | Song et al. (2018) | Song et al. (2018) | Corpus-based | 0.726 | N/A |
| Sa18 | Salle et al. (2018) | Dobó (2019) | Corpus-based | 0.733 | 0.704 |
| ESA | Gabrilovich and Markovitch (2007) | Gabrilovich and Markovitch (2007) | Corpus-based | 0.748 | 0.503 |
| Do19-hybrid | Dobó (2019) | Dobó (2019) | Hybrid | 0.795 | 0.276 |
| TSA | Radinsky et al. (2011) | Radinsky et al. (2011) | Hybrid | 0.80 | N/A |
| CLEAR | Halawi et al. (2012) | Halawi et al. (2012) | Corpus-based | 0.81 | N/A |
| Y&Q | Yih and Qazvinian (2012) | Yih and Qazvinian (2012) | Hybrid | 0.81 | N/A |
| ConceptNet Numberbatch | Speer et al. (2017) | Speer et al. (2017) | Hybrid | 0.828 | N/A |
References
- Listed in alphabetical order.
Dobó, A. (2019). A comprehensive analysis of the parameters in the creation and comparison of feature vectors in distributional semantic models for multiple languages. University of Szeged. GitHub repository
Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. (2002) Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems, 20(1):116-131.
Gabrilovich, Evgeniy, and Shaul Markovitch, Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis, Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India, 2007.
Halawi, Guy, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. (2012). Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1406-1414. ACM.
Hassan, Samer, and Rada Mihalcea: Semantic Relatedness Using Salient Semantic Analysis. AAAI 2011
Hirst, Graeme and David St-Onge. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 305–332, 1998.
Huang, Eric H., Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1 (ACL '12), Vol. 1. Association for Computational Linguistics, Stroudsburg, PA, USA, 873-882.
Islam, A., and Inkpen, D. 2006. Second order co-occurrence pmi for determining the semantic similarity of words. Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006) 1033–1038.
Jarmasz, M. 2003. Roget’s thesaurus as a Lexical Resource for Natural Language Processing. Ph.D. Dissertation, Ottawa Carleton Institute for Computer Science, School of Information Technology and Engineering, University of Ottawa.
Jiang, Jay J. and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics (ROCLING X), Taiwan, pages 19–33, 1997.
Landauer, T. K.; L, T. K.; Laham, D.; Rehder, B.; and Schreiner, M. E. 1997. How well can passage meaning be derived without using word order? a comparison of latent semantic analysis and humans.
Leacock, Claudia and Martin Chodorow. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 265–283, 1998.
Lin, Dekang. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison,WI, pages 296–304, 1998.
Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. (2013). Better word representations with recursive neural networks for morphology. CoNLL-2013: 104.
Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. EMNLP 2014, pp. 1532-1543.
Pilehvar, M.T., D. Jurgens and R. Navigli. Align, Disambiguate and Walk: A Unified Approach for Measuring Semantic Similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), Sofia, Bulgaria, August 4-9, 2013, pp. 1341-1351.
Radinsky, Kira, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. (2011). A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pp. 337-346. ACM.
Resnik, Philip. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453, Montreal, Canada, 1995.
Salle A., Idiart M., and Villavicencio A. (2018) LexVec
Speer, Rob, Joshua Chin and Catherine Havasi. (2017). ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. Proceedings of The 31st AAAI Conference on Artificial Intelligence, San Francisco, CA.
Strube, Michael and Simone Paolo Ponzetto. (2006). WikiRelate! Computing Semantic Relatedness Using Wikipedia. Proceedings of The 21st National Conference on Artificial Intelligence (AAAI), Boston, MA.
Yan Song, Shuming Shi, Jing Li, and Haisong Zhang.2018. Directional skip-gram: Explicitly distinguish-ing left and right context for word embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 175–180.
Yih, W. and Qazvinian, V. (2012). Measuring Word Relatedness Using Heterogeneous Vector Space Models. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2012).