MC-28 Test Collection (State of the art)
- state of the art in Miller & Charles 28 (MC-28) dataset [Resnik, 1995]
- 28 word pairs of the original Miller & Charles 30 (MC-30) dataset [Miller and Charles, 1991], which is a subset of the Rubenstein & Goodenough (RG-65) dataset; two word pairs have generally been omitted for semantic similarity evaluation, as words in these word pairs have not been included in previous versions of WordNet
- Similarity of each pair is scored according to a scale from 0 to 4 (the higher the "similarity of meaning," the higher the number);
- The similarity values in the dataset are the means of judgments made by 38 subjects [Miller and Charles, 1991].
- see also: Similarity (State of the art)
Table of results
- Listed in order of decreasing Spearman's rho.
| Algorithm | Reference for algorithm | Reference for reported results | Type | Spearman correlation [with 95% confidence intervals] |
|---|---|---|---|---|
| Human | Human upper bound | Resnik (1995) | Human | 0.934 [0.861, 0.969] |
| PPR | Agirre et al. (2009) | Agirre et al. (2009) | Hybrid | 0.92 [0.833, 0.962] |
| Gloss Vector | Patwardhan and Pedersen (2006) | Patwardhan and Pedersen (2006) | Lexicon-based | 0.91 [0.813, 0.957] |
| Do19-hybrid | Dobó (2019) | Dobó (2019) | Hybrid | 0.893 [0.780, 0.949] |
| Sp17 | Speer et al. (2017) | Dobó (2019) | Hybrid | 0.892 [0.778, 0.949] |
| JS | Jarmasz and Szpakowicz (2003) | Jarmasz and Szpakowicz (2003) | Lexicon-based | 0.87 [0.736, 0.938] |
| SR | Tsatsaronis et al. (2010) | Tsatsaronis et al. (2010) | Lexicon-based | 0.856 [0.710, 0.931] |
| Do19-corpus | Dobó (2019) | Dobó (2019) | Corpus-based | 0.853 [0.704, 0.930] |
| KC | Kulkarni and Caragea (2009) | Kulkarni and Caragea (2009) | Web-based | 0.835 [0.671, 0.921] |
| Pe14 | Pennington et al. (2014) | Dobó (2019) | Corpus-based | 0.832 [0.666, 0.919] |
| Sa18 | Salle et al. (2018) | Dobó (2019) | Corpus-based | 0.822 [0.648, 0.914] |
| LIN | Lin (1998) | Lin (1998) | Hybrid | 0.82 [0.644, 0.913] |
| RES | Resnik (1995) | Resnik (1995) | Hybrid | 0.81 [0.627, 0.908] |
| DC13 | Dobó and Csirik (2013) | Dobó and Csirik (2013) | Corpus-based | 0.773 [0.562, 0.889] |
| GM | Gabrilovich and Markovitch (2007) | Tsatsaronis et al. (2010) | Corpus-based | 0.72 [0.475, 0.861] |
| WLM | Milne and Witten (2008) | Milne and Witten (2008) | Web-based | 0.70 [0.443, 0.850] |
| SH | Sahami and Heilman (2006) | Agirre et al. (2009) | Web-based | 0.618 [0.319, 0.805] |
References
- Listed alphabetically.
Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Paşca, M., Soroa, A. (2009). A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches. In: 10th Annual Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies. Association for Computa-tional Linguistics, Stroudsburg. pp. 19–27.
Dobó, A. (2019). A comprehensive analysis of the parameters in the creation and comparison of feature vectors in distributional semantic models for multiple languages. University of Szeged. GitHub repository
Dobó, A., and Csirik, J. (2013). Computing semantic similarity using large static corpora. In: van Emde Boas, P. et al. (eds.) SOFSEM 2013: Theory and Practice of Computer Science. LNCS, Vol. 7741. Springer-Verlag, Berlin Heidelberg, pp. 491-502.
Gabrilovich, E., and Markovitch, S. (2007). Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis, Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India.
Jarmasz, M., and Szpakowicz, S. (2003). Roget’s thesaurus and semantic similarity, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03), Borovets, Bulgaria, September, pp. 212-219.
Kulkarni, S., Caragea, D. (2009). Computation of the Semantic Relatedness between Words using Concept Clouds. In: International Conference on Knowledge Discovery and Information Re-trieval. INSTICC Press, Setubal. pp. 183–188.
Lin, D. (1998). An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison,WI, pp. 296–304.
Miller, G., and Charles, W. (1991). Contextual correlates of semantic similarity. Language and Cognitive Processes, 6(1), 1–28.
Milne, D., Witten, I.H. (2008). An Effective, Low-Cost Measure of Semantic Relatedness Obtained from Wikipedia Links, In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: an Evolving Synergy, AAAI Press, Chicago, USA pp. 25-30.
Patwardhan, S., and Pedersen, T. (2006). Using WordNet-based Context Vectors to Estimate the Se-mantic Relatedness of Concepts. In: 11th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Stroudsbur. pp. 1–8.
Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. EMNLP 2014, pp. 1532-1543.
Resnik, P. (1995). Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, Montreal, Canada, pages 448–453.
Sahami, M., Heilman, T.D. (2006). A web-based kernel function for measuring the similarity of short text snippets. In: 15th international conference on World Wide Web. ACM Press, New York. pp. 377–386.
Salle A., Idiart M., and Villavicencio A. (2018) LexVec
Speer, R., Chin, J., and Havasi, C. (2017). Conceptnet 5.5: An open multilingual graph of general knowledge. AAAI-17, pp. 4444-4451.
Tsatsaronis, G., Varlamis, I., and Vazirgiannis, M. (2010). Text Relatedness Based on a Word Thesaurus. Journal of Artificial Intelligence Research 37, 1–39