RG-65 Test Collection (State of the art)

State of the art in Rubenstein & Goodenough (RG-65) dataset

• 65 word pairs;
• Similarity of each pair is scored according to a scale from 0 to 4 (the higher the "similarity of meaning," the higher the number);
• The similarity values in the dataset are the means of judgments made by 51 subjects [Rubenstein and Goodenough, 1965].

Table of results

Algorithm	Reference for algorithm	Reference for reported results	Type	Spearman correlation (ρ)	Pearson correlation (r)
ADW	Pilehvar et al. (2013)	Pilehvar et al. (2013)	Knowledge-based	0.868	0.810
Roget	Jarmasz (2003)	Hassan and Mihalcea (2011)	Knowledge-based	0.804	0.818
WNE	Jarmasz (2003)	Hassan and Mihalcea (2011)	Knowledge-based	0.801	0.787
ESA*	Gabrilovich and Markovitch (2007)	Hassan and Mihalcea (2011)	Corpus-based	0.749	0.716
LSA*	Landauer et al. (1997)	Hassan and Mihalcea (2011)	Corpus-based	0.609	0.644
SOCPMI*	Islam and Inkpen (2006)	Hassan and Mihalcea (2011)	Corpus-based	0.741	0.729
H&S	Hirst and St-Onge (1998)	Hassan and Mihalcea (2011)	Knowledge-based	0.813	0.732
J&C	Jiang and Conrath (1997)	Hassan and Mihalcea (2011)	Knowledge-based	0.804	0.731
L&C	Leacock and Chodorow (1998)	Hassan and Mihalcea (2011)	Knowledge-based	0.797	0.852
Lin	Lin (1998)	Hassan and Mihalcea (2011)	Corpus-based	0.788	0.834
Resnik	Resnik (1995)	Hassan and Mihalcea (2011)	Knowledge-based	0.731	0.800
WikiRelate	Strube and Ponzetto (2006)	Strube and Ponzetto (2006)	Knowledge-based	-	0.530
PPR	Agirre et al. (2009)	Agirre et al. (2009)	Knowledge-based	0.830	-
WLM	Milne and Witten (2008)	Milne and Witten (2008)	Knowledge-based	0.640	-
PPR	Hughes and Ramage (2007)	Hughes and Ramage (2007)	Knowledge-based	0.838	-

Note: values reported by (Hassan and Mihalcea, 2011) are "based on the collected raw data from the respective authors", and those highlighted by (*) are re-implementations.

References

• Herbert Rubenstein and John B. Goodenough. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633, 1965.

• Samer Hassan, Rada Mihalcea: Semantic Relatedness Using Salient Semantic Analysis. AAAI 2011

• Lin, Dekang. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison,WI, pages 296–304, 1998.

• Lin, Dekang. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics (COLING–ACL ’98), Montreal, Canada, pages 768–774, 1998.

• Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, Aitor Soroa: A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches. HLT-NAACL 2009: 19-27

• Hirst, Graeme and David St-Onge. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 305–332, 1998.

• Thad Hughes, Daniel Ramage: Lexical Semantic Relatedness with Random Graph Walks. EMNLP-CoNLL 2007: 581-589.

• Jiang, Jay J. and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics (ROCLING X), Taiwan, pages 19–33, 1997.

• Leacock, Claudia and Martin Chodorow. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 265–283, 1998.

• Resnik, Philip. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453, Montreal, Canada, 1995.

• Jarmasz, M. 2003. Roget’s thesaurus as a Lexical Resource for Natural Language Processing. Ph.D. Dissertation, Ottawa Carleton Institute for Computer Science, School of Information Technology and Engineering, University of Ottawa.

• Landauer, T. K.; L, T. K.; Laham, D.; Rehder, B.; and Schreiner, M. E. 1997. How well can passage meaning be derived without using word order? a comparison of latent semantic analysis and humans.

• Islam, A., and Inkpen, D. 2006. Second order co-occurrence pmi for determining the semantic similarity of words. Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006) 1033–1038.

• M. T. Pilehvar, D. Jurgens and R. Navigli. Align, Disambiguate and Walk: A Unified Approach for Measuring Semantic Similarity. Proc. of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), Sofia, Bulgaria, August 4-9, 2013, pp. 1341-1351.

• Michael Strube, Simone Paolo Ponzetto: WikiRelate! Computing Semantic Relatedness Using Wikipedia. AAAI 2006: 1419-1424