Difference between revisions of "RG-65 Test Collection (State of the art)"

From ACL Wiki
Jump to navigation Jump to search
(10 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
State of the art in Rubenstein & Goodenough (RG-65) dataset
+
* state of the art in Rubenstein & Goodenough (RG-65) dataset
−
 
 
 
* 65 word pairs;  
 
* 65 word pairs;  
 
* Similarity of each pair is scored according to a scale from 0 to 4 (the higher the "similarity of meaning," the higher the number);
 
* Similarity of each pair is scored according to a scale from 0 to 4 (the higher the "similarity of meaning," the higher the number);
 
* The similarity values in the dataset are the means of judgments made by 51 subjects [Rubenstein and Goodenough, 1965].
 
* The similarity values in the dataset are the means of judgments made by 51 subjects [Rubenstein and Goodenough, 1965].
 +
* see also: [[Similarity (State of the art)]]
 +
  
 
== Table of results ==
 
== Table of results ==
 +
 +
* '''Listed in order of decreasing [http://en.wikipedia.org/wiki/Spearman_rank_correlation Spearman's rho].'''
 +
  
 
{| border="1" cellpadding="5" cellspacing="1" width="100%"
 
{| border="1" cellpadding="5" cellspacing="1" width="100%"
Line 13: Line 17:
 
! Reference for reported results
 
! Reference for reported results
 
! Type
 
! Type
−
! Spearman correlation (ρ)
+
! [http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient Spearman correlation] (ρ)
−
! Pearson correlation (r)
+
! [http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient Pearson correlation] (r)
 +
|-
 +
| Y&Q
 +
| Yih and Qazvinian (2012)
 +
| Yih and Qazvinian (2012)
 +
| Hybrid
 +
| 0.890
 +
| -
 
|-
 
|-
 
| ADW
 
| ADW
Line 23: Line 34:
 
| 0.810
 
| 0.810
 
|-
 
|-
−
| Roget
+
| PPR
−
| Jarmasz (2003)
+
| Hughes and Ramage (2007)
−
| Hassan and Mihalcea (2011)
+
| Hughes and Ramage (2007)
 
| Knowledge-based
 
| Knowledge-based
−
| 0.804
+
| 0.838
−
| 0.818
+
| -
 
|-
 
|-
−
| WNE
+
| SSA
−
| Jarmasz (2003)
 
 
| Hassan and Mihalcea (2011)
 
| Hassan and Mihalcea (2011)
−
| Knowledge-based
 
−
| 0.801
 
−
| 0.787
 
−
|-
 
−
| ESA*
 
−
| Gabrilovich and Markovitch (2007)
 
−
| Hassan and Mihalcea (2011)
 
−
| Corpus-based
 
−
| 0.749
 
−
| 0.716
 
−
|-
 
−
| LSA*
 
−
| Landauer et al. (1997)
 
 
| Hassan and Mihalcea (2011)
 
| Hassan and Mihalcea (2011)
 
| Corpus-based
 
| Corpus-based
−
| 0.609
+
| 0.833
−
| 0.644
+
| 0.861
 
|-
 
|-
−
| SOCPMI*
+
| PPR
−
| Islam and Inkpen (2006)
+
| Agirre et al. (2009)
−
| Hassan and Mihalcea (2011)
+
| Agirre et al. (2009)
−
| Corpus-based
+
| Knowledge-based
−
| 0.741
+
| 0.830
−
| 0.729
+
| -
 
|-
 
|-
 
| H&S
 
| H&S
Line 64: Line 61:
 
| 0.813
 
| 0.813
 
| 0.732
 
| 0.732
 +
|-
 +
| Roget
 +
| Jarmasz (2003)
 +
| Hassan and Mihalcea (2011)
 +
| Knowledge-based
 +
| 0.804
 +
| 0.818
 
|-
 
|-
 
| J&C
 
| J&C
Line 71: Line 75:
 
| 0.804
 
| 0.804
 
| 0.731
 
| 0.731
 +
|-
 +
| WNE
 +
| Jarmasz (2003)
 +
| Hassan and Mihalcea (2011)
 +
| Knowledge-based
 +
| 0.801
 +
| 0.787
 
|-
 
|-
 
| L&C
 
| L&C
Line 85: Line 96:
 
| 0.788
 
| 0.788
 
| 0.834
 
| 0.834
 +
|-
 +
| ESA*
 +
| Gabrilovich and Markovitch (2007)
 +
| Hassan and Mihalcea (2011)
 +
| Corpus-based
 +
| 0.749
 +
| 0.716
 +
|-
 +
| SOCPMI*
 +
| Islam and Inkpen (2006)
 +
| Hassan and Mihalcea (2011)
 +
| Corpus-based
 +
| 0.741
 +
| 0.729
 
|-
 
|-
 
| Resnik
 
| Resnik
Line 92: Line 117:
 
| 0.731
 
| 0.731
 
| 0.800
 
| 0.800
−
|-
 
−
| WikiRelate
 
−
| Strube and Ponzetto (2006)
 
−
| Strube and Ponzetto (2006)
 
−
| Knowledge-based
 
−
| -
 
−
| 0.530
 
−
|-
 
−
| PPR
 
−
| Agirre et al. (2009)
 
−
| Agirre et al. (2009)
 
−
| Knowledge-based
 
−
| 0.830
 
−
| -
 
 
|-
 
|-
 
| WLM
 
| WLM
Line 114: Line 125:
 
| -
 
| -
 
|-
 
|-
−
| PPR
+
| LSA*
−
| Hughes and Ramage (2007)
+
| Landauer et al. (1997)
−
| Hughes and Ramage (2007)
+
| Hassan and Mihalcea (2011)
 +
| Corpus-based
 +
| 0.609
 +
| 0.644
 +
|-
 +
| WikiRelate
 +
| Strube and Ponzetto (2006)
 +
| Strube and Ponzetto (2006)
 
| Knowledge-based
 
| Knowledge-based
−
| 0.838
 
 
| -
 
| -
 +
| 0.530
 
|}
 
|}
  
Line 126: Line 144:
 
== References ==
 
== References ==
  
−
* Herbert Rubenstein and John B. Goodenough. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633, 1965.
+
* '''Listed alphabetically.'''
 +
 
 +
 
 +
Agirre, Eneko, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, Aitor Soroa: [http://www.aclweb.org/anthology/N09-1003 A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches]. HLT-NAACL 2009: 19-27
 +
 
 +
Gabrilovich, Evgeniy, and Shaul Markovitch, [http://www.cs.technion.ac.il/~gabr/papers/ijcai-2007-sim.pdf Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis], Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India, 2007.
 +
 
 +
Hassan, Samer, and Rada Mihalcea: [http://www.cse.unt.edu/~rada/papers/hassan.aaai11.pdf‎ Semantic Relatedness Using Salient Semantic Analysis]. AAAI 2011
 +
 
 +
Hirst, Graeme and David St-Onge. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 305–332, 1998.
  
−
* Samer Hassan, Rada Mihalcea: Semantic Relatedness Using Salient Semantic Analysis. AAAI 2011
+
Hughes, Thad, Daniel Ramage, Lexical Semantic Relatedness with Random Graph Walks. EMNLP-CoNLL 2007: 581-589.
  
−
* Lin, Dekang. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison,WI, pages 296–304, 1998.
+
Islam, A., and Inkpen, D. 2006. [http://www.site.uottawa.ca/~mdislam/publications/LREC_06_242.pdf Second order co-occurrence pmi for determining the semantic similarity of words]. Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006) 1033–1038.
  
−
* Lin, Dekang. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics (COLING–ACL ’98), Montreal, Canada, pages 768–774, 1998.
+
Jarmasz, M. 2003. [http://www.arxiv.org/pdf/1204.0140 Roget’s thesaurus as a Lexical Resource for Natural Language Processing]. Ph.D. Dissertation, Ottawa Carleton Institute for Computer Science, School of Information Technology and Engineering, University of Ottawa.
  
−
* Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, Aitor Soroa: A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches. HLT-NAACL 2009: 19-27
+
Jiang, Jay J. and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics (ROCLING X), Taiwan, pages 19–33, 1997.
  
−
* Hirst, Graeme and David St-Onge. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 305–332, 1998.
+
Landauer, T. K.; L, T. K.; Laham, D.; Rehder, B.; and Schreiner, M. E. 1997. How well can passage meaning be derived without using word order? a comparison of latent semantic analysis and humans.
  
−
* Thad Hughes, Daniel Ramage: Lexical Semantic Relatedness with Random Graph Walks. EMNLP-CoNLL 2007: 581-589.
+
Leacock, Claudia and Martin Chodorow. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 265–283, 1998.
  
−
* Jiang, Jay J. and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics (ROCLING X), Taiwan, pages 19–33, 1997.
+
Lin, Dekang. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison,WI, pages 296–304, 1998.
  
−
* Leacock, Claudia and Martin Chodorow. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 265–283, 1998.
+
Milne, David, and Ian H. Witten, An Effective, Low-Cost Measure of Semantic Relatedness Obtained from Wikipedia Links, In Proceedings of AAAI 2008.
  
−
* Resnik, Philip. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453, Montreal, Canada, 1995.
+
Pilehvar, M.T., D. Jurgens and R. Navigli. [http://wwwusers.di.uniroma1.it/~navigli/pubs/ACL_2013_Pilehvar_Jurgens_Navigli.pdf Align, Disambiguate and Walk: A Unified Approach for Measuring Semantic Similarity]. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), Sofia, Bulgaria, August 4-9, 2013, pp. 1341-1351.
  
−
* Jarmasz, M. 2003. Roget’s thesaurus as a Lexical Resource for Natural Language Processing. Ph.D. Dissertation, Ottawa Carleton Institute for Computer Science, School of Information Technology and Engineering, University of Ottawa.
+
Resnik, Philip. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453, Montreal, Canada, 1995.
  
−
* Landauer, T. K.; L, T. K.; Laham, D.; Rehder, B.; and Schreiner, M. E. 1997. How well can passage meaning be derived without using word order? a comparison of latent semantic analysis and humans.
+
Rubenstein, Herbert, and John B. Goodenough. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633, 1965.
  
−
* Islam, A., and Inkpen, D. 2006. Second order co-occurrence pmi for determining the semantic similarity of words. Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006) 1033–1038.
+
Strube, Michael, Simone Paolo Ponzetto: WikiRelate! Computing Semantic Relatedness Using Wikipedia. AAAI 2006: 1419-1424
  
−
* M. T. Pilehvar, D. Jurgens and R. Navigli. Align, Disambiguate and Walk: A Unified Approach for Measuring Semantic Similarity. Proc. of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), Sofia, Bulgaria, August 4-9, 2013, pp. 1341-1351.
+
Yih, W. and Qazvinian, V. (2012). [http://aclweb.org/anthology/N/N12/N12-1077.pdf Measuring Word Relatedness Using Heterogeneous Vector Space Models]. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2012).
  
−
* Michael Strube, Simone Paolo Ponzetto: WikiRelate! Computing Semantic Relatedness Using Wikipedia. AAAI 2006: 1419-1424
+
[[Category:State of the art]]

Revision as of 13:27, 15 January 2014

  • state of the art in Rubenstein & Goodenough (RG-65) dataset
  • 65 word pairs;
  • Similarity of each pair is scored according to a scale from 0 to 4 (the higher the "similarity of meaning," the higher the number);
  • The similarity values in the dataset are the means of judgments made by 51 subjects [Rubenstein and Goodenough, 1965].
  • see also: Similarity (State of the art)


Table of results


Algorithm Reference for algorithm Reference for reported results Type Spearman correlation (ρ) Pearson correlation (r)
Y&Q Yih and Qazvinian (2012) Yih and Qazvinian (2012) Hybrid 0.890 -
ADW Pilehvar et al. (2013) Pilehvar et al. (2013) Knowledge-based 0.868 0.810
PPR Hughes and Ramage (2007) Hughes and Ramage (2007) Knowledge-based 0.838 -
SSA Hassan and Mihalcea (2011) Hassan and Mihalcea (2011) Corpus-based 0.833 0.861
PPR Agirre et al. (2009) Agirre et al. (2009) Knowledge-based 0.830 -
H&S Hirst and St-Onge (1998) Hassan and Mihalcea (2011) Knowledge-based 0.813 0.732
Roget Jarmasz (2003) Hassan and Mihalcea (2011) Knowledge-based 0.804 0.818
J&C Jiang and Conrath (1997) Hassan and Mihalcea (2011) Knowledge-based 0.804 0.731
WNE Jarmasz (2003) Hassan and Mihalcea (2011) Knowledge-based 0.801 0.787
L&C Leacock and Chodorow (1998) Hassan and Mihalcea (2011) Knowledge-based 0.797 0.852
Lin Lin (1998) Hassan and Mihalcea (2011) Corpus-based 0.788 0.834
ESA* Gabrilovich and Markovitch (2007) Hassan and Mihalcea (2011) Corpus-based 0.749 0.716
SOCPMI* Islam and Inkpen (2006) Hassan and Mihalcea (2011) Corpus-based 0.741 0.729
Resnik Resnik (1995) Hassan and Mihalcea (2011) Knowledge-based 0.731 0.800
WLM Milne and Witten (2008) Milne and Witten (2008) Knowledge-based 0.640 -
LSA* Landauer et al. (1997) Hassan and Mihalcea (2011) Corpus-based 0.609 0.644
WikiRelate Strube and Ponzetto (2006) Strube and Ponzetto (2006) Knowledge-based - 0.530

Note: values reported by (Hassan and Mihalcea, 2011) are "based on the collected raw data from the respective authors", and those highlighted by (*) are re-implementations.

References

  • Listed alphabetically.


Agirre, Eneko, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, Aitor Soroa: A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches. HLT-NAACL 2009: 19-27

Gabrilovich, Evgeniy, and Shaul Markovitch, Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis, Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India, 2007.

Hassan, Samer, and Rada Mihalcea: Semantic Relatedness Using Salient Semantic Analysis. AAAI 2011

Hirst, Graeme and David St-Onge. Lexical chains as representations of context for the detection and correction of malapropisms. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 305–332, 1998.

Hughes, Thad, Daniel Ramage, Lexical Semantic Relatedness with Random Graph Walks. EMNLP-CoNLL 2007: 581-589.

Islam, A., and Inkpen, D. 2006. Second order co-occurrence pmi for determining the semantic similarity of words. Proceedings of the International Conference on Language Resources and Evaluation (LREC 2006) 1033–1038.

Jarmasz, M. 2003. Roget’s thesaurus as a Lexical Resource for Natural Language Processing. Ph.D. Dissertation, Ottawa Carleton Institute for Computer Science, School of Information Technology and Engineering, University of Ottawa.

Jiang, Jay J. and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference on Research in Computational Linguistics (ROCLING X), Taiwan, pages 19–33, 1997.

Landauer, T. K.; L, T. K.; Laham, D.; Rehder, B.; and Schreiner, M. E. 1997. How well can passage meaning be derived without using word order? a comparison of latent semantic analysis and humans.

Leacock, Claudia and Martin Chodorow. Combining local context and WordNet similarity for word sense identification. In Christiane Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, pages 265–283, 1998.

Lin, Dekang. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison,WI, pages 296–304, 1998.

Milne, David, and Ian H. Witten, An Effective, Low-Cost Measure of Semantic Relatedness Obtained from Wikipedia Links, In Proceedings of AAAI 2008.

Pilehvar, M.T., D. Jurgens and R. Navigli. Align, Disambiguate and Walk: A Unified Approach for Measuring Semantic Similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), Sofia, Bulgaria, August 4-9, 2013, pp. 1341-1351.

Resnik, Philip. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pages 448–453, Montreal, Canada, 1995.

Rubenstein, Herbert, and John B. Goodenough. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633, 1965.

Strube, Michael, Simone Paolo Ponzetto: WikiRelate! Computing Semantic Relatedness Using Wikipedia. AAAI 2006: 1419-1424

Yih, W. and Qazvinian, V. (2012). Measuring Word Relatedness Using Heterogeneous Vector Space Models. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2012).

Retrieved from "https://aclweb.org/aclwiki/index.php?title=RG-65_Test_Collection_(State_of_the_art)&oldid=10514"