WordSimilarity-353 Test Collection (State of the art)
- WordSimilarity-353 Test Collection
- contains two sets of English word pairs along with human-assigned similarity judgements
- first set (set1) contains 153 word pairs along with their similarity scores assigned by 13 subjects
- second set (set2) contains 200 word pairs with similarity assessed by 16 subjects
- WordSimilarity-353 dataset is available here
- performance is measured by Spearman's rank correlation coefficient
- introduced by Finkelstein et al. (2002)
- subsequently used by many other researchers
Table of results
- Listed in order of increasing Spearman's rho.
|WikiRelate||Strube and Ponzetto (2006)||Wikipedia||0.48|
|simVB+simWN||Finkelstein et al. (2002)||Hybrid||0.55|
|HSMN+csmRNN||Luong et al. (2013)||Corpus-based||0.65|
|ESA-Wikipedia||Gabrilovich and Markovitch (2007)||Hybrid||0.75|
Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. (2002) Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems, 20(1):116-131.
Gabrilovich, Evgeniy, and Shaul Markovitch. (2007). Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis. In IJCAI, vol. 7, pp. 1606-1611.
Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. (2013). Better word representations with recursive neural networks for morphology. CoNLL-2013: 104.
Strube, Michael and Simone Paolo Ponzetto. (2006). WikiRelate! Computing Semantic Relatedness Using Wikipedia. Proceedings of The 21st National Conference on Artificial Intelligence (AAAI), Boston, MA.