WordSimilarity-353 Test Collection (State of the art)
- WordSimilarity-353 Test Collection
- contains two sets of English word pairs along with human-assigned similarity judgements
- first set (set1) contains 153 word pairs along with their similarity scores assigned by 13 subjects
- second set (set2) contains 200 word pairs with similarity assessed by 16 subjects
- WordSimilarity-353 dataset is available here
- performance is measured by Spearman's rank correlation coefficient
- introduced by Finkelstein et al. (2002)
- subsequently used by many other researchers
- see also: Similarity (State of the art)
Table of results
- Listed in order of increasing Spearman's rho.
References
Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. (2002) Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems, 20(1):116-131.
Gabrilovich, Evgeniy, and Shaul Markovitch. (2007). Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis. In IJCAI, vol. 7, pp. 1606-1611.
Halawi, Guy, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. (2012). Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1406-1414. ACM.
Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. (2013). Better word representations with recursive neural networks for morphology. CoNLL-2013: 104.
Radinsky, Kira, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. (2011). A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pp. 337-346. ACM.
Strube, Michael and Simone Paolo Ponzetto. (2006). WikiRelate! Computing Semantic Relatedness Using Wikipedia. Proceedings of The 21st National Conference on Artificial Intelligence (AAAI), Boston, MA.
| Algorithm | Reference for algorithm | Reference for reported results | Type | Spearman's rho | Pearson's r |
|---|---|---|---|---|---|
| L&C | Leacock and Chodorow (1998) | Hassan and Mihalcea (2011) | Knowledge-based | 0.302 | 0.356 |
| WNE | Jarmasz (2003) | Hassan and Mihalcea (2011) | Knowledge-based | 0.305 | 0.271 |
| J&C | Jiang and Conrath 1997 | Hassan and Mihalcea (2011) | Knowledge-based | 0.318 | 0.354 |
| L&C | Leacock and Chodorow (1998) | Hassan and Mihalcea (2011) | Knowledge-based | 0.348 | 0.341 |
| H&S | Hirst and St-Onge (1998) | Hassan and Mihalcea (2011) | Knowledge-based | 0.302 | 0.356 |
| Lin | Lin (1998) | Hassan and Mihalcea (2011) | Corpus-based | 0.348 | 0.357 |
| Resnik | Resnik (1995) | Hassan and Mihalcea (2011) | Knowledge-based | 0.353 | 0.365 |
| ROGET | Jarmasz (2003) | Hassan and Mihalcea (2011) | Knowledge-based | 0.415 | 0.536 |
| C&W | Collobert and Weston (2008) | Collobert and Weston (2008) | Corpus-based | 0.5 | N/A |
| WikiRelate | Strube and Ponzetto (2006) | Strube and Ponzetto (2006) | Corpus-based | N/A | 0.48 |
| LSA | Landauer et al. (1997) | Hassan and Mihalcea (2011) | Corpus-based | 0.581 | 0.492 |
| LSA | Landauer et al. (1997) | Hassan and Mihalcea (2011) | Corpus-based | 0.581 | 0.563 |
| simVB+simWN | Finkelstein et al. (2002) | Finkelstein et al. (2002) | Hybrid | N/A | 0.55 |
| SSA | Hassan and Mihalcea (2011) | Hassan and Mihalcea (2011) | Knowledge-based | 0.622 | 0.629 |
| HSMN+csmRNN | Luong et al. (2013) | Luong et al. (2013) | Corpus-based | 0.65 | N/A |
| Multi-prototype | Huang et al. (2012) | Huang et al. (2012) | Corpus-based | 0.71 | N/A |
| Multi-lingual SSA | Hassan et al. (2011) | Hassan et al. (2011) | Corpus-based | 0.713 | 0.674 |
| ESA | Gabrilovich and Markovitch (2007) | Gabrilovich and Markovitch (2007) | Corpus-based | 0.748 | 0.503 |
| TSA | Radinsky et al. (2011) | Radinsky et al. (2011) | Hybrid | 0.80 | N/A |
| CLEAR | Halawi et al. (2012) | Halawi et al. (2012) | Corpus-based | 0.81 | N/A |