Difference between revisions of "WordSimilarity-353 Test Collection (State of the art)"
Line 18: | Line 18: | ||
|- | |- | ||
! Algorithm | ! Algorithm | ||
− | ! Reference | + | ! Reference for algorithm |
+ | ! Reference for reported results | ||
! Type | ! Type | ||
! Spearman's rho | ! Spearman's rho | ||
+ | ! Pearson's r | ||
+ | |- | ||
+ | | L&C | ||
+ | | Leacock and Chodorow (1998) | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Knowledge-based | ||
+ | | 0.302 | ||
+ | | 0.356 | ||
+ | |- | ||
+ | | WNE | ||
+ | | Jarmasz (2003) | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Knowledge-based | ||
+ | | 0.305 | ||
+ | | 0.271 | ||
+ | |- | ||
+ | | J&C | ||
+ | | Jiang and Conrath 1997 | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Knowledge-based | ||
+ | | 0.318 | ||
+ | | 0.354 | ||
+ | |- | ||
+ | | L&C | ||
+ | | Leacock and Chodorow (1998) | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Knowledge-based | ||
+ | | 0.348 | ||
+ | | 0.341 | ||
+ | |- | ||
+ | | H&S | ||
+ | | Hirst and St-Onge (1998) | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Knowledge-based | ||
+ | | 0.302 | ||
+ | | 0.356 | ||
+ | |- | ||
+ | | Lin | ||
+ | | Lin (1998) | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Corpus-based | ||
+ | | 0.348 | ||
+ | | 0.357 | ||
+ | |- | ||
+ | | Resnik | ||
+ | | Resnik (1995) | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Knowledge-based | ||
+ | | 0.353 | ||
+ | | 0.365 | ||
+ | |- | ||
+ | | ROGET | ||
+ | | Jarmasz (2003) | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Knowledge-based | ||
+ | | 0.415 | ||
+ | | 0.536 | ||
+ | |- | ||
+ | | C&W | ||
+ | | Collobert and Weston (2008) | ||
+ | | Collobert and Weston (2008) | ||
+ | | Corpus-based | ||
+ | | 0.5 | ||
+ | | N/A | ||
|- | |- | ||
| WikiRelate | | WikiRelate | ||
| Strube and Ponzetto (2006) | | Strube and Ponzetto (2006) | ||
− | | | + | | Strube and Ponzetto (2006) |
+ | | Corpus-based | ||
+ | | N/A | ||
| 0.48 | | 0.48 | ||
|- | |- | ||
− | | | + | | LSA |
− | | | + | | Landauer et al. (1997) |
+ | | Hassan and Mihalcea (2011) | ||
+ | | Corpus-based | ||
+ | | 0.581 | ||
+ | | 0.492 | ||
+ | |- | ||
+ | | LSA | ||
+ | | Landauer et al. (1997) | ||
+ | | Hassan and Mihalcea (2011) | ||
| Corpus-based | | Corpus-based | ||
− | | 0. | + | | 0.581 |
+ | | 0.563 | ||
|- | |- | ||
| simVB+simWN | | simVB+simWN | ||
+ | | Finkelstein et al. (2002) | ||
| Finkelstein et al. (2002) | | Finkelstein et al. (2002) | ||
| Hybrid | | Hybrid | ||
+ | | N/A | ||
| 0.55 | | 0.55 | ||
+ | |- | ||
+ | | SSA | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Hassan and Mihalcea (2011) | ||
+ | | Knowledge-based | ||
+ | | 0.622 | ||
+ | | 0.629 | ||
|- | |- | ||
| HSMN+csmRNN | | HSMN+csmRNN | ||
+ | | Luong et al. (2013) | ||
| Luong et al. (2013) | | Luong et al. (2013) | ||
| Corpus-based | | Corpus-based | ||
| 0.65 | | 0.65 | ||
+ | | N/A | ||
|- | |- | ||
− | | | + | | Multi-prototype |
+ | | Huang et al. (2012) | ||
| Huang et al. (2012) | | Huang et al. (2012) | ||
| Corpus-based | | Corpus-based | ||
| 0.71 | | 0.71 | ||
+ | | N/A | ||
+ | |- | ||
+ | | Multi-lingual SSA | ||
+ | | Hassan et al. (2011) | ||
+ | | Hassan et al. (2011) | ||
+ | | Corpus-based | ||
+ | | 0.713 | ||
+ | | 0.674 | ||
|- | |- | ||
− | | ESA | + | | ESA |
+ | | Gabrilovich and Markovitch (2007) | ||
| Gabrilovich and Markovitch (2007) | | Gabrilovich and Markovitch (2007) | ||
− | | | + | | Corpus-based |
− | | 0. | + | | 0.748 |
+ | | 0.503 | ||
|- | |- | ||
| TSA | | TSA | ||
+ | | Radinsky et al. (2011) | ||
| Radinsky et al. (2011) | | Radinsky et al. (2011) | ||
| Hybrid | | Hybrid | ||
| 0.80 | | 0.80 | ||
+ | | N/A | ||
|- | |- | ||
| CLEAR | | CLEAR | ||
+ | | Halawi et al. (2012) | ||
| Halawi et al. (2012) | | Halawi et al. (2012) | ||
| Corpus-based | | Corpus-based | ||
| 0.81 | | 0.81 | ||
− | |} | + | | N/A |
+ | |- | ||
+ | } | ||
== References == | == References == |
Revision as of 20:11, 7 January 2014
- WordSimilarity-353 Test Collection
- contains two sets of English word pairs along with human-assigned similarity judgements
- first set (set1) contains 153 word pairs along with their similarity scores assigned by 13 subjects
- second set (set2) contains 200 word pairs with similarity assessed by 16 subjects
- WordSimilarity-353 dataset is available here
- performance is measured by Spearman's rank correlation coefficient
- introduced by Finkelstein et al. (2002)
- subsequently used by many other researchers
- see also: Similarity (State of the art)
Table of results
- Listed in order of increasing Spearman's rho.
References
Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. (2002) Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems, 20(1):116-131.
Gabrilovich, Evgeniy, and Shaul Markovitch. (2007). Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis. In IJCAI, vol. 7, pp. 1606-1611.
Halawi, Guy, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. (2012). Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1406-1414. ACM.
Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. (2013). Better word representations with recursive neural networks for morphology. CoNLL-2013: 104.
Radinsky, Kira, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. (2011). A word at a time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th international conference on World wide web, pp. 337-346. ACM.
Strube, Michael and Simone Paolo Ponzetto. (2006). WikiRelate! Computing Semantic Relatedness Using Wikipedia. Proceedings of The 21st National Conference on Artificial Intelligence (AAAI), Boston, MA.
Algorithm | Reference for algorithm | Reference for reported results | Type | Spearman's rho | Pearson's r |
---|---|---|---|---|---|
L&C | Leacock and Chodorow (1998) | Hassan and Mihalcea (2011) | Knowledge-based | 0.302 | 0.356 |
WNE | Jarmasz (2003) | Hassan and Mihalcea (2011) | Knowledge-based | 0.305 | 0.271 |
J&C | Jiang and Conrath 1997 | Hassan and Mihalcea (2011) | Knowledge-based | 0.318 | 0.354 |
L&C | Leacock and Chodorow (1998) | Hassan and Mihalcea (2011) | Knowledge-based | 0.348 | 0.341 |
H&S | Hirst and St-Onge (1998) | Hassan and Mihalcea (2011) | Knowledge-based | 0.302 | 0.356 |
Lin | Lin (1998) | Hassan and Mihalcea (2011) | Corpus-based | 0.348 | 0.357 |
Resnik | Resnik (1995) | Hassan and Mihalcea (2011) | Knowledge-based | 0.353 | 0.365 |
ROGET | Jarmasz (2003) | Hassan and Mihalcea (2011) | Knowledge-based | 0.415 | 0.536 |
C&W | Collobert and Weston (2008) | Collobert and Weston (2008) | Corpus-based | 0.5 | N/A |
WikiRelate | Strube and Ponzetto (2006) | Strube and Ponzetto (2006) | Corpus-based | N/A | 0.48 |
LSA | Landauer et al. (1997) | Hassan and Mihalcea (2011) | Corpus-based | 0.581 | 0.492 |
LSA | Landauer et al. (1997) | Hassan and Mihalcea (2011) | Corpus-based | 0.581 | 0.563 |
simVB+simWN | Finkelstein et al. (2002) | Finkelstein et al. (2002) | Hybrid | N/A | 0.55 |
SSA | Hassan and Mihalcea (2011) | Hassan and Mihalcea (2011) | Knowledge-based | 0.622 | 0.629 |
HSMN+csmRNN | Luong et al. (2013) | Luong et al. (2013) | Corpus-based | 0.65 | N/A |
Multi-prototype | Huang et al. (2012) | Huang et al. (2012) | Corpus-based | 0.71 | N/A |
Multi-lingual SSA | Hassan et al. (2011) | Hassan et al. (2011) | Corpus-based | 0.713 | 0.674 |
ESA | Gabrilovich and Markovitch (2007) | Gabrilovich and Markovitch (2007) | Corpus-based | 0.748 | 0.503 |
TSA | Radinsky et al. (2011) | Radinsky et al. (2011) | Hybrid | 0.80 | N/A |
CLEAR | Halawi et al. (2012) | Halawi et al. (2012) | Corpus-based | 0.81 | N/A |