Difference between revisions of "WordSimilarity-353 Test Collection (State of the art)"
Jump to navigation
Jump to search
| (8 intermediate revisions by 2 users not shown) | |||
| Line 8: | Line 8: | ||
* introduced by [http://www.cs.technion.ac.il/~gabr/papers/tois_context.pdf Finkelstein et al. (2002)] | * introduced by [http://www.cs.technion.ac.il/~gabr/papers/tois_context.pdf Finkelstein et al. (2002)] | ||
* subsequently used by many other researchers | * subsequently used by many other researchers | ||
| + | * see also: [[Similarity (State of the art)]] | ||
| Line 13: | Line 14: | ||
* '''Listed in order of increasing [http://en.wikipedia.org/wiki/Spearman_rank_correlation Spearman's rho].''' | * '''Listed in order of increasing [http://en.wikipedia.org/wiki/Spearman_rank_correlation Spearman's rho].''' | ||
| − | |||
{| border="1" cellpadding="5" cellspacing="1" | {| border="1" cellpadding="5" cellspacing="1" | ||
| Line 21: | Line 21: | ||
! Type | ! Type | ||
! Spearman's rho | ! Spearman's rho | ||
| + | |- | ||
| + | | WikiRelate | ||
| + | | Strube and Ponzetto (2006) | ||
| + | | Wikipedia | ||
| + | | 0.48 | ||
| + | |- | ||
| + | | C&W | ||
| + | | Collobert and Weston (2008) | ||
| + | | Corpus-based | ||
| + | | 0.498 | ||
| + | |- | ||
| + | | simVB+simWN | ||
| + | | Finkelstein et al. (2002) | ||
| + | | Hybrid | ||
| + | | 0.55 | ||
|- | |- | ||
| HSMN+csmRNN | | HSMN+csmRNN | ||
| Line 26: | Line 41: | ||
| Corpus-based | | Corpus-based | ||
| 0.65 | | 0.65 | ||
| + | |- | ||
| + | | multi-prototype | ||
| + | | Huang et al. (2012) | ||
| + | | Corpus-based | ||
| + | | 0.713 | ||
|- | |- | ||
| ESA-Wikipedia | | ESA-Wikipedia | ||
| Line 31: | Line 51: | ||
| Hybrid | | Hybrid | ||
| 0.75 | | 0.75 | ||
| + | |- | ||
| + | | TSA | ||
| + | | Radinsky et al. (2011) | ||
| + | | Hybrid | ||
| + | | 0.800 | ||
| + | |- | ||
| + | | CLEAR | ||
| + | | Halawi et al. (2012) | ||
| + | | Corpus-based | ||
| + | | 0.810 | ||
|} | |} | ||
| + | == References == | ||
| − | + | Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. (2002) [http://www.cs.technion.ac.il/~gabr/papers/tois_context.pdf Placing Search in Context: The Concept Revisited]. ACM Transactions on Information Systems, 20(1):116-131. | |
Gabrilovich, Evgeniy, and Shaul Markovitch. (2007). [http://www.cs.technion.ac.il/~gabr/papers/ijcai-2007-sim.pdf Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis]. In IJCAI, vol. 7, pp. 1606-1611. | Gabrilovich, Evgeniy, and Shaul Markovitch. (2007). [http://www.cs.technion.ac.il/~gabr/papers/ijcai-2007-sim.pdf Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis]. In IJCAI, vol. 7, pp. 1606-1611. | ||
| Line 40: | Line 71: | ||
Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. (2013). [http://nlp.stanford.edu/~lmthang/data/papers/conll13_morpho.pdf Better word representations with recursive neural networks for morphology]. CoNLL-2013: 104. | Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. (2013). [http://nlp.stanford.edu/~lmthang/data/papers/conll13_morpho.pdf Better word representations with recursive neural networks for morphology]. CoNLL-2013: 104. | ||
| − | + | Strube, Michael and Simone Paolo Ponzetto. (2006). [http://www.aaai.org/Papers/AAAI/2006/AAAI06-223.pdf WikiRelate! Computing Semantic Relatedness Using Wikipedia]. Proceedings of The 21st National Conference on Artificial Intelligence (AAAI), Boston, MA. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
[[Category:State of the art]] | [[Category:State of the art]] | ||
Revision as of 14:30, 11 November 2013
- WordSimilarity-353 Test Collection
- contains two sets of English word pairs along with human-assigned similarity judgements
- first set (set1) contains 153 word pairs along with their similarity scores assigned by 13 subjects
- second set (set2) contains 200 word pairs with similarity assessed by 16 subjects
- WordSimilarity-353 dataset is available here
- performance is measured by Spearman's rank correlation coefficient
- introduced by Finkelstein et al. (2002)
- subsequently used by many other researchers
- see also: Similarity (State of the art)
Table of results
- Listed in order of increasing Spearman's rho.
| Algorithm | Reference | Type | Spearman's rho |
|---|---|---|---|
| WikiRelate | Strube and Ponzetto (2006) | Wikipedia | 0.48 |
| C&W | Collobert and Weston (2008) | Corpus-based | 0.498 |
| simVB+simWN | Finkelstein et al. (2002) | Hybrid | 0.55 |
| HSMN+csmRNN | Luong et al. (2013) | Corpus-based | 0.65 |
| multi-prototype | Huang et al. (2012) | Corpus-based | 0.713 |
| ESA-Wikipedia | Gabrilovich and Markovitch (2007) | Hybrid | 0.75 |
| TSA | Radinsky et al. (2011) | Hybrid | 0.800 |
| CLEAR | Halawi et al. (2012) | Corpus-based | 0.810 |
References
Finkelstein, Lev, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. (2002) Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems, 20(1):116-131.
Gabrilovich, Evgeniy, and Shaul Markovitch. (2007). Computing Semantic Relatedness Using Wikipedia-based Explicit Semantic Analysis. In IJCAI, vol. 7, pp. 1606-1611.
Luong, Minh-Thang, Richard Socher, and Christopher D. Manning. (2013). Better word representations with recursive neural networks for morphology. CoNLL-2013: 104.
Strube, Michael and Simone Paolo Ponzetto. (2006). WikiRelate! Computing Semantic Relatedness Using Wikipedia. Proceedings of The 21st National Conference on Artificial Intelligence (AAAI), Boston, MA.