Difference between revisions of "SimLex-999 (State of the art)"
Jump to navigation
Jump to search
(Created page with "[http://www.cl.cam.ac.uk/~fh295/simlex.html SimLex-999] aims at a cleaner benchmark of similarity (but not relatedness). Pairs of words were chosen to represent different rang...") |
m (Hill et al. (2014) --> 2014a & 2014b) |
||
| Line 7: | Line 7: | ||
| Neural language model | | Neural language model | ||
| Collobert & Weston (2008)<ref>R. Collobert and J. Weston. 2008. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In International Conference on Machine Learn- ing, ICML.</ref> | | Collobert & Weston (2008)<ref>R. Collobert and J. Weston. 2008. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In International Conference on Machine Learn- ing, ICML.</ref> | ||
| − | | Hill et al. ( | + | | Hill et al. (2014a)<ref name=simlex/> |
| Distributional || 0.268 || - || Trained on Wikipedia | | Distributional || 0.268 || - || Trained on Wikipedia | ||
|- | |- | ||
| Neural language model with global context | | Neural language model with global context | ||
| Huang et al. (2012)<ref>Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 873–882. Association for Computational Linguistics.</ref> | | Huang et al. (2012)<ref>Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 873–882. Association for Computational Linguistics.</ref> | ||
| − | | Hill et al. ( | + | | Hill et al. (2014a)<ref name=simlex/> |
| Distributional || 0.098 || - || Trained on Wikipedia | | Distributional || 0.098 || - || Trained on Wikipedia | ||
|- | |- | ||
| Word2vec | | Word2vec | ||
| Mikolov et al. (2013)<ref>Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations, Scottsdale, Arizona, USA.</ref> | | Mikolov et al. (2013)<ref>Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations, Scottsdale, Arizona, USA.</ref> | ||
| − | | Hill et al. ( | + | | Hill et al. (2014a)<ref name=simlex>Hill, F., Reichart, R., & Korhonen, A. (2014a). SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation. Computation and Language.</ref> |
| Distributional || 0.414 || - || Trained on Wikipedia | | Distributional || 0.414 || - || Trained on Wikipedia | ||
|- | |- | ||
| Line 41: | Line 41: | ||
|- | |- | ||
| RNNenc | | RNNenc | ||
| − | | Hill et al. ( | + | | Hill et al. (2014b)<ref name=rnnenc>Hill, F., Cho, K., Jean, S., Devin, C., & Bengio, Y. (2014b). Not All Neural Embeddings are Born Equal, 1–5.</ref> |
| − | | Hill et al. ( | + | | Hill et al. (2014b)<ref name=rnnenc/> |
| Distributional, multilingual || 0.52 || - | | Distributional, multilingual || 0.52 || - | ||
|} | |} | ||
Revision as of 06:58, 25 June 2015
SimLex-999 aims at a cleaner benchmark of similarity (but not relatedness). Pairs of words were chosen to represent different ranges of similarity and with either high or low association. Subjects were instructed to differentiate between similarity and relatedness and rate regarding the former only.
| Algorithm | Reference for algorithm | Reference for reported results | Type | Spearman's rho | Pearson's r | Notes |
|---|---|---|---|---|---|---|
| Neural language model | Collobert & Weston (2008)[1] | Hill et al. (2014a)[2] | Distributional | 0.268 | - | Trained on Wikipedia |
| Neural language model with global context | Huang et al. (2012)[3] | Hill et al. (2014a)[2] | Distributional | 0.098 | - | Trained on Wikipedia |
| Word2vec | Mikolov et al. (2013)[4] | Hill et al. (2014a)[2] | Distributional | 0.414 | - | Trained on Wikipedia |
| Lesk | Banjade et al. (2015)[5] | 0.404 | 0.347 | |||
| UMBC | Han et al. (2013)[6] | Banjade et al. (2015)[5] | 0.558 | 0.557 | without using POS information | |
| SVR4 | Banjade et al. (2015)[5] | Banjade et al. (2015)[5] | Combined | 0.642 | 0.658 | |
| ESA | Banjade et al. (2015)[5] | 0.271 | 0.145 | |||
| RNNenc | Hill et al. (2014b)[7] | Hill et al. (2014b)[7] | Distributional, multilingual | 0.52 | - |
References
- ↑ R. Collobert and J. Weston. 2008. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In International Conference on Machine Learn- ing, ICML.
- ↑ 2.0 2.1 2.2 Hill, F., Reichart, R., & Korhonen, A. (2014a). SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation. Computation and Language.
- ↑ Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 873–882. Association for Computational Linguistics.
- ↑ Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference of Learning Representations, Scottsdale, Arizona, USA.
- ↑ 5.0 5.1 5.2 5.3 5.4 Banjade, R., Maharjan, N., Niraula, N., Rus, V., & Gautam, D. (2015). Lemon and Tea Are Not Similar: Measuring Word-to-Word Similarity by Combining Different Methods. Computational Linguistics and Intelligent Text Processing, 9041, 335–346. doi:10.1007/978-3-319-18111-0_25
- ↑ Han, L., Kashyap, A., Finin, T., Mayfield, J., Weese, J.: UMBC EBIQUITY-CORE: Semantic textual similarity systems. In: Proceedings of the Second Joint Conference on Lexical and Computational Semantics, vol. 1, pp. 44–52 (2013)
- ↑ 7.0 7.1 Hill, F., Cho, K., Jean, S., Devin, C., & Bengio, Y. (2014b). Not All Neural Embeddings are Born Equal, 1–5.