Difference between revisions of "Analogy (State of the art)"

Revision as of 02:54, 6 January 2017

Analogy task

A proportional analogy holds between two word pairs: a:a* :: b:b* (a is to a* as b is to b*) For example, Tokyo is to Japan as Paris is to France.

With the pair-based methods, given a:a* :: b:?, the task is to find b*.

With set-based methods, the task is to find b* given a set of other pairs (excluding b:b*) that hold the same relation as b:b*.

In NLP analogies (Mikolov's "linguistic regularities"^[1]) are interpreted broadly as basically any "similarities between pairs of words" ^[2], not just semantic.

Available analogy datasets (ordered by date)

Listed by date

Dataset	Reference	Number of questions	Number of relations	Dataset Link	List of state-of-the-art results	Comments
SAT	Turney et al (2003)^[3]	374	misc	available on request from Peter Turney	SAT Analogy Questions (State of the art)	different task formulation: select the correct answer out of 5 proposed alternatives
SemEval 2012 Task 2	Jurgens et al (2012)^[4]	3218	79	SemEval2012-Task2	SemEval-2012 Task 2 (State of the art)	different task formulation: ranking the degree to which a relation applies.
MSR	Mikolov et al. (2013a)^[1]	8,000	8	MSR	Syntactic Analogies (State of the art)	Syntactic (i.e. morphological) questions only
Google	Mikolov et al. (2013b)^[5]	19544	15	Original link deprecated, copy hosted @TensorFlow	Google analogy test set (State of the art)	unbalanced: 8,869 semantic and 10,675 syntactic questions, with 20-70 pairs per category; country:capital relation is over 50% of all semantic questions. Relations in the syntactic part largely the same as MSR.
BATS	Gladkova et al. (2016)^[6]	99,200	40	BATS	Bigger analogy test set (State of the art)	balanced across 4 types of relations: inflectional and derivational morphology, encyclopedic and lexicographic semantics. 10 relations of each type with 50 unique source pairs per relation. Multiple correct answers allowed where applicable.

Methods to solve analogies

Pair-based methods for solving analogies

vector offset a.k.a. 3CosAdd ^[7]
3CosMul ^[2]
others discussed by Linzen (2016)^[8].

Set-based methods for solving analogies

3CosAvg (vector offset averaged over multiple pairs) ^[9]
LRCos (supervised learning of the target class + cosine similarity to the b word) ^[9].

Issues with evaluating word embeddings on analogy task

There is interplay between the chosen embedding, its parameters, particular relations ^[6], and method of solving analogies ^[8] ^[9]. It is possible that analogies not solved by one method can be solved by another method on the same embedding. Therefore results for solving analogies with different methods should be taken as a way to explore or describe an embedding rather than evaluate it.

Notes

↑ ^1.0 ^1.1 Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
↑ ^2.0 ^2.1 Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf
↑ Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366
↑ Jurgens, D. A., Turney, P. D., Mohammad, S. M., & Holyoak, K. J. (2012). Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM) (pp. 356–364). Montréal, Canada, June 7-8, 2012: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation.cfm?id=2387693
↑ Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).
↑ ^6.0 ^6.1 Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf
↑ Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784
↑ ^8.0 ^8.1 Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503
↑ ^9.0 ^9.1 ^9.2 Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf

[Mikolov2013a-1] 1.0 ^1.1 Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784

[Levy2014-2] 2.0 ^2.1 Levy, O., Goldberg, Y., & Ramat-Gan, I. (2014). Linguistic Regularities in Sparse and Explicit Word Representations. In CoNLL (pp. 171–180). Retrieved from http://anthology.aclweb.org/W/W14/W14-1618.pdf

[3] Turney, P., Littman, M. L., Bigham, J., & Shnayder, V. (2003). Combining independent modules to solve multiple-choice synonym and analogy problems. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (pp. 482--489). Retrieved from http://nparc.cisti-icist.nrc-cnrc.gc.ca/npsi/ctrl?action=rtdoc&an=8913366

[4] Jurgens, D. A., Turney, P. D., Mohammad, S. M., & Holyoak, K. J. (2012). Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM) (pp. 356–364). Montréal, Canada, June 7-8, 2012: Association for Computational Linguistics. Retrieved from http://dl.acm.org/citation.cfm?id=2387693

[5] Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations (ICLR).

[Gladkova2016-6] 6.0 ^6.1 Gladkova, A., Drozd, A., & Matsuoka, S. (2016). Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In Proceedings of the NAACL-HLT SRW (pp. 47–54). San Diego, California, June 12-17, 2016: ACL. Retrieved from https://www.aclweb.org/anthology/N/N16/N16-2002.pdf

[7] Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. In HLT-NAACL (pp. 746–751). Retrieved from http://www.aclweb.org/anthology/N13-1#page=784

[Linzen2016-8] 8.0 ^8.1 Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. In Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics. Retrieved from http://anthology.aclweb.org/W16-2503

[LRCos-9] 9.0 ^9.1 ^9.2 Drozd, A., Gladkova, A., & Matsuoka, S. (2016). Word embeddings, analogies, and machine learning: beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3519–3530). Osaka, Japan, December 11-17: ACL. Retrieved from https://www.aclweb.org/anthology/C/C16/C16-1332.pdf

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

@@ Line 59: / Line 59: @@
 | 40
 |[https://s3.amazonaws.com/blackbirdprojects/tut_vsm/BATS_3.0.zip BATS]
-|
+| [[Bigger analogy test set (State of the art)]]
 |balanced across 4 types of relations: inflectional and derivational morphology, encyclopedic and lexicographic semantics. 10 relations of each type with 50 unique source pairs per relation. Multiple correct answers allowed where applicable.
 |-