Difference between revisions of "ESL Synonym Questions (State of the art)"

From ACL Wiki
Jump to: navigation, search
Line 1: Line 1:
* ESL = English as a Second Language
+
In the fashion life, sweet, pleasant style http://www.toptbshop.com/Tory-Burch-Heel-shoes_6_1.htm is a lot of girls to pursue fashion elements, a pleasant little http://www.toptbshop.com/Tory-Burch-Reva-Flats_9_1.htm dress how can a beautiful bag to decorate it! Let me http://www.toptbshop.com/Tory-Burch-New-Arrival_7_1.htm introduce you to several popular http://www.toptbshop.com/Tory-Burch-Heel-shoes_6_1.htm bag to match your dress it sweet and pleasant!This D-shaped large bag is very cool fashion, with the gold chain as a decoration. On a simple bag with clothes with http://www.toptbshop.com/Tory-Burch-Flip-Flops_4_1.htm flowers appropriate.
* 50 multiple-choice synonym questions; 4 choices per question
+
* each question includes a sentence, providing context for the question
+
* ESL questions available on request from [http://www.apperceptual.com/ Peter Turney]
+
* introduced in Turney (2001) as a way of evaluating algorithms for measuring degree of similarity between words
+
* subsequently used by many other researchers
+
 
+
 
+
== Sample question ==
+
 
+
::{| border="0" cellpadding="1" cellspacing="1"
+
|-
+
! Stem:
+
|
+
| "A '''rusty''' nail is not as strong as a clean, new one."
+
|-
+
! Choices:
+
| (a)
+
| corroded
+
|-
+
|
+
| (b)
+
| black
+
|-
+
|
+
| (c)
+
| dirty
+
|-
+
|
+
| (d)
+
| painted
+
|-
+
! Solution:
+
| (a)
+
| corroded
+
|-
+
|}
+
 
+
 
+
== Table of results ==
+
 
+
{| border="1" cellpadding="5" cellspacing="1" width="100%"
+
|-
+
! Algorithm
+
! Reference for algorithm
+
! Reference for experiment
+
! Type
+
! Correct
+
! 95% confidence
+
|-
+
| Random
+
| Random guessing
+
| 1 / 4 = 25.00%
+
| Random
+
| 25.00%
+
| 14.63-40.34%
+
|-
+
| RES
+
| Resnik (1995)
+
| Jarmasz and Szpakowicz (2003)
+
| Hybrid
+
| 32.66%
+
| 21.21-48.77%
+
|-
+
| LC
+
| Leacock and Chodrow (1998)
+
| Jarmasz and Szpakowicz (2003)
+
| Lexicon-based
+
| 36.00%
+
| 22.92-50.81%
+
|-
+
| LIN
+
| Lin (1998)
+
| Jarmasz and Szpakowicz (2003)
+
| Hybrid
+
| 36.00%
+
| 22.92-50.81%
+
|-
+
| JC
+
| Jiang and Conrath (1997)
+
| Jarmasz and Szpakowicz (2003)
+
| Hybrid
+
| 36.00%
+
| 22.92-50.81%
+
|-
+
| HSO
+
| Hirst and St.-Onge (1998)
+
| Jarmasz and Szpakowicz (2003)
+
| Lexicon-based
+
| 62.00%
+
| 47.18-75.35%
+
|-
+
| PMI-IR
+
| Turney (2001)
+
| Turney (2001)
+
| Corpus-based
+
| 74.00%
+
| 59.66-85.37%
+
|-
+
| PMI-IR
+
| Terra and Clarke (2003)
+
| Terra and Clarke (2003)
+
| Corpus-based
+
| 80.00%
+
| 66.28-89.97%
+
|-
+
| JS
+
| Jarmasz and Szpakowicz (2003)
+
| Jarmasz and Szpakowicz (2003)
+
| Lexicon-based
+
| 82.00%
+
| 68.56-91.42%
+
|-
+
|}
+
 
+
 
+
== Explanation of table ==
+
 
+
* '''Algorithm''' = name of algorithm
+
* '''Reference for algorithm''' = where to find out more about given algorithm
+
* '''Reference for experiment''' = where to find out more about evaluation of given algorithm with ESL questions
+
* '''Type''' = general type of algorithm: corpus-based, lexicon-based, hybrid
+
* '''Correct''' = percent of 80 questions that given algorithm answered correctly
+
* '''95% confidence''' = confidence interval calculated using [http://www.quantitativeskills.com/sisa/statistics/onemean.htm Binomial Exact Test]
+
* table rows sorted in order of increasing percent correct
+
* several WordNet-based similarity measures are implemented in [http://www.d.umn.edu/~tpederse/ Ted Pedersen]'s [http://www.d.umn.edu/~tpederse/similarity.html WordNet::Similarity] package
+
* PMI-IR = Pointwise Mutual Information - Information Retrieval
+
* Terra and Clarke (2003) call the ESL Synonym Questions "TS1"
+
 
+
== Caveats ==
+
 
+
* the performance of a corpus-based algorithm depends on the corpus, so the difference in performance between two corpus-based systems may be due to the different corpora, rather than the different algorithms
+
* the ESL questions include nouns, verbs, and adjectives, but some of the WordNet-based algorithms were only designed to work with nouns
+
 
+
 
+
== References ==
+
 
+
Hirst, G., and St-Onge, D. (1998). [http://mirror.eacoss.org/documentation/ITLibrary/IRIS/Data/1997/Hirst/Lexical/1997-Hirst-Lexical.pdf Lexical chains as representation of context for the detection and correction of malapropisms]. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, 305-332.
+
 
+
Jarmasz, M., and Szpakowicz, S. (2003). [http://www.csi.uottawa.ca/~szpak/recent_papers/TR-2003-01.pdf Roget’s thesaurus and semantic similarity], ''Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP-03)'', Borovets, Bulgaria, September, pp. 212-219.
+
 
+
Jiang, J.J., and Conrath, D.W. (1997). [http://wortschatz.uni-leipzig.de/~sbordag/aalw05/Referate/03_Assoziationen_BudanitskyResnik/Jiang_Conrath_97.pdf Semantic similarity based on corpus statistics and lexical taxonomy]. ''Proceedings of the International Conference on Research in Computational Linguistics'', Taiwan.
+
 
+
Leacock, C., and Chodorow, M. (1998). Combining local context and WordNet similarity for word sense identification. In C. Fellbaum (ed.), ''WordNet: An Electronic Lexical Database''. Cambridge: MIT Press, pp. 265-283.
+
 
+
Lin, D. (1998). [http://www.cs.ualberta.ca/~lindek/papers/sim.pdf An information-theoretic definition of similarity]. ''Proceedings of the 15th International Conference on Machine Learning (ICML-98)'', Madison, WI, pp. 296-304.
+
 
+
Resnik, P. (1995). [http://citeseer.ist.psu.edu/resnik95using.html Using information content to evaluate semantic similarity]. ''Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI-95)'', Montreal, pp. 448-453.
+
 
+
Terra, E., and Clarke, C.L.A. (2003). [http://acl.ldc.upenn.edu/N/N03/N03-1032.pdf Frequency estimates for statistical word similarity measures]. ''Proceedings of the Human Language Technology and North American Chapter of Association of Computational Linguistics Conference 2003 (HLT/NAACL 2003)'', pp. 244–251.
+
 
+
Turney, P.D. (2001). [http://arxiv.org/abs/cs.LG/0212033 Mining the Web for synonyms: PMI-IR versus LSA on TOEFL]. ''Proceedings of the Twelfth European Conference on Machine Learning (ECML-2001)'', Freiburg, Germany, pp. 491-502.
+
 
+
== See also ==
+
 
+
* [[Attributional and Relational Similarity (State of the art)]]
+
* [[SAT Analogy Questions]]
+
* [[TOEFL Synonym Questions]]
+
* [[State of the art]]
+
 
+
 
+
[[Category:State of the art]]
+

Revision as of 23:55, 19 September 2011

In the fashion life, sweet, pleasant style http://www.toptbshop.com/Tory-Burch-Heel-shoes_6_1.htm is a lot of girls to pursue fashion elements, a pleasant little http://www.toptbshop.com/Tory-Burch-Reva-Flats_9_1.htm dress how can a beautiful bag to decorate it! Let me http://www.toptbshop.com/Tory-Burch-New-Arrival_7_1.htm introduce you to several popular http://www.toptbshop.com/Tory-Burch-Heel-shoes_6_1.htm bag to match your dress it sweet and pleasant!This D-shaped large bag is very cool fashion, with the gold chain as a decoration. On a simple bag with clothes with http://www.toptbshop.com/Tory-Burch-Flip-Flops_4_1.htm flowers appropriate.