Difference between revisions of "Textual Entailment Resource Pool"

From ACL Wiki
Jump to: navigation, search
(Other data sets)
(20 intermediate revisions by 6 users not shown)
Line 16: Line 16:
 
* [http://l2r.cs.uiuc.edu/~cogcomp/kindleDemo.php Entailment Demo] (from the University of Illinois at Urbana-Champaign) - INACTIVE (as of 2010-12-22)
 
* [http://l2r.cs.uiuc.edu/~cogcomp/kindleDemo.php Entailment Demo] (from the University of Illinois at Urbana-Champaign) - INACTIVE (as of 2010-12-22)
 
* [http://edits.fbk.eu/ EDITS - Edit Distance Textual Entailment Suite] (open source software developed by [http://hlt.fbk.eu/ Human Language Technology (HLT) group at FBK-Irst])
 
* [http://edits.fbk.eu/ EDITS - Edit Distance Textual Entailment Suite] (open source software developed by [http://hlt.fbk.eu/ Human Language Technology (HLT) group at FBK-Irst])
* [http://www.cs.biu.ac.il/~nlp/downloads/biutee/index.html BIUTEE] - Bar Ilan University Textual Entailment Engine (open source).  
+
* [http://u.cs.biu.ac.il/~nlp/downloads/biutee/protected-biutee.html BIUTEE] - Bar Ilan University Textual Entailment Engine (open source)
 +
* [http://hltfbk.github.io/Excitement-Open-Platform/ EXCITEMENT Open Platform (EOP)] - A generic multi-lingual platform for textual inference made available to the scientific and technological communities by the [http://www.excitement-project.eu/ EU project EXCITEMENT]
  
 
== RTE data sets ==
 
== RTE data sets ==
Line 25: Line 26:
 
* [http://www.nist.gov/tac/data/past/2008/RTE-4.html RTE4 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements]
 
* [http://www.nist.gov/tac/data/past/2008/RTE-4.html RTE4 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements]
 
* [http://www.nist.gov/tac/data/past/2009/RTE-5.html RTE5 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements]
 
* [http://www.nist.gov/tac/data/past/2009/RTE-5.html RTE5 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements]
* [http://www.nist.gov/tac/2010/RTE/index.html RTE6 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements]
+
* [http://www.nist.gov/tac/data/past/2010/RTE-6_Main_Task.html RTE6 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements]
 +
* [http://www.nist.gov/tac/2011/RTE/index.html RTE7 dataset] - provided by [http://www.nist.gov/index.html NIST] - freely available upon request. For details see [http://www.nist.gov/tac/data/forms/index.html TAC User Agreements]
 +
* [http://www.cs.york.ac.uk/semeval-2013/task7/  The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge] at SemEval 2013
 +
 
 +
 
 +
=== RTE data sets translated in other languages ===
 +
* [http://www.dfki.de/~neumann/resources/RTE3_DE_V1.2_2013-12-02.zip RTE3 dataset translated in German] - provided by [http://www.excitement-project.eu/ EXCITEMENT]
 +
* [http://www.excitement-project.eu/attachments/article/97/RTE3-ITA_V1_2012-10-04.zip RTE3 dataset translated in Italian] - provided by [http://www.excitement-project.eu/ EXCITEMENT]
 +
 
  
 
=== Other data sets ===
 
=== Other data sets ===
Line 37: Line 46:
 
* [https://agora.cs.illinois.edu/display/rtedata/Explanation+Based+Analysis+of+RTE+Data Explanation-Based Analysis annotation of RTE 5 Main Task subset] described in [http://l2r.cs.uiuc.edu/~danr/Papers/SammonsVyRo10.pdf  this ACL 2010 paper]
 
* [https://agora.cs.illinois.edu/display/rtedata/Explanation+Based+Analysis+of+RTE+Data Explanation-Based Analysis annotation of RTE 5 Main Task subset] described in [http://l2r.cs.uiuc.edu/~danr/Papers/SammonsVyRo10.pdf  this ACL 2010 paper]
 
* [http://art.uniroma2.it/zanzotto/resources/WIKI_FINAL_CORPUS_v1.zip Wiki Entailment Corpus] A RTE-like set of entailment pairs extracted from Wikipedia revisions described in [http://aclweb.org/anthology/W/W10/W10-3504.pdf  this paper]
 
* [http://art.uniroma2.it/zanzotto/resources/WIKI_FINAL_CORPUS_v1.zip Wiki Entailment Corpus] A RTE-like set of entailment pairs extracted from Wikipedia revisions described in [http://aclweb.org/anthology/W/W10/W10-3504.pdf  this paper]
 +
* [https://github.com/daoudclarke/rte-experiment The Guardian Headlines Entailment Training Dataset] An automatically generated dataset of 32,000 pairs similar to the RTE-1 dataset.
 +
* [http://nlp.uned.es/clef-qa/ave/ Answer Validation Exercise at CLEF 2006 (AVE 2006)]
 +
* [http://www.evalita.it/2009/tasks/te The Textual Entailment Task for Italian] at [http://www.evalita.it/2009 EVALITA 2009] An evaluation exercise on TE for Italian.
 +
* [http://www.cs.york.ac.uk/semeval-2012/task8/ Cross-Lingual Textual Entailment for Content Synchronization] The Cross-Lingual Textual Entailment task at [http://www.cs.york.ac.uk/semeval-2012/‎ SemEval 2012].
 +
* [http://www.cs.york.ac.uk/semeval-2013/task8/ Cross-Lingual Textual Entailment for Content Synchronization] The Cross-Lingual Textual Entailment task at [http://www.cs.york.ac.uk/semeval-2013/‎ SemEval 2013].
  
 
== Knowledge Resources ==
 
== Knowledge Resources ==
Line 43: Line 57:
 
* a [[RTE Knowledge Resources#Call for Resources|call for resources]], inviting system developers to share the resources used by their own TE engines, to both help improve the TE technology and further test and evaluate such resources;
 
* a [[RTE Knowledge Resources#Call for Resources|call for resources]], inviting system developers to share the resources used by their own TE engines, to both help improve the TE technology and further test and evaluate such resources;
 
* [[RTE Knowledge Resources#Ablation tests|the ablation tests]] carried out in the RTE challenges in order to evaluate the impact of knowledge resources and tools on TE system performances;
 
* [[RTE Knowledge Resources#Ablation tests|the ablation tests]] carried out in the RTE challenges in order to evaluate the impact of knowledge resources and tools on TE system performances;
* [[RTE Knowledge Resources#Publicly available Resources|lists of knowledge resources]], both publically available and unpublished, used by systems participating in the last RTE challenges.
+
* [[RTE Knowledge Resources#Publicly available Resources|lists of knowledge resources]], both publicly available and unpublished, used by systems participating in the last RTE challenges.
 
<!-- * [https://agora.cs.illinois.edu/display/rtedata/Explanation+Based+Analysis+of+RTE+Data Explanation-Based Analysis annotation of RTE 5 Main Task subset] described in [http://l2r.cs.uiuc.edu/~danr/Papers/SammonsVyRo10.pdf  this ACL 2010 paper] -->
 
<!-- * [https://agora.cs.illinois.edu/display/rtedata/Explanation+Based+Analysis+of+RTE+Data Explanation-Based Analysis annotation of RTE 5 Main Task subset] described in [http://l2r.cs.uiuc.edu/~danr/Papers/SammonsVyRo10.pdf  this ACL 2010 paper] -->
 +
 +
== Projects ==
 +
* [http://www.cosyne.eu/ CoSyne EU project] The Cross-Lingual Multilingual Content Synchronization with Wikis.
 +
* [http://www.excitement-project.eu/ EXCITEMENT EU project] EXploring Customer Interactions through Textual EntailMENT.
 +
* [http://qallme.fbk.eu/ QALL-ME EU project] Question Answering Learning technologies in a multiLingual and Multimodal Environment.
  
 
== Tools ==
 
== Tools ==
Line 63: Line 82:
  
 
=== Similarity / Relatedness Tools ===
 
=== Similarity / Relatedness Tools ===
* [http://ixa2.si.ehu.es/ukb UKB]: Open source WordNet-based similarity/relatedness tool, includes also pre-computed semantic vectors for all words
+
* [http://ixa2.si.ehu.es/ukb UKB]: Open source [[WordNet]]-based similarity/relatedness tool, includes also pre-computed semantic vectors for all words
  
 
=== Corpus Readers ===
 
=== Corpus Readers ===
Line 78: Line 97:
 
== References ==
 
== References ==
  
*[[Textual Entailment References#Workshops and Tutorials | Workshops and Tutorials ]]
+
*[[Textual Entailment References#Tutorials | Tutorials ]] and [[Textual Entailment References#Workshops | Workshops ]]
 
*[[Textual Entailment References#Papers in recent conferences and other workshops | Papers in recent conferences and other workshops ]]
 
*[[Textual Entailment References#Papers in recent conferences and other workshops | Papers in recent conferences and other workshops ]]
 
*[[Textual Entailment References#Journal papers | Journal papers ]]
 
*[[Textual Entailment References#Journal papers | Journal papers ]]

Revision as of 05:14, 9 December 2013

Textual Entailment > Resources:


Textual entailment systems rely on many different types of NLP resources, including term banks, paraphrase lists, parsers, named-entity recognizers, etc. With so many resources being continuously released and improved, it can be difficult to know which particular resource to use when developing a system.

In response, the Recognizing Textual Entailment (RTE) shared task community initiated a new activity for building this Textual Entailment Resource Pool. RTE participants and any other member of the NLP community are encouraged to contribute to the pool.

In an effort to determine the relative impact of the resources, RTE participants are strongly encouraged to report, whenever possible, the contribution to the overall performance of each utilized resource. Formal qualitative and quantitative results should be included in a separate section of the system report as well as posted on the talk pages of this Textual Entailment Resource Pool.

Adding a new resource is very easy. See how to use existing templates to do this in Help:Using Templates.

Complete RTE Systems

RTE data sets

Past campaigns data sets


RTE data sets translated in other languages


Other data sets

Knowledge Resources

The RTE Knowledge Resources page presents:

  • a call for resources, inviting system developers to share the resources used by their own TE engines, to both help improve the TE technology and further test and evaluate such resources;
  • the ablation tests carried out in the RTE challenges in order to evaluate the impact of knowledge resources and tools on TE system performances;
  • lists of knowledge resources, both publicly available and unpublished, used by systems participating in the last RTE challenges.

Projects

Tools

Parsers

Role Labelling

Entity Recognition Tools

Similarity / Relatedness Tools

  • UKB: Open source WordNet-based similarity/relatedness tool, includes also pre-computed semantic vectors for all words

Corpus Readers

  • NLTK provides a corpus reader for the data from RTE Challenges 1, 2, and 3 - see the Corpus Readers Guide for more information.

Related Libraries

  • PyPES general purpose library containing evaluation environment for RTE and McPIET text inference engine based on the ERG (English Resource Grammar)

Text Normalizers

Java number normalizer (Beta) A tool for converting textual representations of numbers to a standard numerical string.

References

Links