Difference between revisions of "WordNet - RTE Users"

From ACL Wiki
Jump to: navigation, search
m
Line 24: Line 24:
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| Boeing
+
| AUEB
 
| RTE4
 
| RTE4
| 2.0
+
|  
| Semantic relation between words
+
|  
| No formal evaluation. Plays a role in most entailments found
+
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| DFKI
+
| BIU
 
| RTE4
 
| RTE4
 
| 3.0
 
| 3.0
| Semantic relation between words
+
| Derived on the fly lexical entailment rules, using synonyms, hypernyms (up to two levels)  and derivations. Also used as part of our novel lexical-syntactic resource
| No separate evaluation
+
| 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel [[Lexical-Syntactic rule base - RTE Users|lexical-syntactic rule base]]
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| CERES
+
| Boeing
 
| RTE4
 
| RTE4
| 3.0
+
| 2.0
| Hypernyms, antonyms, indexWords (N,V,Adj,Adv)
+
| Semantic relation between words
| Used, but no evaluation performed
+
| No formal evaluation. Plays a role in most entailments found
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
Line 52: Line 52:
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| BIU
+
| CERES
 
| RTE4
 
| RTE4
 
| 3.0
 
| 3.0
| Derived on the fly lexical entailment rules, using synonyms, hypernyms (up to two levels) and derivations. Also used as part of our novel lexical-syntactic resource
+
| Hypernyms, antonyms, indexWords (N,V,Adj,Adv)
| 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel [[Lexical-Syntactic rule base - RTE Users|lexical-syntactic rule base]]
+
| Used, but no evaluation performed
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| FbkIrst
+
| DFKI
 
| RTE4
 
| RTE4
 
| 3.0
 
| 3.0
| Lexical similarity
+
| Semantic relation between words
| No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical  similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet
+
| No separate evaluation
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| AUEB
+
| DLSIUAES
 
| RTE4
 
| RTE4
 
|  
 
|  
Line 73: Line 73:
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| DLSIUAES
+
| EMORY
 
| RTE4
 
| RTE4
 
|  
 
|  
Line 80: Line 80:
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
| EMORY
+
| FbkIrst
 
| RTE4
 
| RTE4
|  
+
| 3.0
|  
+
| Lexical similarity
| ''Data taken from the RTE4 proceedings. Participants are recommended to add further information.''
+
| No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical  similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
Line 176: Line 176:
 
| Synonymy resolution
 
| Synonymy resolution
 
| Replacing the words of H with their synonyms in T: on RTE3 data sets 2% improvement
 
| Replacing the words of H with their synonyms in T: on RTE3 data sets 2% improvement
 
|- bgcolor="#ECECEC" align="left"
 
| VENSES
 
| RTE3
 
| 3.0
 
| Semantic relation between words
 
| No evaluation of the resource
 
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"
Line 190: Line 183:
 
| Semantic distance between words
 
| Semantic distance between words
 
|  
 
|  
 +
 +
|- bgcolor="#ECECEC" align="left"
 +
| VENSES
 +
| RTE3
 +
| 3.0
 +
| Semantic relation between words
 +
| No evaluation of the resource
  
 
|- bgcolor="#ECECEC" align="left"
 
|- bgcolor="#ECECEC" align="left"

Revision as of 07:15, 26 November 2009

When not otherwise specified, the data about version, usage and evaluation of the resource have been provided by participants themselves.

Participants* Campaign Version Specific usage description Evaluations / Comments
AUEB RTE5 During the calculation of the similarity measures we treat words from T and H that are synonyms according to WordNet as identical. Ablation test performed. Negative impact of the resource: -2% accuracy on two-way, -2.67% on three-way task.
BIU RTE5 3.0 Synonyms, hyponyms (2 levels away from the original term), the hyponym_instance relation and derivations. Ablation test performed. Positive impact of the resource: 2.5% accuracy on two-way task.
AUEB RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
BIU RTE4 3.0 Derived on the fly lexical entailment rules, using synonyms, hypernyms (up to two levels) and derivations. Also used as part of our novel lexical-syntactic resource 0.8% improvement in ablation test on RTE-4. Potential contribution is higher since this resource partially overlaps with the novel lexical-syntactic rule base
Boeing RTE4 2.0 Semantic relation between words No formal evaluation. Plays a role in most entailments found
Cambridge RTE4 3.0 Meaning postulates from WordNet noun hyponymy, e.g. forall x: cat(x) -> animal(x) No systematic evaluation
CERES RTE4 3.0 Hypernyms, antonyms, indexWords (N,V,Adj,Adv) Used, but no evaluation performed
DFKI RTE4 3.0 Semantic relation between words No separate evaluation
DLSIUAES RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
EMORY RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
FbkIrst RTE4 3.0 Lexical similarity No precise evaluation of the resource has been carried out. In our second run we used a combined system (EDITSneg + EDITSallbutneg), and we had an improvement of 0.6% in accuracy with respect to the first run in which only EDITSneg was used. EDITSallbutneg exploits lexical similarity (WordNet similarity), but we can’t affirm with precision that the improvement is due only to the use of WordNet
FSC RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
IIT RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
IPD RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
OAQA RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
QUANTA RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
SAGAN RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
Stanford RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
UAIC RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
UMD RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
UNED RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
Uoeltg RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
UPC RTE4 Data taken from the RTE4 proceedings. Participants are recommended to add further information.
AUEB RTE3 2.1 Synonymy resolution Replacing the words of H with their synonyms in T: on RTE3 data sets 2% improvement
UIUC RTE3 Semantic distance between words
VENSES RTE3 3.0 Semantic relation between words No evaluation of the resource
New user Participants are encouraged to contribute.
Total: 24


[*] For further information about participants, click here: RTE Challenges - Data about participants

   Return to RTE Knowledge Resources