Difference between revisions of "RTE5 - Ablation Tests"

From ACL Wiki
Jump to navigation Jump to search
Line 4: Line 4:
 
! Ablated Resource
 
! Ablated Resource
 
! Team Run
 
! Team Run
−
! Relative accuracy - 2way
+
! <small>Relative accuracy - 2way</small>
−
! Relative accuracy - 3way
+
! <small>Relative accuracy - 3way</small>
 
! Resource Usage Description
 
! Resource Usage Description
  
Line 217: Line 217:
 
| style="text-align: right;"| 0.0500
 
| style="text-align: right;"| 0.0500
 
| NE module: NERs, in order to identify Persons, Locations, Jobs, Languages, etc; Perl patterns built by us for RTE4 in order to identify numbers and dates; our own resources extracted from Wikipedia in order to identify a "distance" between one name entity from hypothesis and name entities from text
 
| NE module: NERs, in order to identify Persons, Locations, Jobs, Languages, etc; Perl patterns built by us for RTE4 in order to identify numbers and dates; our own resources extracted from Wikipedia in order to identify a "distance" between one name entity from hypothesis and name entities from text
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| AUEBNLP1.3way
 +
| style="text-align: right;"| -0.0200
 +
| style="text-align: right;"| -0.0267
 +
| Synonyms
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| BIU1.2way
 +
| style="text-align: right;"| 0.0250
 +
| style="text-align: right;"|
 +
| Synonyms, hyponyms (2 levels away from the original term), hyponym_instance and derivations
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| Boeing3.3way
 +
| style="text-align: right;"| 0.0400
 +
| style="text-align: right;"| 0.0567
 +
|
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| DFKI1.3way
 +
| style="text-align: right;"| -0.0017
 +
| style="text-align: right;"| 0
 +
|
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| DFKI2.3way
 +
| style="text-align: right;"| 0.0016
 +
| style="text-align: right;"| 0.0034
 +
|
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| DFKI3.3way
 +
| style="text-align: right;"| 0.0017
 +
| style="text-align: right;"| 0.0017
 +
|
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| DLSIUAES1.2way
 +
| style="text-align: right;"| 0.0083
 +
| style="text-align: right;"|
 +
| Similarity between lemmata, computed by WordNet-based metrics
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| DLSIUAES1.3way
 +
| style="text-align: right;"| -0.0050
 +
| style="text-align: right;"| -0.0033
 +
| Similarity between lemmata, computed by WordNet-based metrics
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| JU_CSE_TAC1.2way
 +
| style="text-align: right;"| 0.0034
 +
| style="text-align: right;"|
 +
| WordNet based Unigram match
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| PeMoZa1.2way
 +
| style="text-align: right;"| -0.0050
 +
| style="text-align: right;"|
 +
| Derivational Morphology from WordNet
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| PeMoZa1.2way
 +
| style="text-align: right;"| 0.0133
 +
| style="text-align: right;"|
 +
| Verb Entailment from Wordnet
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| PeMoZa2.2way
 +
| style="text-align: right;"| 0.0100
 +
| style="text-align: right;"|
 +
| Derivational Morphology from WordNet
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| PeMoZa2.2way
 +
| style="text-align: right;"| -0.0033
 +
| style="text-align: right;"|
 +
| Verb Entailment from Wordnet
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| QUANTA1.2way
 +
| style="text-align: right;"| -0.0017
 +
| style="text-align: right;"|
 +
| We use several relations from wordnet, such as synonyms, hyponym, hypernym et al.
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| Sagan1.3way
 +
| style="text-align: right;"| 0
 +
| style="text-align: right;"| -0.0083
 +
| The system is based on machine learning approach. The ablation test was obtained with 2 less features using WordNet in the training and testing steps.
 +
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| Siel_093.3way
 +
| style="text-align: right;"| 0.0034
 +
| style="text-align: right;"| -0.0017
 +
| Similarity between nouns using WN tool
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
| ssl1.3way
 +
| style="text-align: right;"| 0
 +
| style="text-align: right;"| 0.0067
 +
| WordNet Analysis
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
|
 +
| style="text-align: right;"|
 +
| style="text-align: right;"|
 +
|
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
|
 +
| style="text-align: right;"|
 +
| style="text-align: right;"|
 +
|
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
|
 +
| style="text-align: right;"|
 +
| style="text-align: right;"|
 +
|
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
|
 +
| style="text-align: right;"|
 +
| style="text-align: right;"|
 +
|
 +
 +
|- bgcolor="#ECECEC" "align="left"
 +
| WordNet
 +
|
 +
| style="text-align: right;"|
 +
| style="text-align: right;"|
 +
|
  
 
|}
 
|}

Revision as of 08:29, 24 November 2009

Ablated Resource Team Run Relative accuracy - 2way Relative accuracy - 3way Resource Usage Description
Acronym guide Siel_093.3way 0 0 Acronym Resolution
Acronym guide +
UAIC_Acronym_rules 		UAIC20091.3way 0.0017 0.0016 We start from acronym-guide, but additional we use a rule that consider for expressions like Xaaaa Ybbbb Zcccc the acronym XYZ, regardless of length of text with this form.
DIRT BIU1.2way 0.0133 Inference rules
DIRT Boeing3.3way -0.0117 0
DIRT UAIC20091.3way 0.0017 0.0033 We transform text and hypothesis with MINIPAR into dependency trees: use of DIRT relations to map verbs in T with verbs in H
Framenet DLSIUAES1.2way 0.0116 frame-to-frame similarity metric
Framenet DLSIUAES1.3way -0.0017 -0.0017 frame-to-frame similarity metric
Framenet UB.dmirg3.2way 0
Grady Ward’s MOBY Thesaurus +
Roget's Thesaurus 		VensesTeam2.2way 0.0283 Semantic fields are used as semantic similarity matching, in all cases of non identical lemmas
MontyLingua Tool Siel_093.3way 0 0 For the VerbOcean, the verbs have to be in the base form. We used the "MontyLingua" tool to convert the verbs into their base form
NEGATION_rules by UAIC UAIC20091.3way 0 -0.0134 Negation rules check in the dependency trees on verbs descending branches to see if some categories of words that change the meaning are found.
NER UI_ccg1.2way 0.0483 Named Entity recognition/comparison
PropBank cswhu1.3way 0.0200 0.0317 syntactic and semantic parsing
Stanford NER QUANTA1.2way 0.0067 We use Named Entity similarity as a feature
Stopword list FBKirst1.2way 0.0150 -0.1028
Training data from RTE1, 2, 3 PeMoZa3.2way 0
Training data from RTE1, 2, 3 PeMoZa3.2way 0
Training data from RTE2 PeMoZa3.2way 0.0066
Training data from RTE2, 3 PeMoZa3.2way 0
VerbOcean DFKI1.3way 0 0.0017
VerbOcean DFKI2.3way 0.0033 0.0050
VerbOcean DFKI3.3way 0.0017 0.0017
VerbOcean FBKirst1.2way -0.0016 -0.1028 Rules extracted from VerbOcean
VerbOcean QUANTA1.2way 0 We use "opposite-of" relation in VerbOcean as a feature
VerbOcean Siel_093.3way 0 0 Similarity/anthonymy/unrelatedness between verbs
WikiPedia BIU1.2way -0.0100 Lexical rules extracted from Wikipedia definition sentences, title parenthesis, redirect and hyperlink relations
WikiPedia cswhu1.3way 0.0133 0.0334 Lexical semantic rules
WikiPedia FBKirst1.2way 0.0100 Rules extracted from WP using Latent Semantic Analysis (LSA)
WikiPedia UAIC20091.3way 0.0117 0.0150 Relations between named entities
Wikipedia +
NER's (LingPipe, GATE) +
Perl patterns 		UAIC20091.3way 0.0617 0.0500 NE module: NERs, in order to identify Persons, Locations, Jobs, Languages, etc; Perl patterns built by us for RTE4 in order to identify numbers and dates; our own resources extracted from Wikipedia in order to identify a "distance" between one name entity from hypothesis and name entities from text
WordNet AUEBNLP1.3way -0.0200 -0.0267 Synonyms
WordNet BIU1.2way 0.0250 Synonyms, hyponyms (2 levels away from the original term), hyponym_instance and derivations
WordNet Boeing3.3way 0.0400 0.0567
WordNet DFKI1.3way -0.0017 0
WordNet DFKI2.3way 0.0016 0.0034
WordNet DFKI3.3way 0.0017 0.0017
WordNet DLSIUAES1.2way 0.0083 Similarity between lemmata, computed by WordNet-based metrics
WordNet DLSIUAES1.3way -0.0050 -0.0033 Similarity between lemmata, computed by WordNet-based metrics
WordNet JU_CSE_TAC1.2way 0.0034 WordNet based Unigram match
WordNet PeMoZa1.2way -0.0050 Derivational Morphology from WordNet
WordNet PeMoZa1.2way 0.0133 Verb Entailment from Wordnet
WordNet PeMoZa2.2way 0.0100 Derivational Morphology from WordNet
WordNet PeMoZa2.2way -0.0033 Verb Entailment from Wordnet
WordNet QUANTA1.2way -0.0017 We use several relations from wordnet, such as synonyms, hyponym, hypernym et al.
WordNet Sagan1.3way 0 -0.0083 The system is based on machine learning approach. The ablation test was obtained with 2 less features using WordNet in the training and testing steps.


WordNet Siel_093.3way 0.0034 -0.0017 Similarity between nouns using WN tool
WordNet ssl1.3way 0 0.0067 WordNet Analysis
WordNet
WordNet
WordNet
WordNet
WordNet
Retrieved from "https://aclweb.org/aclwiki/index.php?title=RTE5_-_Ablation_Tests&oldid=7370"