RTE5 - Ablation Tests

Ablated Resource	Team Run	Relative accuracy - 2way	Relative accuracy - 3way	Resource Usage Description
Acronym guide	Siel_093.3way	0	0	Acronym Resolution
Acronym guide + UAIC_Acronym_rules	UAIC20091.3way	0.0017	0.0016	We start from acronym-guide, but additional we use a rule that consider for expressions like Xaaaa Ybbbb Zcccc the acronym XYZ, regardless of length of text with this form.
DIRT	BIU1.2way	0.0133		Inference rules
DIRT	Boeing3.3way	-0.0117	0
DIRT	UAIC20091.3way	0.0017	0.0033	We transform text and hypothesis with MINIPAR into dependency trees: use of DIRT relations to map verbs in T with verbs in H
Framenet	DLSIUAES1.2way	0.0116		frame-to-frame similarity metric
Framenet	DLSIUAES1.3way	-0.0017	-0.0017	frame-to-frame similarity metric
Framenet	UB.dmirg3.2way	0
Grady Ward’s MOBY Thesaurus + Roget's Thesaurus	VensesTeam2.2way	0.0283		Semantic fields are used as semantic similarity matching, in all cases of non identical lemmas
MontyLingua Tool	Siel_093.3way	0	0	For the VerbOcean, the verbs have to be in the base form. We used the "MontyLingua" tool to convert the verbs into their base form
NEGATION_rules by UAIC	UAIC20091.3way	0	-0.0134	Negation rules check in the dependency trees on verbs descending branches to see if some categories of words that change the meaning are found.
NER	UI_ccg1.2way	0.0483		Named Entity recognition/comparison
PropBank	cswhu1.3way	0.0200	0.0317	syntactic and semantic parsing
Stanford NER	QUANTA1.2way	0.0067		We use Named Entity similarity as a feature
Stopword list	FBKirst1.2way	0.0150	-0.1028
Training data from RTE1, 2, 3	PeMoZa3.2way	0
Training data from RTE1, 2, 3	PeMoZa3.2way	0
Training data from RTE2	PeMoZa3.2way	0.0066
Training data from RTE2, 3	PeMoZa3.2way	0
VerbOcean	DFKI1.3way	0	0.0017
VerbOcean	DFKI2.3way	0.0033	0.0050
VerbOcean	DFKI3.3way	0.0017	0.0017
VerbOcean	FBKirst1.2way	-0.0016	-0.1028	Rules extracted from VerbOcean
VerbOcean	QUANTA1.2way	0		We use "opposite-of" relation in VerbOcean as a feature
VerbOcean	Siel_093.3way	0	0	Similarity/anthonymy/unrelatedness between verbs
WikiPedia	BIU1.2way	-0.0100		Lexical rules extracted from Wikipedia definition sentences, title parenthesis, redirect and hyperlink relations
WikiPedia	cswhu1.3way	0.0133	0.0334	Lexical semantic rules
WikiPedia	FBKirst1.2way	0.0100		Rules extracted from WP using Latent Semantic Analysis (LSA)
WikiPedia	UAIC20091.3way	0.0117	0.0150	Relations between named entities
Wikipedia + NER's (LingPipe, GATE) + Perl patterns	UAIC20091.3way	0.0617	0.0500	NE module: NERs, in order to identify Persons, Locations, Jobs, Languages, etc; Perl patterns built by us for RTE4 in order to identify numbers and dates; our own resources extracted from Wikipedia in order to identify a "distance" between one name entity from hypothesis and name entities from text