Modular Approach to Error Analysis and Evaluation for Multilingual Question Answering

Hideki Shima, Mengqiu Wang, Frank Lin, Teruko Mitamura


Abstract
Multilingual Question Answering systems are generally very complex, integrating several sub-modules to achieve their result. Global metrics (such as average precision and recall) are insufficient when evaluating the performance of individual sub-modules and their influence on each other. In this paper, we present a modular approach to error analysis and evaluation; we use manually-constructed, gold-standard input for each module to obtain an upper-bound for the (local) performance of that module. This approach enables us to identify existing problem areas quickly, and to target improvements accordingly.
Anthology ID:
L06-1489
Volume:
Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)
Month:
May
Year:
2006
Address:
Genoa, Italy
Editors:
Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, Bente Maegaard, Joseph Mariani, Jan Odijk, Daniel Tapias
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2006/pdf/782_pdf.pdf
DOI:
Bibkey:
Cite (ACL):
Hideki Shima, Mengqiu Wang, Frank Lin, and Teruko Mitamura. 2006. Modular Approach to Error Analysis and Evaluation for Multilingual Question Answering. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA).
Cite (Informal):
Modular Approach to Error Analysis and Evaluation for Multilingual Question Answering (Shima et al., LREC 2006)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2006/pdf/782_pdf.pdf