MUC-7 (State of the art)
Jump to navigation
Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
- Performance measure: F = 2 * Precision * Recall / (Recall + Precision)
- Precision: percentage of named entities found by the algorithm that are correct
- Recall: percentage of named entities defined in the corpus that were found by the program
- Exact calculation of precision and recall is explained in the MUC scoring software
- Training data: Training section of MUC-7 dataset
- Dryrun data: Dryrun section of MUC-7 dataset
- Testing data: Formal section of MUC-7 dataset
Table of results
| System name | Short description | System type (1) | Main publications | Software | Results |
|---|---|---|---|---|---|
| Annotator | Human annotator | - | MUC-7 proceedings | - | 97.60% |
| LTG | Best MUC-7 participant | H | Mikheev, Grover and Moens (1998) | - | 93.39% |
| Balie | Unsupervised approach: no prior training | U | Nadeau, Turney and Matwin (2006) | sourceforge.net | 77.71% (2) |
| Baseline | Vocabulary transfer from training to testing | S | Whitelaw and Patrick (2003) | - | 58.89% (2) |
- (1) System type: R = hand-crafted rules, S = supervised learning, U = unsupervised learning, H = hybrid
- (2) Calculated on Enamex types only.
References
Mikheev, A., Grover, C. and Moens, M. (1998). Description of the LTG system used for MUC-7. Proceedings of the Seventh Message Understanding Conference (MUC-7). Fairfax, Virginia.
Nadeau, D., Turney, P. D. and Matwin, S. (2006) Unsupervised Named-Entity Recognition: Generating Gazetteers and Resolving Ambiguity. Proceedings 19th Canadian Conference on Artificial Intelligence. Québec, Canada.
Whitelaw, C. and Patrick, J. (2003) Evaluating Corpora for Named Entity Recognition Using Character-Level Features. Proceeding of the 16th Australian Conference on AI. Perth, Australia.