Difference between revisions of "POS Tagging (State of the art)"

From ACL Wiki
Jump to: navigation, search
(WSJ)
(WSJ)
Line 16: Line 16:
 
! System name
 
! System name
 
! Short description
 
! Short description
! Main publications
+
! Main publication
 
! Software
 
! Software
 +
! Extra Data?***
 
! All tokens
 
! All tokens
 
! Unknown words
 
! Unknown words
Line 25: Line 26:
 
| Brants (2000)
 
| Brants (2000)
 
| [http://www.coli.uni-saarland.de/~thorsten/tnt/ TnT]
 
| [http://www.coli.uni-saarland.de/~thorsten/tnt/ TnT]
 +
| No
 
| 96.46%
 
| 96.46%
 
| 85.86%
 
| 85.86%
Line 32: Line 34:
 
| Tsuruoka, et al (2005)
 
| Tsuruoka, et al (2005)
 
| [http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/ GENiA]
 
| [http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/ GENiA]
 +
| No
 
| 97.05%
 
| 97.05%
 
| Not available
 
| Not available
Line 39: Line 42:
 
| Collins (2002)
 
| Collins (2002)
 
| Not available
 
| Not available
 +
| No
 
| 97.11%
 
| 97.11%
 
| Not available
 
| Not available
Line 46: Line 50:
 
| Tsuruoka and Tsujii (2005)
 
| Tsuruoka and Tsujii (2005)
 
| [http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/postagger/ Easiest-first]
 
| [http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/postagger/ Easiest-first]
 +
| No
 
| 97.15%
 
| 97.15%
 
| Not available
 
| Not available
Line 53: Line 58:
 
| Giménez and Márquez (2004)
 
| Giménez and Márquez (2004)
 
| [http://www.lsi.upc.es/~nlp/SVMTool/ SVMTool]
 
| [http://www.lsi.upc.es/~nlp/SVMTool/ SVMTool]
 +
| No
 
| 97.16%
 
| 97.16%
 
| 89.01%
 
| 89.01%
 +
|-
 +
| Morče/COMPOST
 +
| Averaged Perceptron
 +
| Spoustová et al. (2009)
 +
| [http://ufal.mff.cuni.cz/compost]
 +
| No
 +
| 97.23%
 +
| Not available
 
|-
 
|-
 
| Stanford Tagger 1.0
 
| Stanford Tagger 1.0
Line 60: Line 74:
 
| Toutanova et al. (2003)
 
| Toutanova et al. (2003)
 
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]
 
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]
 +
| No
 
| 97.24%
 
| 97.24%
 
| 89.04%
 
| 89.04%
Line 65: Line 80:
 
| Stanford Tagger 2.0
 
| Stanford Tagger 2.0
 
| Maximum entropy cyclic dependency network
 
| Maximum entropy cyclic dependency network
 +
| Manning (2011)
 
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]
 
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]
 +
| No
 +
| 97.29%
 +
| 89.70%
 +
|-
 +
| Stanford Tagger 2.0
 +
| Maximum entropy cyclic dependency network
 +
| Manning (2011)
 
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]
 
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]
 +
| Yes
 
| 97.32%
 
| 97.32%
 
| 90.79%
 
| 90.79%
 
|-
 
|-
 
| LTAG-spinal
 
| LTAG-spinal
| bidirectional perceptron learning
+
| Bidirectional perceptron learning
 
| Shen et al. (2007)
 
| Shen et al. (2007)
 
| [http://www.cis.upenn.edu/~xtag/spinal/ LTAG-spinal]
 
| [http://www.cis.upenn.edu/~xtag/spinal/ LTAG-spinal]
 +
| No
 
| 97.33%
 
| 97.33%
 +
| Not available
 +
|-
 +
| Morče/COMPOST
 +
| Averaged Perceptron
 +
| Spoustová et al. (2009)
 +
| [http://ufal.mff.cuni.cz/compost]
 +
| Yes
 +
| 97.44%
 
| Not available
 
| Not available
 
|-
 
|-
 
| SCCN
 
| SCCN
| semi-supervised CNN
+
| Semi-supervised condensed nearest neighbor
 
| Søgaard (2011)
 
| Søgaard (2011)
 
| [http://cst.dk/anders/scnn/ SCCN]
 
| [http://cst.dk/anders/scnn/ SCCN]
 +
| Yes
 
| 97.50%
 
| 97.50%
| n/a
+
| Not available
 
|}
 
|}
  
Line 88: Line 122:
  
 
(**) GENiA: Results are for models trained and tested on the given corpora (to be comparable to other results).  The distributed GENiA tagger is trained on a mixed training corpus and gets 96.94% on WSJ, and 98.26% on GENiA biomedical English.
 
(**) GENiA: Results are for models trained and tested on the given corpora (to be comparable to other results).  The distributed GENiA tagger is trained on a mixed training corpus and gets 96.94% on WSJ, and 98.26% on GENiA biomedical English.
 +
 +
(***) Extra data: Whether system training exploited (usually large amounts of) extra unlabeled text, such as by semi-supervised learning, self-training, or using distributional similarity features, beyond the standard supervised training data.
  
 
== References ==
 
== References ==

Revision as of 09:44, 13 September 2011

Test collections

  • Performance measure: per token accuracy. (The convention is for this to be measured on all tokens, including punctuation tokens and other unambiguous tokens.)
  • English
    • Penn Treebank Wall Street Journal (WSJ). The splits of data for this data set were not standardized early on (unlike for parsing) and early work uses various data splits defined by counts of tokens or by sections. Most work from 2002 on adopts the following data splits, introduced by Collins (2002):
      • Training data: sections 0-18
      • Development test data: sections 19-21
      • Testing data: sections 22-24


Tables of results

WSJ

System name Short description Main publication Software Extra Data?*** All tokens Unknown words
TnT* Hidden markov model Brants (2000) TnT No 96.46% 85.86%
GENiA Tagger** Maximum entropy cyclic dependency network Tsuruoka, et al (2005) GENiA No 97.05% Not available
Averaged Perceptron Averaged Perception discriminative sequence model Collins (2002) Not available No 97.11% Not available
Maxent easiest-first Maximum entropy bidirectional easiest-first inference Tsuruoka and Tsujii (2005) Easiest-first No 97.15% Not available
SVMTool SVM-based tagger and tagger generator Giménez and Márquez (2004) SVMTool No 97.16% 89.01%
Morče/COMPOST Averaged Perceptron Spoustová et al. (2009) [1] No 97.23% Not available
Stanford Tagger 1.0 Maximum entropy cyclic dependency network Toutanova et al. (2003) Stanford Tagger No 97.24% 89.04%
Stanford Tagger 2.0 Maximum entropy cyclic dependency network Manning (2011) Stanford Tagger No 97.29% 89.70%
Stanford Tagger 2.0 Maximum entropy cyclic dependency network Manning (2011) Stanford Tagger Yes 97.32% 90.79%
LTAG-spinal Bidirectional perceptron learning Shen et al. (2007) LTAG-spinal No 97.33% Not available
Morče/COMPOST Averaged Perceptron Spoustová et al. (2009) [2] Yes 97.44% Not available
SCCN Semi-supervised condensed nearest neighbor Søgaard (2011) SCCN Yes 97.50% Not available

(*) TnT: Accuracy is as reported by Giménez and Márquez (2004) for the given test collection. Brants (2000) reports 96.7% token accuracy and 85.5% unknown word accuracy on a 10-fold cross-validation of the Penn WSJ corpus.

(**) GENiA: Results are for models trained and tested on the given corpora (to be comparable to other results). The distributed GENiA tagger is trained on a mixed training corpus and gets 96.94% on WSJ, and 98.26% on GENiA biomedical English.

(***) Extra data: Whether system training exploited (usually large amounts of) extra unlabeled text, such as by semi-supervised learning, self-training, or using distributional similarity features, beyond the standard supervised training data.

References

  • Søgaard, Anders. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). Portland, Oregon

See also