https://aclweb.org/aclwiki/api.php?action=feedcontributions&user=Kiwibird&feedformat=atomACL Wiki - User contributions [en]2024-03-19T11:32:31ZUser contributionsMediaWiki 1.35.2https://aclweb.org/aclwiki/index.php?title=Resources_for_Guarani&diff=11250Resources for Guarani2015-10-22T06:32:57Z<p>Kiwibird: Created page with "* https://code.google.com/p/hltdi-l3/source/browse/#hg%2Fdicts bilingual dictionaries es-gn, GPL3"</p>
<hr />
<div>* https://code.google.com/p/hltdi-l3/source/browse/#hg%2Fdicts bilingual dictionaries es-gn, GPL3</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Norwegian&diff=11206Resources for Norwegian2015-09-01T18:10:56Z<p>Kiwibird: /* Free software */</p>
<hr />
<div>==Corpora==<br />
===Free software===<br />
* http://www.nb.no/sprakbanken/show?serial=sbr-36&lang=nn CC-BY EU-corpora (Acquis Communautaire), translations in tmx format from English to Nynorsk (52527 tu's) and Bokmål (733081 tu's)<br />
<br />
===Proprietary===<br />
* [http://corpora.informatik.uni-leipzig.de/ Norwegian plain text and Co-occurrences at LCC] ("the corpora may be used for scientific purposes only and not passed on to third parties")<br />
<br />
==Timeline Analysis==<br />
* [http://wortschatz.uni-leipzig.de/wdtno/ Ord I Dag]<br />
<br />
==Machine translation systems==<br />
<br />
===Free software===<br />
<br />
* [http://www.apertium.org Apertium] Norwegian Nynorsk<->Norwegian Bokmål, GPL v2<br />
** [http://wiki.apertium.org/wiki/Apertium-nn-nb wiki] with installation information etc.<br />
<br />
===Proprietary===<br />
<br />
==Lexical resources==<br />
===Free software===<br />
* [http://svn.emmtee.net/tags/topp/parc/pargram/norwegian/bokmal/bokmal-nkllex.lfg Bokmål LFG lexicon] with POS and count/mass, GPL<br />
* [http://www.edd.uio.no/prosjekt/ordbanken/ Norsk ordbank], full form dictionaries for Nynorsk (106,789 lemmata) and Bokmål (142,899 lemmata), GPL <br />
** [http://savannah.nongnu.org/projects/ordbanken/ alternative download with cli lookup interface]<br />
* [http://www.nb.no/spraakbanken/tilgjengelege-ressursar/leksikalske-databasar SCARRIE, Bokmål full form dictionary], XML, about 75,000 lemmata, CC-BY unported<br />
<br />
===Unknown license===<br />
* [http://www.nb.no/spraakbanken/tilgjengelege-ressursar/leksikalske-databasar "Leksikalsk database for norsk, opphavleg produsert av NST"], lexical database with SAMPA transcriptions, meant for speech technology<br />
<br />
==Parsing/disambiguation==<br />
===Free software===<br />
* [http://www.hf.uio.no/tekstlab/tagger.html Oslo-Bergen-taggeren], [[Constraint Grammar]] disambiguator, GPL<br />
** [https://github.com/noklesta/The-Oslo-Bergen-Tagger source and packages on github]<br />
** [http://maximos.aksis.uib.no/Aksis-wiki/Oslo-Bergen_Tagger older alternative download site]<br />
** [http://apertium.svn.sourceforge.net/viewvc/apertium/trunk/apertium-nn-nb/ the version used in Apertium]<br />
** [https://github.com/ogrim/clj-obt Clojure bindings]<br />
<br />
<br />
* [http://www.hf.ntnu.no/hf/isk/Ansatte/petter.haugereid/norsyg.html Norsyg], [[HPSG]] grammar for Norwegian bokmål, LGPL. Implemented in [[LKB]], works with the full ''Norsk ordbank'' lexicon.<br />
<br />
[[Category:Resources by language|Norwegian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Persian&diff=11186Resources for Persian2015-08-11T18:45:04Z<p>Kiwibird: /* Morphology tools */</p>
<hr />
<div>== Corpora ==<br />
===Free===<br />
*[http://www.ling.ohio-state.edu/~jonsafari/corpora VOA Persian Corpus 2003-2008] (public domain)<br />
<br />
===Proprietary===<br />
<!-- Please keep this list in alphabetical order --><br />
*[http://ece.ut.ac.ir/DBRG/Bijankhan/ Bijankhan corpus] (gratis for research/non-commercial purposes)<br />
*[http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC96S50 CALLFRIEND Farsi (speech)], LDC<br />
*[http://ece.ut.ac.ir/dbrg/hamshahri/ Hamshahri corpus] (gratis for research/non-commercial purposes)<br />
*[http://www.elda.org/catalogue/en/speech/S0112.html Persian speech database Farsdat], ELRA<br />
<br />
<br />
==Lexical resources==<br />
===Free===<br />
*[http://www.ling.ohio-state.edu/~jonsafari/corpora/wikipedia_fa-en_20120217.txt.xz Persian - English dictionary], derived from Wikipedia article names. Retains Wikipedia's CC-BY-SA 3.0 license.<br />
<br />
===Proprietary===<br />
*[http://pwn.ir Persian WordNet]<br />
<br />
<br />
==Machine translation==<br />
===Free===<br />
*[http://ece.ut.ac.ir/node/100869?destination=node%2F100869 Tehran English-Persian Parallel Corpus] by Mohammad Taher Pilevar, NLP Lab, University of Tehran. For research or non-commercial use.<br />
<br />
===Proprietary===<br />
*[http://crl.nmsu.edu/Research/Projects/shiraz/index.html The Shiraz project] (Persian -> English)<br />
<br />
==Morphology tools==<br />
===Free===<br />
*[http://sourceforge.net/projects/perstem Perstem] - Persian stemmer, light morphological analyzer, and character set converter.<br />
*[http://apertium.svn.sourceforge.net/svnroot/apertium/incubator/apertium-tg-fa/apertium-tg-fa.fa.dix Morphological dictionary] &mdash; compiled using [[lttoolbox]].<br />
*[http://stp.lingfil.uu.se/~mojgan/ BLARK by Mojgan Seraji] – normaliser, tokeniser, segmentation, hunpos model for PoS-tagging and (java) dependency parser, all GPL<br />
<br />
==Parsing==<br />
===Free===<br />
* [http://ufal.mff.cuni.cz/hamledt HamleDT], harmonized dependency treebanks of many languages, common annotation style.<br />
* [http://www.ling.ohio-state.edu/~jonsafari/persianlg/ Persian dictionaries] for the [http://www.abisource.com/projects/link-grammar/ Link-Grammar parser]. By [http://www.ling.ohio-state.edu/~jonsafari/ Jon Dehdari]. These require the Perstem stemming package, above.<br />
* [http://stp.lingfil.uu.se/~mojgan/UPDT.html Uppsala Persian Dependency Treebank], Creative Commons Attribution 3.0 License<br />
<br />
===Proprietary===<br />
*[http://dadegan.ir/en/persiandependencytreebank Dadegan Dependency Treebank] for research purposes only.<br />
*[http://hpsg.fu-berlin.de/~ghayoomi/PTB.html HPSG Persian Treebank (PerTreeBank)] for academic research purposes only.<br />
<br />
<br />
<br />
==Bibliography==<br />
* Dehdari, Jon, and Deryle Lonsdale. 2008. [http://www.ling.ohio-state.edu/~jonsafari/papers/dehdari_lonsdale_2005.pdf A link grammar parser for Persian]. In Karimi, S., Samiian, V., and Stilo, D., editors, ''Aspects of Iranian Linguistics'', volume 1. Cambridge Scholars Press. ISBN: 978-18-471-8639-3 ([http://www.ling.ohio-state.edu/~jonsafari/bib/dehdarilonsdale2005.bib.txt BIB])<br />
<br />
* Feili, H. and G. Ghassem-Sani (2004) "[http://sharif.edu/~sani/papers/Feili_SaniE2.pdf An Application of Lexicalized Grammars in English-Persian Translation]". ''Proceedings of the 16th European Conference on Artificial Intelligence (ECAI 2004)'', 24-27 Aug. 2004, Universidad Politecnica de Valencia, Valencia, Spain, pp. 596-600.<br />
* Megerdoomian, K. (2000) "[http://crl.nmsu.edu/Research/Projects/shiraz/publications/papers/Cicling.pdf Unification-Based Persian Morphology]". ''Proceedings of CICLing 2000'', Alexander Gelbukh, Center of Investigation on Computation-IPN, Mexico, 2000.<br />
* Megerdoomian, K. (2004) "[http://acl.ldc.upenn.edu/coling2004/W5/pdf/W5-7.pdf Finite-State Morphological Analysis of Persian]". ''COLING 2004 Computational Approaches to Arabic Script-based Languages''. Ali Farghaly and Karine Megerdoomian editors, Geneva, Switzerland, 2004, pgs. 35-41.<br />
* Mohammad Amin Farajian (2011). [http://world-comp.org/p2011/ICA4953.pdf PEN: Parallel English-Persian News Corpus]. Proceedings of 2011 International Conference on Artificial Intelligence (ICAI'11), Nevada, USA.<br />
<br />
==See also==<br />
*[[Resources for Kurdish]]<br />
*[[Resources for Tajik]]<br />
<br />
==External links==<br />
*https://wiki.iranianlinguistics.org/wiki/Main_Page: NLP Resources for Persian]<br />
*[http://www.ling.ohio-state.edu/~jonsafari/persian_nlp.html the Jon safari] (link parser, small lexicon, stemmer, morphological analysis tools)<br />
<br />
<br />
[[Category:Resources by language|Persian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Tatar&diff=11089Resources for Tatar2015-06-19T10:37:54Z<p>Kiwibird: /* Corpora */</p>
<hr />
<div>==Morphology==<br />
<br />
===Free software===<br />
* http://wiki.apertium.org/wiki/Apertium-tat GPL analyser and disambiguator<br />
<br />
===Proprietary===<br />
<br />
==Machine translation==<br />
<br />
===Free software===<br />
* http://wiki.apertium.org/wiki/Kazakh_and_Tatar rule-based, GPL license<br />
* http://wiki.apertium.org/wiki/Tatar_and_Russian rule-based, GPL license<br />
<br />
===Proprietary===<br />
* http://tatar.com.ru/trans.php<br />
<br />
==Corpora==<br />
===Proprietary===<br />
* http://corpus.tatar/index_en.php (so far only freely searchable for non-commercial scientific or educational use)<br />
<br />
===Free/open licence===<br />
<br />
* http://tatoeba.org/deu/sentences/show_all_in/tat/none/none/indifferent Tatoeba sentences in Tatar<br />
<br />
==Bibliography==<br />
<br />
==External links==<br />
<br />
<br />
[[Category:Resources by language|Tatar]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Tatar&diff=11088Resources for Tatar2015-06-19T10:37:29Z<p>Kiwibird: Created page with "==Morphology== ===Free software=== * http://wiki.apertium.org/wiki/Apertium-tat GPL analyser and disambiguator ===Proprietary=== ==Machine translation== ===Free software==..."</p>
<hr />
<div>==Morphology==<br />
<br />
===Free software===<br />
* http://wiki.apertium.org/wiki/Apertium-tat GPL analyser and disambiguator<br />
<br />
===Proprietary===<br />
<br />
==Machine translation==<br />
<br />
===Free software===<br />
* http://wiki.apertium.org/wiki/Kazakh_and_Tatar rule-based, GPL license<br />
* http://wiki.apertium.org/wiki/Tatar_and_Russian rule-based, GPL license<br />
<br />
===Proprietary===<br />
* http://tatar.com.ru/trans.php<br />
<br />
==Corpora==<br />
===Proprietary===<br />
* http://corpus.tatar/index_en.php (so far only freely searchable for non-commercial scientific or educational use)<br />
<br />
===Free/open licence===<br />
<br />
==Bibliography==<br />
<br />
==External links==<br />
<br />
<br />
[[Category:Resources by language|Tatar]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Arabic&diff=11028Resources for Arabic2015-04-20T09:21:26Z<p>Kiwibird: /* Morphology */</p>
<hr />
<div>==Morphology==<br />
<br />
===Free software===<br />
*[https://sourceforge.net/projects/aramorph/ AraMorph - Perl] - An Arabic morphological analyzer and part-of-speech tagger written in Perl (originally by Tim Buckwalter)<br />
*[http://www.nongnu.org/aramorph/ AraMorph - Java] - An Arabic morphological analyzer and part-of-speech tagger rewritten in Java for [http://lucene.apache.org/ Lucene]<br />
<br />
===Proprietary===<br />
*[http://www.arabic-morphology.com Xerox Arabic Morphological Analyzer and Generator]<br />
<br />
==WordNets==<br />
<br />
===Free software===<br />
* http://compling.hss.ntu.edu.sg/omw/ Hebrew Wordnet with links to all the other Open Multilingual Wordnets<br />
<br />
===Proprietary===<br />
* http://babelnet.org/ (available for download for "Non-Commercial" use)<br />
<br />
==Parsers==<br />
===Free software===<br />
* [http://www.cis.upenn.edu/~dbikel/software.html#stat-parser Bikel's implementation of Collins Parser] by [http://www.cis.upenn.edu/~dbikel/ Dan Bikel].<br />
* [http://www.ling.ohio-state.edu/~jonsafari/arabiclg/arabiclg.20060829.tar.bz2 Arabic dictionaries], by [http://www.ling.ohio-state.edu/~jonsafari/ Jon Dehdari], for the [http://www.abisource.com/projects/link-grammar/ Link-Grammar parser]. These require the Aramorph stemming package, above. <br />
* [https://sourceforge.net/apps/trac/elixir-fm/wiki ElixirFM] ([http://quest.ms.mff.cuni.cz/cgi-bin/elixir/index.fcgi online interface here]) is a Functional Arabic Morphology written in Haskell and Perl; the lexicon is a "re-processed" version of the Buckwalter analyser.<br />
* [http://sourceforge.net/projects/sarf Sarf] - Arabic Morphology System (all in Java)<br />
<br />
==Corpora==<br />
===Proprietary===<br />
*[http://www.ldc.upenn.edu/Catalog/LDC2001T55.html Arabic Newswire Part 1], 76 million tokens, annotation: paragraphs<br />
<br />
===Free/open licence===<br />
* [http://github.com/anastaw/Meedan-Memory Meedan-Memory], Arabic-English TMX (sentence-aligned), ~467,000 words on the English side, [http://www.opendatacommons.org/licenses/odbl/ Open Database Licence]<br />
* [http://quran.uk.net/ Quranic Arabic Corpus], 77,430 words of Quranic Arabic, with manually verified contextual POS, inflection, derivation; [[dependency grammar]] annotation is planned.<br />
* [http://www1.ccls.columbia.edu/~ybenajiba/downloads.html Arabic NER corpora] by [http://www1.ccls.columbia.edu/~ybenajiba/ Yassine Benajiba], 150,000+ words.<br />
* [http://www.euromatrixplus.net/multi-un/ UN parallel corpora]<br />
* [http://ufal.mff.cuni.cz/hamledt HamleDT], harmonized dependency treebanks of many languages, common annotation style.<br />
<br />
==Bibliography==<br />
<br />
==External links==<br />
*[http://www.elsnet.org/acl2001-arabic.html ACL/EACL 2001 Workshop on Arabic NLP]<br />
*[http://www1.cs.columbia.edu/~mdiab/software/ASVMTools_2.0.tar.gz Basic Arabic Processing Tools]<br />
*[http://acl.ldc.upenn.edu/coling2004/W5/index.html COLING 2004 Workshop on computational approaches to Arabic script-based languages]<br />
<br />
<br />
[[Category:Resources by language|Arabic]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Hebrew&diff=11027Resources for Hebrew2015-04-20T09:20:02Z<p>Kiwibird: </p>
<hr />
<div>* [http://www.ivrix.org.il/projects/spell-checker/ Hebrew Spellchecker]<br />
* [http://search.cpan.org/dist/Lingua-HE-Sentence/ Lingua-HE-Sentence-0.13] - Perl module for splitting Hebrew text into sentences<br />
* http://compling.hss.ntu.edu.sg/omw/ Hebrew Wordnet with links to all the other Open Multilingual Wordnets<br />
<br />
[[Category:Resources by language|Hebrew]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Hebrew&diff=11026Resources for Hebrew2015-04-20T09:19:48Z<p>Kiwibird: </p>
<hr />
<div>* [http://www.ivrix.org.il/projects/spell-checker/ Hebrew Spellchecker]<br />
* [http://search.cpan.org/dist/Lingua-HE-Sentence/ Lingua-HE-Sentence-0.13] - Perl module for splitting Hebrew text into sentences<br />
* http://compling.hss.ntu.edu.sg/omw/ Hebrew Wordnet with links to all the other Open Wordnets<br />
<br />
[[Category:Resources by language|Hebrew]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Thai&diff=10630Resources for Thai2014-04-10T11:51:40Z<p>Kiwibird: Created page with " https://github.com/veer66/Yaitron Yaitron English-Thai and Thai-English XML dictionary, license seems standard 4-clause"</p>
<hr />
<div> https://github.com/veer66/Yaitron Yaitron English-Thai and Thai-English XML dictionary, license seems standard 4-clause</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=List_of_resources_by_language&diff=10629List of resources by language2014-04-10T11:51:16Z<p>Kiwibird: /* T */</p>
<hr />
<div>List of pages which give links and commentary on computational resources by language.<br />
<br />
Quick links:<br />
<br />
* [[Resources for English]]<br />
* [[Multilingual resources|Resources for Multilingual Applications]]<br />
<br />
See also:<br />
<br />
* [http://www.ethnologue.com/ Ethnologue: Languages of the World]<br />
* [[Language Identification Tools]]<br />
<br />
<br />
==A==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Albanian]]<br />
* [[Resources for Amharic]]<br />
* [[Resources for Arabic]]<br />
* [[Resources for Afrikaans]]<br />
<br />
==B==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Basque]]<br />
* [[Resources for Bulgarian]]<br />
* [[Resources for Breton]]<br />
<br />
==C==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Catalan]]<br />
* [[Resources for Chinese]]<br />
* [[Resources for Croatian]] (see also [[Resources for Serbian]], [[Resources for Bosnian]], [[Resources for Serbo-Croatian]])<br />
* [[Resources for Czech]]<br />
<br />
==D==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Danish]]<br />
* [[Resources for Dutch]]<br />
<br />
==E==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for English]]<br />
* [[Resources for Esperanto]]<br />
* [[Resources for Estonian]]<br />
<br />
==F==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Faroese]]<br />
* [[Resources for Finnish]]<br />
* [[Resources for French]]<br />
<br />
==G==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Galician]]<br />
* [[Resources for Georgian]]<br />
* [[Resources for German]]<br />
* [[Resources for Greek]]<br />
* [[Resources for Greenlandic]]<br />
<br />
==H==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Haitian]]<br />
* [[Resources for Hebrew]]<br />
* [[Resources for Hindi]]<br />
* [[Resources for Hungarian]]<br />
<br />
==I==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Icelandic]]<br />
* [[Resources for Indonesian]]<br />
* [[Resources for Inuktitut]]<br />
* [[Resources for Iñupiaq]]<br />
* [[Resources for Iranian]]<br />
* [[Resources for Italian]]<br />
* [[Resources for Irish]]<br />
<br />
==J==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Japanese]]<br />
<br />
==K==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Kannada]]<br />
* [[Resources for Korean]]<br />
* [[Resources for Komi]]<br />
* [[Resources for Kurdish]]<br />
<br />
==L==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Lithuanian]]<br />
<br />
==M==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Macedonian]]<br />
* [[Resources for Malay]]<br />
* [[Resources for Maltese]]<br />
* [[Resources for Montenegrin]]<br />
* [[Multilingual resources|Resources for Multilingual Applications]]<br />
<br />
==N==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Norwegian]]<br />
* [[Resources for Navajo]]<br />
<br />
==O==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Occitan]]<br />
<br />
==P==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Pashto]]<br />
* [[Resources for Persian]]<br />
* [[Resources for Polish]]<br />
* [[Resources for Portugese]]<br />
* [[Resources for Punjabi]]<br />
<br />
==Q==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Quechua]]<br />
<br />
==R==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Romanian]]<br />
* [[Resources for Russian]]<br />
<br />
==S==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Sámi]]<br />
* [[Resources for Sanskrit]]<br />
* [[Resources for Slovak]]<br />
* [[Resources for Slovenian]]<br />
* [[Resources for Sorbian]]<br />
* [[Resources for Spanish]]<br />
* [[Resources for Swahili]]<br />
* [[Resources for Swedish]]<br />
<br />
==T==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Tajik]]<br />
* [[Resources for Turkish]]<br />
* [[Resources for Tigrinya]]<br />
* [[Resources for Telugu]]<br />
* [[Resources for Thai]]<br />
<br />
==U==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Ukrainian]]<br />
* [[Resources for Urdu]]<br />
<br />
==V==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Vietnamese]]<br />
<br />
==W==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Welsh]]<br />
<br />
==Z==<br />
__NOTOC__<br />
{{compactTOC2}}<br />
* [[Resources for Zulu]]<br />
<br />
==See also==<br />
<br />
* [[Resources for African languages]]<br />
<br />
[[Category:Resources by language|*]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Part-of-speech_tagging&diff=9898Part-of-speech tagging2012-12-29T17:25:59Z<p>Kiwibird: /* Software */</p>
<hr />
<div>'''Part-of-speech tagging''' is the task of assigning a part-of-speech tag to each word in a given text.<br />
<br />
==History==<br />
<br />
==Further reading==<br />
<br />
==Software==<br />
*[http://sourceforge.net/projects/acopost ACOPOST] - a collection of taggers using maximum entropy, second order Markov, exemplar, and transformation-based models. See also [http://hermes.sourceforge.net/acopost.html this site]. Free, open source license.<br />
*[http://danieldk.org/Code/Citar Citar] - uses [http://en.wikipedia.org/wiki/Trigram trigram]-based [http://en.wikipedia.org/wiki/Hidden_Markov_model HMM]s. Free, open source license.<br />
*[http://crfpp.sourceforge.net CRF++] - uses [http://en.wikipedia.org/wiki/Conditional_random_field Conditional random fields]. Free, open source license (dual: LGPL, New BSD). C++.<br />
*[http://crftagger.sourceforge.net CRFTagger] - for English. Free, open source license. Java.<br />
*[http://sourceforge.net/projects/gposttl GPoSTTL] - Enhanced [http://en.wikipedia.org/wiki/Brill_tagger TBL] tagger for English. Open source license.<br />
*[http://code.google.com/p/hunpos/ HunPos] - uses trigram-based HMMs. Free, open source license. OCaml.<br />
*[http://cogcomp.cs.illinois.edu/page/software_view/3 Illinois LBJ POS Tagger] - Uses averaged [http://en.wikipedia.org/wiki/Perceptron Perceptron] based sequential model. Java API, Free, open source license.<br />
*[http://ilk.uvt.nl/mbt Memory-based tagger] (MBT) - uses [http://ilk.uvt.nl/timbl TiMBL]. Free, open source license.<br />
*[http://ufal.mff.cuni.cz/morce/index.php Morče] - Uses Averaged [http://en.wikipedia.org/wiki/Perceptron Perceptron] based model. Free, open source license (GPL2).<br />
*[http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger] - uses [http://en.wikipedia.org/wiki/Logistic_regression Maximum entropy models]. Free, open source license. Java.<br />
*[http://www.lsi.upc.es/~nlp/SVMTool SVMTool] - uses [http://en.wikipedia.org/wiki/Support_vector_machine Support vector machines]. Free, open source license, but depends on non-Free/open source [http://svmlight.joachims.org/ SVMlight].<br />
<br />
==See also==<br />
*[[POS Tagging (State of the art)]]<br />
<br />
==External links==<br />
*[http://en.wikipedia.org/wiki/Part-of-speech_tagging Wikipedia article on POS tagging]<br />
<br />
[[Category:Morphology]]<br />
[[Category:Syntax]]<br />
[[Category:Software]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Part-of-speech_tagging&diff=9897Part-of-speech tagging2012-12-29T17:22:43Z<p>Kiwibird: /* Software */</p>
<hr />
<div>'''Part-of-speech tagging''' is the task of assigning a part-of-speech tag to each word in a given text.<br />
<br />
==History==<br />
<br />
==Further reading==<br />
<br />
==Software==<br />
*[http://sourceforge.net/projects/acopost ACOPOST] - a collection of taggers using maximum entropy, second order Markov, exemplar, and transformation-based models. See also [http://hermes.sourceforge.net/acopost.html this site]. Free, open source license.<br />
*[http://danieldk.org/Code/Citar Citar] - uses [http://en.wikipedia.org/wiki/Trigram trigram]-based [http://en.wikipedia.org/wiki/Hidden_Markov_model HMM]s. Free, open source license.<br />
*[http://crfpp.sourceforge.net CRF++] - uses [http://en.wikipedia.org/wiki/Conditional_random_field Conditional random fields]. Free, open source license (dual: LGPL, New BSD). C++.<br />
*[http://crftagger.sourceforge.net CRFTagger] - for English. Free, open source license. Java.<br />
*[http://sourceforge.net/projects/gposttl GPoSTTL] - Enhanced [http://en.wikipedia.org/wiki/Brill_tagger TBL] tagger for English. Open source license.<br />
*[http://code.google.com/p/hunpos/ HunPos] - uses trigram-based HMMs. Free, open source license. OCaml.<br />
*[http://cogcomp.cs.illinois.edu/page/software_view/3 Illinois LBJ POS Tagger] - Uses averaged [http://en.wikipedia.org/wiki/Perceptron Perceptron] based sequential model. Java API, Free, open source license.<br />
*[http://ilk.uvt.nl/mbt Memory-based tagger] (MBT) - uses [http://ilk.uvt.nl/timbl TiMBL]. Free, open source license.<br />
*[http://ufal.mff.cuni.cz/morce/index.php Morče] - Uses Averaged [http://en.wikipedia.org/wiki/Perceptron Perceptron] based model. Free, open source license (GPL2).<br />
*[http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger] - uses [http://en.wikipedia.org/wiki/Logistic_regression Maximum entropy models]. Free, open source license.<br />
*[http://www.lsi.upc.es/~nlp/SVMTool SVMTool] - uses [http://en.wikipedia.org/wiki/Support_vector_machine Support vector machines]. Free, open source license, but depends on non-Free/open source [http://svmlight.joachims.org/ SVMlight].<br />
<br />
==See also==<br />
*[[POS Tagging (State of the art)]]<br />
<br />
==External links==<br />
*[http://en.wikipedia.org/wiki/Part-of-speech_tagging Wikipedia article on POS tagging]<br />
<br />
[[Category:Morphology]]<br />
[[Category:Syntax]]<br />
[[Category:Software]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Part-of-speech_tagging&diff=9896Part-of-speech tagging2012-12-29T17:21:58Z<p>Kiwibird: /* Software */</p>
<hr />
<div>'''Part-of-speech tagging''' is the task of assigning a part-of-speech tag to each word in a given text.<br />
<br />
==History==<br />
<br />
==Further reading==<br />
<br />
==Software==<br />
*[http://sourceforge.net/projects/acopost ACOPOST] - a collection of taggers using maximum entropy, second order Markov, exemplar, and transformation-based models. See also [http://hermes.sourceforge.net/acopost.html this site]. Free, open source license.<br />
*[http://danieldk.org/Code/Citar Citar] - uses [http://en.wikipedia.org/wiki/Trigram trigram]-based [http://en.wikipedia.org/wiki/Hidden_Markov_model HMM]s. Free, open source license.<br />
*[http://crfpp.sourceforge.net CRF++] - uses [http://en.wikipedia.org/wiki/Conditional_random_field Conditional random fields]. Free, open source license. C++.<br />
*[http://crftagger.sourceforge.net CRFTagger] - for English. Free, open source license. Java.<br />
*[http://sourceforge.net/projects/gposttl GPoSTTL] - Enhanced [http://en.wikipedia.org/wiki/Brill_tagger TBL] tagger for English. Open source license.<br />
*[http://code.google.com/p/hunpos/ HunPos] - uses trigram-based HMMs. Free, open source license. OCaml.<br />
*[http://cogcomp.cs.illinois.edu/page/software_view/3 Illinois LBJ POS Tagger] - Uses averaged [http://en.wikipedia.org/wiki/Perceptron Perceptron] based sequential model. Java API, Free, open source license.<br />
*[http://ilk.uvt.nl/mbt Memory-based tagger] (MBT) - uses [http://ilk.uvt.nl/timbl TiMBL]. Free, open source license.<br />
*[http://ufal.mff.cuni.cz/morce/index.php Morče] - Uses Averaged [http://en.wikipedia.org/wiki/Perceptron Perceptron] based model. Free, open source license (GPL2).<br />
*[http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger] - uses [http://en.wikipedia.org/wiki/Logistic_regression Maximum entropy models]. Free, open source license.<br />
*[http://www.lsi.upc.es/~nlp/SVMTool SVMTool] - uses [http://en.wikipedia.org/wiki/Support_vector_machine Support vector machines]. Free, open source license, but depends on non-Free/open source [http://svmlight.joachims.org/ SVMlight].<br />
<br />
==See also==<br />
*[[POS Tagging (State of the art)]]<br />
<br />
==External links==<br />
*[http://en.wikipedia.org/wiki/Part-of-speech_tagging Wikipedia article on POS tagging]<br />
<br />
[[Category:Morphology]]<br />
[[Category:Syntax]]<br />
[[Category:Software]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=POS_Tagging_(State_of_the_art)&diff=9895POS Tagging (State of the art)2012-12-29T17:12:57Z<p>Kiwibird: /* FTB */</p>
<hr />
<div>==Test collections==<br />
* '''Performance measure:''' per token accuracy. (The convention is for this to be measured on all tokens, including punctuation tokens and other unambiguous tokens.)<br />
* '''English'''<br />
** '''Penn Treebank''' ''Wall Street Journal'' (WSJ) release 3 (LDC99T42). The splits of data for this task were not standardized early on (unlike for parsing) and early work uses various data splits defined by counts of tokens or by sections. Most work from 2002 on adopts the following data splits, introduced by Collins (2002):<br />
*** '''Training data:''' sections 0-18<br />
*** '''Development test data:''' sections 19-21<br />
*** '''Testing data:''' sections 22-24<br />
<br />
* '''French'''<br />
** '''French TreeBank''' (FTB, Abeillé et al; 2003) ''Le Monde'', December 2007 version, 28-tag tagset (CC tagset, Crabbé and Candito, 2008). Classical data split (10-10-80):<br />
*** '''Training data:''' sentences 2471 to 12351<br />
*** '''Development test data:''' sentences 1236 to 2470<br />
*** '''Testing data:''' sentences 1 to 1235<br />
<br />
<br />
== Tables of results ==<br />
<br />
===WSJ===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
! License<br />
|-<br />
| TnT*<br />
| Hidden markov model<br />
| Brants (2000)<br />
| [http://www.coli.uni-saarland.de/~thorsten/tnt/ TnT]<br />
| No<br />
| 96.46%<br />
| 85.86%<br />
| Unknown<br />
|-<br />
| MElt<br />
| MEMM with external lexical information<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 96.96%<br />
| 91.29%<br />
| CeCILL-C<br />
|-<br />
| GENiA Tagger**<br />
| Maximum entropy cyclic dependency network<br />
| Tsuruoka, et al (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/ GENiA]<br />
| No<br />
| 97.05%<br />
| Not available<br />
| Gratis for non-commercial usage<br />
|-<br />
| Averaged Perceptron<br />
| Averaged Perception discriminative sequence model<br />
| Collins (2002)<br />
| Not available<br />
| No<br />
| 97.11%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Maxent easiest-first<br />
| Maximum entropy bidirectional easiest-first inference<br />
| Tsuruoka and Tsujii (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/postagger/ Easiest-first]<br />
| No<br />
| 97.15%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SVMTool<br />
| SVM-based tagger and tagger generator<br />
| Giménez and Márquez (2004)<br />
| [http://www.lsi.upc.es/~nlp/SVMTool/ SVMTool]<br />
| No<br />
| 97.16%<br />
| 89.01%<br />
| LGPL 2.1<br />
|-<br />
| LAPOS<br />
| Perceptron based training with lookahead<br />
| Tsuruoka, Miyao, and Kazama (2011)<br />
| [http://www.logos.t.u-tokyo.ac.jp/~tsuruoka/lapos/ LAPOS]<br />
| No<br />
| 97.22%<br />
| Not available<br />
| MIT<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost COMPOST]<br />
| No<br />
| 97.23%<br />
| Not available<br />
| Non-free ([http://ufal.mff.cuni.cz/compost/register.php academic-only])<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost COMPOST]<br />
| Yes<br />
| 97.44%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Stanford Tagger 1.0<br />
| Maximum entropy cyclic dependency network<br />
| Toutanova et al. (2003)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.24%<br />
| 89.04%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.29%<br />
| 89.70%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| Yes<br />
| 97.32%<br />
| 90.79%<br />
| GPL v2+<br />
|-<br />
| LTAG-spinal<br />
| Bidirectional perceptron learning<br />
| Shen et al. (2007)<br />
| [http://www.cis.upenn.edu/~xtag/spinal/ LTAG-spinal]<br />
| No<br />
| 97.33%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SCCN<br />
| Semi-supervised condensed nearest neighbor<br />
| Søgaard (2011)<br />
| [http://cst.dk/anders/scnn/ SCCN]<br />
| Yes<br />
| 97.50%<br />
| Not available<br />
| Unknown<br />
|}<br />
<br />
(*) TnT: Accuracy is as reported by Giménez and Márquez (2004) for the given test collection. Brants (2000) reports 96.7% token accuracy and 85.5% unknown word accuracy on a 10-fold cross-validation of the Penn WSJ corpus.<br />
<br />
(**) GENiA: Results are for models trained and tested on the given corpora (to be comparable to other results). The distributed GENiA tagger is trained on a mixed training corpus and gets 96.94% on WSJ, and 98.26% on GENiA biomedical English.<br />
<br />
(***) Extra data: Whether system training exploited (usually large amounts of) extra unlabeled text, such as by semi-supervised learning, self-training, or using distributional similarity features, beyond the standard supervised training data.<br />
<br />
===FTB===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
! License<br />
|-<br />
| Morfette<br />
| Perceptron with external lexical information*<br />
| Chrupała et al. (2008), Seddah et al. (2010)<br />
| [http://sites.google.com/site/morfetteweb/ Morfette]<br />
| No<br />
| 97.68%<br />
| 90.52%<br />
| New BSD<br />
|-<br />
| SEM<br />
| CRF with external lexical information*<br />
| Constant et al. (2011)<br />
| [http://www.univ-orleans.fr/lifo/Members/Isabelle.Tellier/SEM.html SEM]<br />
| No<br />
| 97.7%<br />
| Not available<br />
| "GNU"(?)<br />
|-<br />
| MElt<br />
| MEMM with external lexical information*<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 97.80%<br />
| 91.77%<br />
| CeCILL-C<br />
|}<br />
<br />
(*) External lexical information from the Lefff lexicon (Sagot 2010, [https://gforge.inria.fr/frs/?group_id=482 Alexina project])<br />
<br />
== References ==<br />
<br />
* Brants, Thorsten. 2000. [http://acl.ldc.upenn.edu/A/A00/A00-1031.pdf TnT -- A Statistical Part-of-Speech Tagger]. "6th Applied Natural Language Processing Conference".<br />
<br />
* Chrupała, Grzegorz, Dinu, Georgiana and van Genabith, Josef. 2008. [http://www.lrec-conf.org/proceedings/lrec2008/pdf/594_paper.pdf Learning Morphology with Morfette]. "LREC 2008". <br />
<br />
* Collins, Michael. 2002. [http://people.csail.mit.edu/mcollins/papers/tagperc.pdf Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms]. ''EMNLP 2002''.<br />
<br />
* Constant, Matthieu, Tellier, Isabelle, Duchier, Denys, Dupont, Yoann, Sigogne, Anthony, and Billot, Sylvie. [http://www.lirmm.fr/~lopez/TALN2011/Longs-TALN+RECITAL/Tellier_taln11_submission_54.pdf Intégrer des connaissances linguistiques dans un CRF : application à l'apprentissage d'un segmenteur-étiqueteur du français]. "TALN'11"<br />
<br />
* Denis, Pascal and Sagot, Benoît. 2009. [http://alpage.inria.fr/~sagot/pub/paclic09tagging.pdf Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort]. "PACLIC 2009"<br />
<br />
* Giménez, J., and Márquez, L. 2004. [http://www.lsi.upc.es/~nlp/SVMTool/lrec2004-gm.pdf SVMTool: A general POS tagger generator based on Support Vector Machines]. ''Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC'04)''. Lisbon, Portugal. <br />
<br />
* Manning, Christopher D. 2011. Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In Alexander Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing, 12th International Conference, CICLing 2011, Proceedings, Part I. Lecture Notes in Computer Science 6608, pp. 171--189. Springer.<br />
<br />
* Seddah, Djamé, Chrupała, Grzegorz, Çetinoglu, Özlem and Candito, Marie. 2010. [http://aclweb.org/anthology-new/W/W10/W10-1410.pdf Lemmatization and Lexicalized Statistical Parsing of Morphologically Rich Languages: the Case of French] "SPMRL 2010 (NAACL 2010 workshop)"<br />
<br />
* Shen, L., Satta, G., and Joshi, A. 2007. [http://acl.ldc.upenn.edu/P/P07/P07-1096.pdf Guided learning for bidirectional sequence classification]. ''Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL 2007)'', pages 760-767.<br />
<br />
* Søgaard, Anders. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). Portland, Oregon.<br />
<br />
* Spoustová, Drahomíra "Johanka", Jan Hajič, Jan Raab and Miroslav Spousta. 2009. Semi-supervised Training for the Averaged Perceptron POS Tagger. Proceedings of the 12 EACL, pages 763-771.<br />
<br />
* Toutanova, K., Klein, D., Manning, C.D., Yoram Singer, Y. 2003. [http://nlp.stanford.edu/kristina/papers/tagging.pdf Feature-rich part-of-speech tagging with a cyclic dependency network]. ''Proceedings of HLT-NAACL 2003'', pages 252-259.<br />
<br />
* Tsuruoka, Yoshimasa, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/pci05.pdf Developing a Robust Part-of-Speech Tagger for Biomedical Text, Advances in Informatics]" - ''10th Panhellenic Conference on Informatics'', '''LNCS 3746''', pp. 382-392, 2005 <br />
<br />
* Tsuruoka, Yoshimasa, Yusuke Miyao, and Jun’ichi Kazama. 2011. "[http://aclweb.org/anthology-new/W/W11/W11-0328.pdf Learning with Lookahead: Can History-Based Models Rival Globally Optimized Models?]" ''Proceedings of the Fifteenth Conference on Computational Natural Language Learning'', pp 238–246, 2011.<br />
<br />
* Tsuruoka, Yoshimasa and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/emnlp05bidir.pdf Bidirectional Inference with the Easiest-First Strategy for Tagging Sequence Data]", ''Proceedings of HLT/EMNLP 2005'', pp. 467-474.<br />
<br />
== See also ==<br />
* [[POS Induction (State of the art)]]<br />
* [[Part-of-speech tagging]]<br />
* [[State of the art]]<br />
<br />
<br />
[[Category:State of the art]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=POS_Tagging_(State_of_the_art)&diff=9894POS Tagging (State of the art)2012-12-29T17:10:22Z<p>Kiwibird: sort</p>
<hr />
<div>==Test collections==<br />
* '''Performance measure:''' per token accuracy. (The convention is for this to be measured on all tokens, including punctuation tokens and other unambiguous tokens.)<br />
* '''English'''<br />
** '''Penn Treebank''' ''Wall Street Journal'' (WSJ) release 3 (LDC99T42). The splits of data for this task were not standardized early on (unlike for parsing) and early work uses various data splits defined by counts of tokens or by sections. Most work from 2002 on adopts the following data splits, introduced by Collins (2002):<br />
*** '''Training data:''' sections 0-18<br />
*** '''Development test data:''' sections 19-21<br />
*** '''Testing data:''' sections 22-24<br />
<br />
* '''French'''<br />
** '''French TreeBank''' (FTB, Abeillé et al; 2003) ''Le Monde'', December 2007 version, 28-tag tagset (CC tagset, Crabbé and Candito, 2008). Classical data split (10-10-80):<br />
*** '''Training data:''' sentences 2471 to 12351<br />
*** '''Development test data:''' sentences 1236 to 2470<br />
*** '''Testing data:''' sentences 1 to 1235<br />
<br />
<br />
== Tables of results ==<br />
<br />
===WSJ===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
! License<br />
|-<br />
| TnT*<br />
| Hidden markov model<br />
| Brants (2000)<br />
| [http://www.coli.uni-saarland.de/~thorsten/tnt/ TnT]<br />
| No<br />
| 96.46%<br />
| 85.86%<br />
| Unknown<br />
|-<br />
| MElt<br />
| MEMM with external lexical information<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 96.96%<br />
| 91.29%<br />
| CeCILL-C<br />
|-<br />
| GENiA Tagger**<br />
| Maximum entropy cyclic dependency network<br />
| Tsuruoka, et al (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/ GENiA]<br />
| No<br />
| 97.05%<br />
| Not available<br />
| Gratis for non-commercial usage<br />
|-<br />
| Averaged Perceptron<br />
| Averaged Perception discriminative sequence model<br />
| Collins (2002)<br />
| Not available<br />
| No<br />
| 97.11%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Maxent easiest-first<br />
| Maximum entropy bidirectional easiest-first inference<br />
| Tsuruoka and Tsujii (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/postagger/ Easiest-first]<br />
| No<br />
| 97.15%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SVMTool<br />
| SVM-based tagger and tagger generator<br />
| Giménez and Márquez (2004)<br />
| [http://www.lsi.upc.es/~nlp/SVMTool/ SVMTool]<br />
| No<br />
| 97.16%<br />
| 89.01%<br />
| LGPL 2.1<br />
|-<br />
| LAPOS<br />
| Perceptron based training with lookahead<br />
| Tsuruoka, Miyao, and Kazama (2011)<br />
| [http://www.logos.t.u-tokyo.ac.jp/~tsuruoka/lapos/ LAPOS]<br />
| No<br />
| 97.22%<br />
| Not available<br />
| MIT<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost COMPOST]<br />
| No<br />
| 97.23%<br />
| Not available<br />
| Non-free ([http://ufal.mff.cuni.cz/compost/register.php academic-only])<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost COMPOST]<br />
| Yes<br />
| 97.44%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Stanford Tagger 1.0<br />
| Maximum entropy cyclic dependency network<br />
| Toutanova et al. (2003)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.24%<br />
| 89.04%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.29%<br />
| 89.70%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| Yes<br />
| 97.32%<br />
| 90.79%<br />
| GPL v2+<br />
|-<br />
| LTAG-spinal<br />
| Bidirectional perceptron learning<br />
| Shen et al. (2007)<br />
| [http://www.cis.upenn.edu/~xtag/spinal/ LTAG-spinal]<br />
| No<br />
| 97.33%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SCCN<br />
| Semi-supervised condensed nearest neighbor<br />
| Søgaard (2011)<br />
| [http://cst.dk/anders/scnn/ SCCN]<br />
| Yes<br />
| 97.50%<br />
| Not available<br />
| Unknown<br />
|}<br />
<br />
(*) TnT: Accuracy is as reported by Giménez and Márquez (2004) for the given test collection. Brants (2000) reports 96.7% token accuracy and 85.5% unknown word accuracy on a 10-fold cross-validation of the Penn WSJ corpus.<br />
<br />
(**) GENiA: Results are for models trained and tested on the given corpora (to be comparable to other results). The distributed GENiA tagger is trained on a mixed training corpus and gets 96.94% on WSJ, and 98.26% on GENiA biomedical English.<br />
<br />
(***) Extra data: Whether system training exploited (usually large amounts of) extra unlabeled text, such as by semi-supervised learning, self-training, or using distributional similarity features, beyond the standard supervised training data.<br />
<br />
===FTB===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
|-<br />
| Morfette<br />
| Perceptron with external lexical information*<br />
| Chrupała et al. (2008), Seddah et al. (2010)<br />
| [http://sites.google.com/site/morfetteweb/ Morfette]<br />
| No<br />
| 97.68%<br />
| 90.52%<br />
|-<br />
| SEM<br />
| CRF with external lexical information*<br />
| Constant et al. (2011)<br />
| [http://www.univ-orleans.fr/lifo/Members/Isabelle.Tellier/SEM.html SEM]<br />
| No<br />
| 97.7%<br />
| Not available<br />
|-<br />
| MElt<br />
| MEMM with external lexical information*<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 97.80%<br />
| 91.77%<br />
|}<br />
<br />
(*) External lexical information from the Lefff lexicon (Sagot 2010, [https://gforge.inria.fr/frs/?group_id=482 Alexina project])<br />
<br />
<br />
== References ==<br />
<br />
* Brants, Thorsten. 2000. [http://acl.ldc.upenn.edu/A/A00/A00-1031.pdf TnT -- A Statistical Part-of-Speech Tagger]. "6th Applied Natural Language Processing Conference".<br />
<br />
* Chrupała, Grzegorz, Dinu, Georgiana and van Genabith, Josef. 2008. [http://www.lrec-conf.org/proceedings/lrec2008/pdf/594_paper.pdf Learning Morphology with Morfette]. "LREC 2008". <br />
<br />
* Collins, Michael. 2002. [http://people.csail.mit.edu/mcollins/papers/tagperc.pdf Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms]. ''EMNLP 2002''.<br />
<br />
* Constant, Matthieu, Tellier, Isabelle, Duchier, Denys, Dupont, Yoann, Sigogne, Anthony, and Billot, Sylvie. [http://www.lirmm.fr/~lopez/TALN2011/Longs-TALN+RECITAL/Tellier_taln11_submission_54.pdf Intégrer des connaissances linguistiques dans un CRF : application à l'apprentissage d'un segmenteur-étiqueteur du français]. "TALN'11"<br />
<br />
* Denis, Pascal and Sagot, Benoît. 2009. [http://alpage.inria.fr/~sagot/pub/paclic09tagging.pdf Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort]. "PACLIC 2009"<br />
<br />
* Giménez, J., and Márquez, L. 2004. [http://www.lsi.upc.es/~nlp/SVMTool/lrec2004-gm.pdf SVMTool: A general POS tagger generator based on Support Vector Machines]. ''Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC'04)''. Lisbon, Portugal. <br />
<br />
* Manning, Christopher D. 2011. Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In Alexander Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing, 12th International Conference, CICLing 2011, Proceedings, Part I. Lecture Notes in Computer Science 6608, pp. 171--189. Springer.<br />
<br />
* Seddah, Djamé, Chrupała, Grzegorz, Çetinoglu, Özlem and Candito, Marie. 2010. [http://aclweb.org/anthology-new/W/W10/W10-1410.pdf Lemmatization and Lexicalized Statistical Parsing of Morphologically Rich Languages: the Case of French] "SPMRL 2010 (NAACL 2010 workshop)"<br />
<br />
* Shen, L., Satta, G., and Joshi, A. 2007. [http://acl.ldc.upenn.edu/P/P07/P07-1096.pdf Guided learning for bidirectional sequence classification]. ''Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL 2007)'', pages 760-767.<br />
<br />
* Søgaard, Anders. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). Portland, Oregon.<br />
<br />
* Spoustová, Drahomíra "Johanka", Jan Hajič, Jan Raab and Miroslav Spousta. 2009. Semi-supervised Training for the Averaged Perceptron POS Tagger. Proceedings of the 12 EACL, pages 763-771.<br />
<br />
* Toutanova, K., Klein, D., Manning, C.D., Yoram Singer, Y. 2003. [http://nlp.stanford.edu/kristina/papers/tagging.pdf Feature-rich part-of-speech tagging with a cyclic dependency network]. ''Proceedings of HLT-NAACL 2003'', pages 252-259.<br />
<br />
* Tsuruoka, Yoshimasa, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/pci05.pdf Developing a Robust Part-of-Speech Tagger for Biomedical Text, Advances in Informatics]" - ''10th Panhellenic Conference on Informatics'', '''LNCS 3746''', pp. 382-392, 2005 <br />
<br />
* Tsuruoka, Yoshimasa, Yusuke Miyao, and Jun’ichi Kazama. 2011. "[http://aclweb.org/anthology-new/W/W11/W11-0328.pdf Learning with Lookahead: Can History-Based Models Rival Globally Optimized Models?]" ''Proceedings of the Fifteenth Conference on Computational Natural Language Learning'', pp 238–246, 2011.<br />
<br />
* Tsuruoka, Yoshimasa and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/emnlp05bidir.pdf Bidirectional Inference with the Easiest-First Strategy for Tagging Sequence Data]", ''Proceedings of HLT/EMNLP 2005'', pp. 467-474.<br />
<br />
== See also ==<br />
* [[POS Induction (State of the art)]]<br />
* [[Part-of-speech tagging]]<br />
* [[State of the art]]<br />
<br />
<br />
[[Category:State of the art]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=POS_Tagging_(State_of_the_art)&diff=9893POS Tagging (State of the art)2012-12-29T17:08:49Z<p>Kiwibird: /* WSJ */</p>
<hr />
<div>==Test collections==<br />
* '''Performance measure:''' per token accuracy. (The convention is for this to be measured on all tokens, including punctuation tokens and other unambiguous tokens.)<br />
* '''English'''<br />
** '''Penn Treebank''' ''Wall Street Journal'' (WSJ) release 3 (LDC99T42). The splits of data for this task were not standardized early on (unlike for parsing) and early work uses various data splits defined by counts of tokens or by sections. Most work from 2002 on adopts the following data splits, introduced by Collins (2002):<br />
*** '''Training data:''' sections 0-18<br />
*** '''Development test data:''' sections 19-21<br />
*** '''Testing data:''' sections 22-24<br />
<br />
* '''French'''<br />
** '''French TreeBank''' (FTB, Abeillé et al; 2003) ''Le Monde'', December 2007 version, 28-tag tagset (CC tagset, Crabbé and Candito, 2008). Classical data split (10-10-80):<br />
*** '''Training data:''' sentences 2471 to 12351<br />
*** '''Development test data:''' sentences 1236 to 2470<br />
*** '''Testing data:''' sentences 1 to 1235<br />
<br />
<br />
== Tables of results ==<br />
<br />
===WSJ===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
! License<br />
|-<br />
| TnT*<br />
| Hidden markov model<br />
| Brants (2000)<br />
| [http://www.coli.uni-saarland.de/~thorsten/tnt/ TnT]<br />
| No<br />
| 96.46%<br />
| 85.86%<br />
| Unknown<br />
|-<br />
| MElt<br />
| MEMM with external lexical information<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 96.96%<br />
| 91.29%<br />
| CeCILL-C<br />
|-<br />
| GENiA Tagger**<br />
| Maximum entropy cyclic dependency network<br />
| Tsuruoka, et al (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/ GENiA]<br />
| No<br />
| 97.05%<br />
| Not available<br />
| Gratis for non-commercial usage<br />
|-<br />
| Averaged Perceptron<br />
| Averaged Perception discriminative sequence model<br />
| Collins (2002)<br />
| Not available<br />
| No<br />
| 97.11%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Maxent easiest-first<br />
| Maximum entropy bidirectional easiest-first inference<br />
| Tsuruoka and Tsujii (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/postagger/ Easiest-first]<br />
| No<br />
| 97.15%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SVMTool<br />
| SVM-based tagger and tagger generator<br />
| Giménez and Márquez (2004)<br />
| [http://www.lsi.upc.es/~nlp/SVMTool/ SVMTool]<br />
| No<br />
| 97.16%<br />
| 89.01%<br />
| LGPL 2.1<br />
|-<br />
| LAPOS<br />
| Perceptron based training with lookahead<br />
| Tsuruoka, Miyao, and Kazama (2011)<br />
| [http://www.logos.t.u-tokyo.ac.jp/~tsuruoka/lapos/ LAPOS]<br />
| No<br />
| 97.22%<br />
| Not available<br />
| MIT<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost]<br />
| No<br />
| 97.23%<br />
| Not available<br />
| Non-free ([http://ufal.mff.cuni.cz/compost/register.php academic-only])<br />
|-<br />
| Stanford Tagger 1.0<br />
| Maximum entropy cyclic dependency network<br />
| Toutanova et al. (2003)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.24%<br />
| 89.04%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.29%<br />
| 89.70%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| Yes<br />
| 97.32%<br />
| 90.79%<br />
| GPL v2+<br />
|-<br />
| LTAG-spinal<br />
| Bidirectional perceptron learning<br />
| Shen et al. (2007)<br />
| [http://www.cis.upenn.edu/~xtag/spinal/ LTAG-spinal]<br />
| No<br />
| 97.33%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost]<br />
| Yes<br />
| 97.44%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SCCN<br />
| Semi-supervised condensed nearest neighbor<br />
| Søgaard (2011)<br />
| [http://cst.dk/anders/scnn/ SCCN]<br />
| Yes<br />
| 97.50%<br />
| Not available<br />
| Unknown<br />
|}<br />
<br />
(*) TnT: Accuracy is as reported by Giménez and Márquez (2004) for the given test collection. Brants (2000) reports 96.7% token accuracy and 85.5% unknown word accuracy on a 10-fold cross-validation of the Penn WSJ corpus.<br />
<br />
(**) GENiA: Results are for models trained and tested on the given corpora (to be comparable to other results). The distributed GENiA tagger is trained on a mixed training corpus and gets 96.94% on WSJ, and 98.26% on GENiA biomedical English.<br />
<br />
(***) Extra data: Whether system training exploited (usually large amounts of) extra unlabeled text, such as by semi-supervised learning, self-training, or using distributional similarity features, beyond the standard supervised training data.<br />
<br />
===FTB===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
|-<br />
| Morfette<br />
| Perceptron with external lexical information*<br />
| Chrupała et al. (2008), Seddah et al. (2010)<br />
| [http://sites.google.com/site/morfetteweb/ Morfette]<br />
| No<br />
| 97.68%<br />
| 90.52%<br />
|-<br />
| SEM<br />
| CRF with external lexical information*<br />
| Constant et al. (2011)<br />
| [http://www.univ-orleans.fr/lifo/Members/Isabelle.Tellier/SEM.html SEM]<br />
| No<br />
| 97.7%<br />
| Not available<br />
|-<br />
| MElt<br />
| MEMM with external lexical information*<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 97.80%<br />
| 91.77%<br />
|}<br />
<br />
(*) External lexical information from the Lefff lexicon (Sagot 2010, [https://gforge.inria.fr/frs/?group_id=482 Alexina project])<br />
<br />
<br />
== References ==<br />
<br />
* Brants, Thorsten. 2000. [http://acl.ldc.upenn.edu/A/A00/A00-1031.pdf TnT -- A Statistical Part-of-Speech Tagger]. "6th Applied Natural Language Processing Conference".<br />
<br />
* Chrupała, Grzegorz, Dinu, Georgiana and van Genabith, Josef. 2008. [http://www.lrec-conf.org/proceedings/lrec2008/pdf/594_paper.pdf Learning Morphology with Morfette]. "LREC 2008". <br />
<br />
* Collins, Michael. 2002. [http://people.csail.mit.edu/mcollins/papers/tagperc.pdf Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms]. ''EMNLP 2002''.<br />
<br />
* Constant, Matthieu, Tellier, Isabelle, Duchier, Denys, Dupont, Yoann, Sigogne, Anthony, and Billot, Sylvie. [http://www.lirmm.fr/~lopez/TALN2011/Longs-TALN+RECITAL/Tellier_taln11_submission_54.pdf Intégrer des connaissances linguistiques dans un CRF : application à l'apprentissage d'un segmenteur-étiqueteur du français]. "TALN'11"<br />
<br />
* Denis, Pascal and Sagot, Benoît. 2009. [http://alpage.inria.fr/~sagot/pub/paclic09tagging.pdf Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort]. "PACLIC 2009"<br />
<br />
* Giménez, J., and Márquez, L. 2004. [http://www.lsi.upc.es/~nlp/SVMTool/lrec2004-gm.pdf SVMTool: A general POS tagger generator based on Support Vector Machines]. ''Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC'04)''. Lisbon, Portugal. <br />
<br />
* Manning, Christopher D. 2011. Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In Alexander Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing, 12th International Conference, CICLing 2011, Proceedings, Part I. Lecture Notes in Computer Science 6608, pp. 171--189. Springer.<br />
<br />
* Seddah, Djamé, Chrupała, Grzegorz, Çetinoglu, Özlem and Candito, Marie. 2010. [http://aclweb.org/anthology-new/W/W10/W10-1410.pdf Lemmatization and Lexicalized Statistical Parsing of Morphologically Rich Languages: the Case of French] "SPMRL 2010 (NAACL 2010 workshop)"<br />
<br />
* Shen, L., Satta, G., and Joshi, A. 2007. [http://acl.ldc.upenn.edu/P/P07/P07-1096.pdf Guided learning for bidirectional sequence classification]. ''Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL 2007)'', pages 760-767.<br />
<br />
* Søgaard, Anders. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). Portland, Oregon.<br />
<br />
* Spoustová, Drahomíra "Johanka", Jan Hajič, Jan Raab and Miroslav Spousta. 2009. Semi-supervised Training for the Averaged Perceptron POS Tagger. Proceedings of the 12 EACL, pages 763-771.<br />
<br />
* Toutanova, K., Klein, D., Manning, C.D., Yoram Singer, Y. 2003. [http://nlp.stanford.edu/kristina/papers/tagging.pdf Feature-rich part-of-speech tagging with a cyclic dependency network]. ''Proceedings of HLT-NAACL 2003'', pages 252-259.<br />
<br />
* Tsuruoka, Yoshimasa, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/pci05.pdf Developing a Robust Part-of-Speech Tagger for Biomedical Text, Advances in Informatics]" - ''10th Panhellenic Conference on Informatics'', '''LNCS 3746''', pp. 382-392, 2005 <br />
<br />
* Tsuruoka, Yoshimasa, Yusuke Miyao, and Jun’ichi Kazama. 2011. "[http://aclweb.org/anthology-new/W/W11/W11-0328.pdf Learning with Lookahead: Can History-Based Models Rival Globally Optimized Models?]" ''Proceedings of the Fifteenth Conference on Computational Natural Language Learning'', pp 238–246, 2011.<br />
<br />
* Tsuruoka, Yoshimasa and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/emnlp05bidir.pdf Bidirectional Inference with the Easiest-First Strategy for Tagging Sequence Data]", ''Proceedings of HLT/EMNLP 2005'', pp. 467-474.<br />
<br />
== See also ==<br />
* [[POS Induction (State of the art)]]<br />
* [[Part-of-speech tagging]]<br />
* [[State of the art]]<br />
<br />
<br />
[[Category:State of the art]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=POS_Tagging_(State_of_the_art)&diff=9892POS Tagging (State of the art)2012-12-29T17:06:16Z<p>Kiwibird: /* WSJ */</p>
<hr />
<div>==Test collections==<br />
* '''Performance measure:''' per token accuracy. (The convention is for this to be measured on all tokens, including punctuation tokens and other unambiguous tokens.)<br />
* '''English'''<br />
** '''Penn Treebank''' ''Wall Street Journal'' (WSJ) release 3 (LDC99T42). The splits of data for this task were not standardized early on (unlike for parsing) and early work uses various data splits defined by counts of tokens or by sections. Most work from 2002 on adopts the following data splits, introduced by Collins (2002):<br />
*** '''Training data:''' sections 0-18<br />
*** '''Development test data:''' sections 19-21<br />
*** '''Testing data:''' sections 22-24<br />
<br />
* '''French'''<br />
** '''French TreeBank''' (FTB, Abeillé et al; 2003) ''Le Monde'', December 2007 version, 28-tag tagset (CC tagset, Crabbé and Candito, 2008). Classical data split (10-10-80):<br />
*** '''Training data:''' sentences 2471 to 12351<br />
*** '''Development test data:''' sentences 1236 to 2470<br />
*** '''Testing data:''' sentences 1 to 1235<br />
<br />
<br />
== Tables of results ==<br />
<br />
===WSJ===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
! License<br />
|-<br />
| TnT*<br />
| Hidden markov model<br />
| Brants (2000)<br />
| [http://www.coli.uni-saarland.de/~thorsten/tnt/ TnT]<br />
| No<br />
| 96.46%<br />
| 85.86%<br />
| Unknown<br />
|-<br />
| MElt<br />
| MEMM with external lexical information<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 96.96%<br />
| 91.29%<br />
| Unknown<br />
|-<br />
| GENiA Tagger**<br />
| Maximum entropy cyclic dependency network<br />
| Tsuruoka, et al (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/ GENiA]<br />
| No<br />
| 97.05%<br />
| Not available<br />
| Gratis for non-commercial usage<br />
|-<br />
| Averaged Perceptron<br />
| Averaged Perception discriminative sequence model<br />
| Collins (2002)<br />
| Not available<br />
| No<br />
| 97.11%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Maxent easiest-first<br />
| Maximum entropy bidirectional easiest-first inference<br />
| Tsuruoka and Tsujii (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/postagger/ Easiest-first]<br />
| No<br />
| 97.15%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SVMTool<br />
| SVM-based tagger and tagger generator<br />
| Giménez and Márquez (2004)<br />
| [http://www.lsi.upc.es/~nlp/SVMTool/ SVMTool]<br />
| No<br />
| 97.16%<br />
| 89.01%<br />
| LGPL 2.1<br />
|-<br />
| LAPOS<br />
| Perceptron based training with lookahead<br />
| Tsuruoka, Miyao, and Kazama (2011)<br />
| [http://www.logos.t.u-tokyo.ac.jp/~tsuruoka/lapos/ LAPOS]<br />
| No<br />
| 97.22%<br />
| Not available<br />
| MIT<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost]<br />
| No<br />
| 97.23%<br />
| Not available<br />
| Non-free ([http://ufal.mff.cuni.cz/compost/register.php academic-only])<br />
|-<br />
| Stanford Tagger 1.0<br />
| Maximum entropy cyclic dependency network<br />
| Toutanova et al. (2003)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.24%<br />
| 89.04%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.29%<br />
| 89.70%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| Yes<br />
| 97.32%<br />
| 90.79%<br />
| GPL v2+<br />
|-<br />
| LTAG-spinal<br />
| Bidirectional perceptron learning<br />
| Shen et al. (2007)<br />
| [http://www.cis.upenn.edu/~xtag/spinal/ LTAG-spinal]<br />
| No<br />
| 97.33%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost]<br />
| Yes<br />
| 97.44%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SCCN<br />
| Semi-supervised condensed nearest neighbor<br />
| Søgaard (2011)<br />
| [http://cst.dk/anders/scnn/ SCCN]<br />
| Yes<br />
| 97.50%<br />
| Not available<br />
| Unknown<br />
|}<br />
<br />
(*) TnT: Accuracy is as reported by Giménez and Márquez (2004) for the given test collection. Brants (2000) reports 96.7% token accuracy and 85.5% unknown word accuracy on a 10-fold cross-validation of the Penn WSJ corpus.<br />
<br />
(**) GENiA: Results are for models trained and tested on the given corpora (to be comparable to other results). The distributed GENiA tagger is trained on a mixed training corpus and gets 96.94% on WSJ, and 98.26% on GENiA biomedical English.<br />
<br />
(***) Extra data: Whether system training exploited (usually large amounts of) extra unlabeled text, such as by semi-supervised learning, self-training, or using distributional similarity features, beyond the standard supervised training data.<br />
<br />
===FTB===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
|-<br />
| Morfette<br />
| Perceptron with external lexical information*<br />
| Chrupała et al. (2008), Seddah et al. (2010)<br />
| [http://sites.google.com/site/morfetteweb/ Morfette]<br />
| No<br />
| 97.68%<br />
| 90.52%<br />
|-<br />
| SEM<br />
| CRF with external lexical information*<br />
| Constant et al. (2011)<br />
| [http://www.univ-orleans.fr/lifo/Members/Isabelle.Tellier/SEM.html SEM]<br />
| No<br />
| 97.7%<br />
| Not available<br />
|-<br />
| MElt<br />
| MEMM with external lexical information*<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 97.80%<br />
| 91.77%<br />
|}<br />
<br />
(*) External lexical information from the Lefff lexicon (Sagot 2010, [https://gforge.inria.fr/frs/?group_id=482 Alexina project])<br />
<br />
<br />
== References ==<br />
<br />
* Brants, Thorsten. 2000. [http://acl.ldc.upenn.edu/A/A00/A00-1031.pdf TnT -- A Statistical Part-of-Speech Tagger]. "6th Applied Natural Language Processing Conference".<br />
<br />
* Chrupała, Grzegorz, Dinu, Georgiana and van Genabith, Josef. 2008. [http://www.lrec-conf.org/proceedings/lrec2008/pdf/594_paper.pdf Learning Morphology with Morfette]. "LREC 2008". <br />
<br />
* Collins, Michael. 2002. [http://people.csail.mit.edu/mcollins/papers/tagperc.pdf Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms]. ''EMNLP 2002''.<br />
<br />
* Constant, Matthieu, Tellier, Isabelle, Duchier, Denys, Dupont, Yoann, Sigogne, Anthony, and Billot, Sylvie. [http://www.lirmm.fr/~lopez/TALN2011/Longs-TALN+RECITAL/Tellier_taln11_submission_54.pdf Intégrer des connaissances linguistiques dans un CRF : application à l'apprentissage d'un segmenteur-étiqueteur du français]. "TALN'11"<br />
<br />
* Denis, Pascal and Sagot, Benoît. 2009. [http://alpage.inria.fr/~sagot/pub/paclic09tagging.pdf Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort]. "PACLIC 2009"<br />
<br />
* Giménez, J., and Márquez, L. 2004. [http://www.lsi.upc.es/~nlp/SVMTool/lrec2004-gm.pdf SVMTool: A general POS tagger generator based on Support Vector Machines]. ''Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC'04)''. Lisbon, Portugal. <br />
<br />
* Manning, Christopher D. 2011. Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In Alexander Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing, 12th International Conference, CICLing 2011, Proceedings, Part I. Lecture Notes in Computer Science 6608, pp. 171--189. Springer.<br />
<br />
* Seddah, Djamé, Chrupała, Grzegorz, Çetinoglu, Özlem and Candito, Marie. 2010. [http://aclweb.org/anthology-new/W/W10/W10-1410.pdf Lemmatization and Lexicalized Statistical Parsing of Morphologically Rich Languages: the Case of French] "SPMRL 2010 (NAACL 2010 workshop)"<br />
<br />
* Shen, L., Satta, G., and Joshi, A. 2007. [http://acl.ldc.upenn.edu/P/P07/P07-1096.pdf Guided learning for bidirectional sequence classification]. ''Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL 2007)'', pages 760-767.<br />
<br />
* Søgaard, Anders. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). Portland, Oregon.<br />
<br />
* Spoustová, Drahomíra "Johanka", Jan Hajič, Jan Raab and Miroslav Spousta. 2009. Semi-supervised Training for the Averaged Perceptron POS Tagger. Proceedings of the 12 EACL, pages 763-771.<br />
<br />
* Toutanova, K., Klein, D., Manning, C.D., Yoram Singer, Y. 2003. [http://nlp.stanford.edu/kristina/papers/tagging.pdf Feature-rich part-of-speech tagging with a cyclic dependency network]. ''Proceedings of HLT-NAACL 2003'', pages 252-259.<br />
<br />
* Tsuruoka, Yoshimasa, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/pci05.pdf Developing a Robust Part-of-Speech Tagger for Biomedical Text, Advances in Informatics]" - ''10th Panhellenic Conference on Informatics'', '''LNCS 3746''', pp. 382-392, 2005 <br />
<br />
* Tsuruoka, Yoshimasa, Yusuke Miyao, and Jun’ichi Kazama. 2011. "[http://aclweb.org/anthology-new/W/W11/W11-0328.pdf Learning with Lookahead: Can History-Based Models Rival Globally Optimized Models?]" ''Proceedings of the Fifteenth Conference on Computational Natural Language Learning'', pp 238–246, 2011.<br />
<br />
* Tsuruoka, Yoshimasa and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/emnlp05bidir.pdf Bidirectional Inference with the Easiest-First Strategy for Tagging Sequence Data]", ''Proceedings of HLT/EMNLP 2005'', pp. 467-474.<br />
<br />
== See also ==<br />
* [[POS Induction (State of the art)]]<br />
* [[Part-of-speech tagging]]<br />
* [[State of the art]]<br />
<br />
<br />
[[Category:State of the art]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=POS_Tagging_(State_of_the_art)&diff=9891POS Tagging (State of the art)2012-12-29T17:01:39Z<p>Kiwibird: The SVMTool library is licensed under LGPL 2.1</p>
<hr />
<div>==Test collections==<br />
* '''Performance measure:''' per token accuracy. (The convention is for this to be measured on all tokens, including punctuation tokens and other unambiguous tokens.)<br />
* '''English'''<br />
** '''Penn Treebank''' ''Wall Street Journal'' (WSJ) release 3 (LDC99T42). The splits of data for this task were not standardized early on (unlike for parsing) and early work uses various data splits defined by counts of tokens or by sections. Most work from 2002 on adopts the following data splits, introduced by Collins (2002):<br />
*** '''Training data:''' sections 0-18<br />
*** '''Development test data:''' sections 19-21<br />
*** '''Testing data:''' sections 22-24<br />
<br />
* '''French'''<br />
** '''French TreeBank''' (FTB, Abeillé et al; 2003) ''Le Monde'', December 2007 version, 28-tag tagset (CC tagset, Crabbé and Candito, 2008). Classical data split (10-10-80):<br />
*** '''Training data:''' sentences 2471 to 12351<br />
*** '''Development test data:''' sentences 1236 to 2470<br />
*** '''Testing data:''' sentences 1 to 1235<br />
<br />
<br />
== Tables of results ==<br />
<br />
===WSJ===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
! License<br />
|-<br />
| TnT*<br />
| Hidden markov model<br />
| Brants (2000)<br />
| [http://www.coli.uni-saarland.de/~thorsten/tnt/ TnT]<br />
| No<br />
| 96.46%<br />
| 85.86%<br />
| Unknown<br />
|-<br />
| MElt<br />
| MEMM with external lexical information<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 96.96%<br />
| 91.29%<br />
| Unknown<br />
|-<br />
| GENiA Tagger**<br />
| Maximum entropy cyclic dependency network<br />
| Tsuruoka, et al (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/ GENiA]<br />
| No<br />
| 97.05%<br />
| Not available<br />
| Gratis for non-commercial usage<br />
|-<br />
| Averaged Perceptron<br />
| Averaged Perception discriminative sequence model<br />
| Collins (2002)<br />
| Not available<br />
| No<br />
| 97.11%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Maxent easiest-first<br />
| Maximum entropy bidirectional easiest-first inference<br />
| Tsuruoka and Tsujii (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/postagger/ Easiest-first]<br />
| No<br />
| 97.15%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SVMTool<br />
| SVM-based tagger and tagger generator<br />
| Giménez and Márquez (2004)<br />
| [http://www.lsi.upc.es/~nlp/SVMTool/ SVMTool]<br />
| No<br />
| 97.16%<br />
| 89.01%<br />
| LGPL 2.1<br />
|-<br />
| LAPOS<br />
| Perceptron based training with lookahead<br />
| Tsuruoka, Miyao, and Kazama (2011)<br />
| [http://www.logos.t.u-tokyo.ac.jp/~tsuruoka/lapos/ LAPOS]<br />
| No<br />
| 97.22%<br />
| Not available<br />
| MIT<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost]<br />
| No<br />
| 97.23%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Stanford Tagger 1.0<br />
| Maximum entropy cyclic dependency network<br />
| Toutanova et al. (2003)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.24%<br />
| 89.04%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.29%<br />
| 89.70%<br />
| GPL v2+<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| Yes<br />
| 97.32%<br />
| 90.79%<br />
| GPL v2+<br />
|-<br />
| LTAG-spinal<br />
| Bidirectional perceptron learning<br />
| Shen et al. (2007)<br />
| [http://www.cis.upenn.edu/~xtag/spinal/ LTAG-spinal]<br />
| No<br />
| 97.33%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost]<br />
| Yes<br />
| 97.44%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SCCN<br />
| Semi-supervised condensed nearest neighbor<br />
| Søgaard (2011)<br />
| [http://cst.dk/anders/scnn/ SCCN]<br />
| Yes<br />
| 97.50%<br />
| Not available<br />
| Unknown<br />
|}<br />
<br />
(*) TnT: Accuracy is as reported by Giménez and Márquez (2004) for the given test collection. Brants (2000) reports 96.7% token accuracy and 85.5% unknown word accuracy on a 10-fold cross-validation of the Penn WSJ corpus.<br />
<br />
(**) GENiA: Results are for models trained and tested on the given corpora (to be comparable to other results). The distributed GENiA tagger is trained on a mixed training corpus and gets 96.94% on WSJ, and 98.26% on GENiA biomedical English.<br />
<br />
(***) Extra data: Whether system training exploited (usually large amounts of) extra unlabeled text, such as by semi-supervised learning, self-training, or using distributional similarity features, beyond the standard supervised training data.<br />
<br />
===FTB===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
|-<br />
| Morfette<br />
| Perceptron with external lexical information*<br />
| Chrupała et al. (2008), Seddah et al. (2010)<br />
| [http://sites.google.com/site/morfetteweb/ Morfette]<br />
| No<br />
| 97.68%<br />
| 90.52%<br />
|-<br />
| SEM<br />
| CRF with external lexical information*<br />
| Constant et al. (2011)<br />
| [http://www.univ-orleans.fr/lifo/Members/Isabelle.Tellier/SEM.html SEM]<br />
| No<br />
| 97.7%<br />
| Not available<br />
|-<br />
| MElt<br />
| MEMM with external lexical information*<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 97.80%<br />
| 91.77%<br />
|}<br />
<br />
(*) External lexical information from the Lefff lexicon (Sagot 2010, [https://gforge.inria.fr/frs/?group_id=482 Alexina project])<br />
<br />
<br />
== References ==<br />
<br />
* Brants, Thorsten. 2000. [http://acl.ldc.upenn.edu/A/A00/A00-1031.pdf TnT -- A Statistical Part-of-Speech Tagger]. "6th Applied Natural Language Processing Conference".<br />
<br />
* Chrupała, Grzegorz, Dinu, Georgiana and van Genabith, Josef. 2008. [http://www.lrec-conf.org/proceedings/lrec2008/pdf/594_paper.pdf Learning Morphology with Morfette]. "LREC 2008". <br />
<br />
* Collins, Michael. 2002. [http://people.csail.mit.edu/mcollins/papers/tagperc.pdf Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms]. ''EMNLP 2002''.<br />
<br />
* Constant, Matthieu, Tellier, Isabelle, Duchier, Denys, Dupont, Yoann, Sigogne, Anthony, and Billot, Sylvie. [http://www.lirmm.fr/~lopez/TALN2011/Longs-TALN+RECITAL/Tellier_taln11_submission_54.pdf Intégrer des connaissances linguistiques dans un CRF : application à l'apprentissage d'un segmenteur-étiqueteur du français]. "TALN'11"<br />
<br />
* Denis, Pascal and Sagot, Benoît. 2009. [http://alpage.inria.fr/~sagot/pub/paclic09tagging.pdf Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort]. "PACLIC 2009"<br />
<br />
* Giménez, J., and Márquez, L. 2004. [http://www.lsi.upc.es/~nlp/SVMTool/lrec2004-gm.pdf SVMTool: A general POS tagger generator based on Support Vector Machines]. ''Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC'04)''. Lisbon, Portugal. <br />
<br />
* Manning, Christopher D. 2011. Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In Alexander Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing, 12th International Conference, CICLing 2011, Proceedings, Part I. Lecture Notes in Computer Science 6608, pp. 171--189. Springer.<br />
<br />
* Seddah, Djamé, Chrupała, Grzegorz, Çetinoglu, Özlem and Candito, Marie. 2010. [http://aclweb.org/anthology-new/W/W10/W10-1410.pdf Lemmatization and Lexicalized Statistical Parsing of Morphologically Rich Languages: the Case of French] "SPMRL 2010 (NAACL 2010 workshop)"<br />
<br />
* Shen, L., Satta, G., and Joshi, A. 2007. [http://acl.ldc.upenn.edu/P/P07/P07-1096.pdf Guided learning for bidirectional sequence classification]. ''Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL 2007)'', pages 760-767.<br />
<br />
* Søgaard, Anders. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). Portland, Oregon.<br />
<br />
* Spoustová, Drahomíra "Johanka", Jan Hajič, Jan Raab and Miroslav Spousta. 2009. Semi-supervised Training for the Averaged Perceptron POS Tagger. Proceedings of the 12 EACL, pages 763-771.<br />
<br />
* Toutanova, K., Klein, D., Manning, C.D., Yoram Singer, Y. 2003. [http://nlp.stanford.edu/kristina/papers/tagging.pdf Feature-rich part-of-speech tagging with a cyclic dependency network]. ''Proceedings of HLT-NAACL 2003'', pages 252-259.<br />
<br />
* Tsuruoka, Yoshimasa, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/pci05.pdf Developing a Robust Part-of-Speech Tagger for Biomedical Text, Advances in Informatics]" - ''10th Panhellenic Conference on Informatics'', '''LNCS 3746''', pp. 382-392, 2005 <br />
<br />
* Tsuruoka, Yoshimasa, Yusuke Miyao, and Jun’ichi Kazama. 2011. "[http://aclweb.org/anthology-new/W/W11/W11-0328.pdf Learning with Lookahead: Can History-Based Models Rival Globally Optimized Models?]" ''Proceedings of the Fifteenth Conference on Computational Natural Language Learning'', pp 238–246, 2011.<br />
<br />
* Tsuruoka, Yoshimasa and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/emnlp05bidir.pdf Bidirectional Inference with the Easiest-First Strategy for Tagging Sequence Data]", ''Proceedings of HLT/EMNLP 2005'', pp. 467-474.<br />
<br />
== See also ==<br />
* [[POS Induction (State of the art)]]<br />
* [[Part-of-speech tagging]]<br />
* [[State of the art]]<br />
<br />
<br />
[[Category:State of the art]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9885Language Identification Tools2012-12-19T14:41:02Z<p>Kiwibird: wops</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ C library (BSD license)<br />
** Interfaces to the C library libtextcat:<br />
*** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
*** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat<br />
*** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat implementation<br />
*** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java reimplementation<br />
** http://thomas.mangin.com//content/texcat-in-python.html – a python implementation by Thomas Mangin<br />
** http://www.mnogosearch.org/guesser/ – another C reimplementation<br />
<br />
<br />
* Languid/GuessLanguage, trigram based<br />
** http://languid.cantbedone.org/ (dead link) original Perl version by Maciej Ceglowski <br />
** http://websvn.kde.org/branches/work/sonnet-refactoring/common/nlp/guesslanguage.cpp?view=markup C++ version by Jacob R Rideout for KDE<br />
** https://bitbucket.org/spirit/guess_language Python3 version by Phi-Long Do, supports Python2 via lib3to2<br />
<br />
<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier Java (Apache 2.0 license)<br />
** https://code.google.com/p/language-detection/ source code, data for 53 languages<br />
** https://code.google.com/p/lang-guess/ lang-guess is a fork of language-detection<br />
<br />
<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
<br />
* LID http://www.cavar.me/damir/LID/ Python and Scheme (GPL3)<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]<br />
* [http://www.let.rug.nl/~vannoord/TextCat/competitors.html TextCat competitors] – list compiled by Gertjan van Noord</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9884Language Identification Tools2012-12-17T11:08:52Z<p>Kiwibird: /* Free Software */</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ C library (BSD license)<br />
** Interfaces to the C library libtextcat:<br />
*** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
*** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat<br />
*** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat implementation<br />
*** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java reimplementation<br />
** http://thomas.mangin.com//content/texcat-in-python.html – a python implementation by Thomas Mangin<br />
** http://www.mnogosearch.org/guesser/ – another C reimplementation<br />
<br />
<br />
* Languid/GuessLanguage, trigram based<br />
** http://languid.cantbedone.org/ (dead link) original Perl version by Maciej Ceglowski <br />
** http://websvn.kde.org/branches/work/sonnet-refactoring/common/nlp/guesslanguage.cpp?view=markup C++ version by Jacob R Rideout for KDE<br />
** https://bitbucket.org/spirit/guess_language Python2 version by Phi-Long Do, supports Python3 via lib3to2<br />
<br />
<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier Java (Apache 2.0 license)<br />
** https://code.google.com/p/language-detection/ source code, data for 53 languages<br />
** https://code.google.com/p/lang-guess/ lang-guess is a fork of language-detection<br />
<br />
<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
<br />
* LID http://www.cavar.me/damir/LID/ Python and Scheme (GPL3)<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]<br />
* [http://www.let.rug.nl/~vannoord/TextCat/competitors.html TextCat competitors] – list compiled by Gertjan van Noord</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9883Language Identification Tools2012-12-17T11:08:43Z<p>Kiwibird: /* Free Software */</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ C library (BSD license)<br />
** Interfaces to the C library libtextcat:<br />
*** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
*** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat<br />
*** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat implementation<br />
*** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java reimplementation<br />
** http://thomas.mangin.com//content/texcat-in-python.html – a python implementation by Thomas Mangin<br />
** http://www.mnogosearch.org/guesser/ – another C reimplementation<br />
<br />
* Languid/GuessLanguage, trigram based<br />
** http://languid.cantbedone.org/ (dead link) original Perl version by Maciej Ceglowski <br />
** http://websvn.kde.org/branches/work/sonnet-refactoring/common/nlp/guesslanguage.cpp?view=markup C++ version by Jacob R Rideout for KDE<br />
** https://bitbucket.org/spirit/guess_language Python2 version by Phi-Long Do, supports Python3 via lib3to2<br />
<br />
<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier Java (Apache 2.0 license)<br />
** https://code.google.com/p/language-detection/ source code, data for 53 languages<br />
** https://code.google.com/p/lang-guess/ lang-guess is a fork of language-detection<br />
<br />
<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
<br />
* LID http://www.cavar.me/damir/LID/ Python and Scheme (GPL3)<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]<br />
* [http://www.let.rug.nl/~vannoord/TextCat/competitors.html TextCat competitors] – list compiled by Gertjan van Noord</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification&diff=9882Language Identification2012-12-17T11:01:58Z<p>Kiwibird: Redirected page to Language Detection</p>
<hr />
<div>#REDIRECT [[Language Detection]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Chinese&diff=9854Resources for Chinese2012-12-06T16:05:18Z<p>Kiwibird: /* Unknown license */</p>
<hr />
<div>==Tools==<br />
===Free software===<br />
* [https://github.com/yzhang/rseg rseg] word segmentation; written in ruby (no compilation, no hard dependencies apart from ruby), comes with a model (MIT license)<br />
* [https://code.google.com/p/ctbparser/ ctbparser] word segmentation, POS tagging, NER, dependency parsing, all using Conditional Random Fields; written in C++ (LGPL license)<br />
* [http://www.cl.cam.ac.uk/~yz360/zpar.html ZPar] word segmentation, POS tagging, CFG/dep/CCG parsing of Chinese and English; written in C++ (GPL3 license)<br />
* [http://code.google.com/p/duduplus/ DuDuPlus: a graph-based dependency parser for English and Chinese] ("Other Open Source" license?)<br />
** where is the source code?<br />
<br />
==Data==<br />
===Free software===<br />
* [http://corpora.heliohost.org/ HC Corpora] 1606811 lines of [http://en.wikipedia.org/wiki/Fair_use Fair Use] excerpts from news, blogs, twitter<br />
<br />
===Unknown license===<br />
* [http://www.chinesecomputing.com Chinese Computing] <br />
* [http://www.icl.pku.edu.cn/icl_groups/corpus/dwldform1.asp Word Segmented and POS tagged People Daily Corpus at ICL of Peking University]<br />
* [http://corpus.leeds.ac.uk/frqc/i-zh-char.num.html Frequency list of characters in the Internet corpus]<br />
* [http://corpus.leeds.ac.uk/frqc/internet-zh.num Frequency list of lexical items in the Internet corpus]<br />
* [http://www.ling.lancs.ac.uk/corplang/lcmc/ Lancaster Corpus of Mandarin Chinese]<br />
<br />
<br />
[[Category:Resources by language|Chinese]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Chinese&diff=9853Resources for Chinese2012-12-06T16:02:55Z<p>Kiwibird: /* Free software */</p>
<hr />
<div>==Tools==<br />
===Free software===<br />
* [https://github.com/yzhang/rseg rseg] word segmentation; written in ruby (no compilation, no hard dependencies apart from ruby), comes with a model (MIT license)<br />
* [https://code.google.com/p/ctbparser/ ctbparser] word segmentation, POS tagging, NER, dependency parsing, all using Conditional Random Fields; written in C++ (LGPL license)<br />
* [http://www.cl.cam.ac.uk/~yz360/zpar.html ZPar] word segmentation, POS tagging, CFG/dep/CCG parsing of Chinese and English; written in C++ (GPL3 license)<br />
* [http://code.google.com/p/duduplus/ DuDuPlus: a graph-based dependency parser for English and Chinese] ("Other Open Source" license?)<br />
** where is the source code?<br />
<br />
==Data==<br />
===Unknown license===<br />
* [http://www.chinesecomputing.com Chinese Computing] <br />
* [http://www.icl.pku.edu.cn/icl_groups/corpus/dwldform1.asp Word Segmented and POS tagged People Daily Corpus at ICL of Peking University]<br />
* [http://corpus.leeds.ac.uk/frqc/i-zh-char.num.html Frequency list of characters in the Internet corpus]<br />
* [http://corpus.leeds.ac.uk/frqc/internet-zh.num Frequency list of lexical items in the Internet corpus]<br />
* [http://www.ling.lancs.ac.uk/corplang/lcmc/ Lancaster Corpus of Mandarin Chinese]<br />
<br />
<br />
[[Category:Resources by language|Chinese]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Chinese&diff=9852Resources for Chinese2012-12-06T16:02:31Z<p>Kiwibird: /* Free software */</p>
<hr />
<div>==Tools==<br />
===Free software===<br />
* [https://github.com/yzhang/rseg rseg] word segmentation, in ruby (no compilation, no hard dependencies apart from ruby), comes with a model (MIT license)<br />
* [https://code.google.com/p/ctbparser/ ctbparser] word segmentation, POS tagging, NER, dependency parsing, all using Conditional Random Fields; written in C++ (LGPL license)<br />
* [http://www.cl.cam.ac.uk/~yz360/zpar.html ZPar] word segmentation, POS tagging, CFG/dep/CCG parsing of Chinese and English; written in C++ (GPL3 license)<br />
* [http://code.google.com/p/duduplus/ DuDuPlus: a graph-based dependency parser for English and Chinese] ("Other Open Source" license?)<br />
** where is the source code?<br />
<br />
==Data==<br />
===Unknown license===<br />
* [http://www.chinesecomputing.com Chinese Computing] <br />
* [http://www.icl.pku.edu.cn/icl_groups/corpus/dwldform1.asp Word Segmented and POS tagged People Daily Corpus at ICL of Peking University]<br />
* [http://corpus.leeds.ac.uk/frqc/i-zh-char.num.html Frequency list of characters in the Internet corpus]<br />
* [http://corpus.leeds.ac.uk/frqc/internet-zh.num Frequency list of lexical items in the Internet corpus]<br />
* [http://www.ling.lancs.ac.uk/corplang/lcmc/ Lancaster Corpus of Mandarin Chinese]<br />
<br />
<br />
[[Category:Resources by language|Chinese]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Chinese&diff=9850Resources for Chinese2012-12-06T15:56:29Z<p>Kiwibird: </p>
<hr />
<div>==Tools==<br />
===Free software===<br />
* [https://github.com/yzhang/rseg rseg] word segmentation, in ruby (no compilation, no hard dependencies apart from ruby), comes with a model (MIT license)<br />
* [http://code.google.com/p/duduplus/ DuDuPlus: a graph-based dependency parser for English and Chinese] ("Other Open Source" – where is the source code though?)<br />
<br />
==Data==<br />
===Unknown license===<br />
* [http://www.chinesecomputing.com Chinese Computing] <br />
* [http://www.icl.pku.edu.cn/icl_groups/corpus/dwldform1.asp Word Segmented and POS tagged People Daily Corpus at ICL of Peking University]<br />
* [http://corpus.leeds.ac.uk/frqc/i-zh-char.num.html Frequency list of characters in the Internet corpus]<br />
* [http://corpus.leeds.ac.uk/frqc/internet-zh.num Frequency list of lexical items in the Internet corpus]<br />
* [http://www.ling.lancs.ac.uk/corplang/lcmc/ Lancaster Corpus of Mandarin Chinese]<br />
<br />
<br />
[[Category:Resources by language|Chinese]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9845Language Identification Tools2012-12-06T09:25:36Z<p>Kiwibird: /* Free Software */</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ C library (BSD license)<br />
** Interfaces to the C library libtextcat:<br />
*** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
*** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat<br />
*** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat implementation<br />
*** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java reimplementation<br />
** http://thomas.mangin.com//content/texcat-in-python.html – a python implementation by Thomas Mangin<br />
** http://www.mnogosearch.org/guesser/ – another C reimplementation<br />
<br />
<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier Java (Apache 2.0 license)<br />
** https://code.google.com/p/language-detection/ source code, data for 53 languages<br />
** https://code.google.com/p/lang-guess/ lang-guess is a fork of language-detection<br />
<br />
<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
<br />
* LID http://www.cavar.me/damir/LID/ Python and Scheme (GPL3)<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]<br />
* [http://www.let.rug.nl/~vannoord/TextCat/competitors.html TextCat competitors] – list compiled by Gertjan van Noord</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9844Language Identification Tools2012-12-06T09:13:44Z<p>Kiwibird: /* Free Software */</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ C library (BSD license)<br />
** Interfaces to the C library libtextcat:<br />
*** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
*** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat<br />
*** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat implementation<br />
*** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java reimplementation<br />
** http://thomas.mangin.com//content/texcat-in-python.html – a python implementation by Thomas Mangin<br />
** http://www.mnogosearch.org/guesser/ – another C reimplementation<br />
<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier Java (Apache 2.0 license)<br />
** https://code.google.com/p/language-detection/ source code, data for 53 languages<br />
** https://code.google.com/p/lang-guess/ lang-guess is a fork of language-detection<br />
<br />
<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
<br />
* LID http://www.cavar.me/damir/LID/ Python and Scheme (GPL3)<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]<br />
* [http://www.let.rug.nl/~vannoord/TextCat/competitors.html TextCat competitors] – list compiled by Gertjan van Noord</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9843Language Identification Tools2012-12-06T09:08:23Z<p>Kiwibird: /* Free Software */</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ C library (BSD license)<br />
** Interfaces to the C library libtextcat:<br />
*** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
*** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat<br />
*** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat implementation<br />
*** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java reimplementation<br />
<br />
<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier Java (Apache 2.0 license)<br />
** https://code.google.com/p/language-detection/ source code, data for 53 languages<br />
** https://code.google.com/p/lang-guess/ lang-guess is a fork of language-detection<br />
<br />
<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
<br />
* LID http://www.cavar.me/damir/LID/ Python and Scheme (GPL3)<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]<br />
* [http://www.let.rug.nl/~vannoord/TextCat/competitors.html TextCat competitors] – list compiled by Gertjan van Noord</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9842Language Identification Tools2012-12-06T09:01:34Z<p>Kiwibird: /* Free Software */</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ (BSD license)<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat<br />
** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java implementation<br />
** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat<br />
** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier Java (Apache 2.0 license)<br />
** https://code.google.com/p/language-detection/ source code, data for 53 languages<br />
** https://code.google.com/p/lang-guess/ lang-guess is a fork of language-detection<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
* LID http://www.cavar.me/damir/LID/ Python and Scheme (GPL3)<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]<br />
* [http://www.let.rug.nl/~vannoord/TextCat/competitors.html TextCat competitors] – list compiled by Gertjan van Noord</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9841Language Identification Tools2012-12-06T08:59:19Z<p>Kiwibird: /* See also */</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ (BSD license)<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat<br />
** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java implementation<br />
** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat<br />
** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier (Apache license)<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]<br />
* [http://www.let.rug.nl/~vannoord/TextCat/competitors.html TextCat competitors] – list compiled by Gertjan van Noord</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_(State_of_the_art)&diff=9840Language Identification (State of the art)2012-12-06T08:56:09Z<p>Kiwibird: /* See also */</p>
<hr />
<div>== "Standard" measure: ==<br />
<br />
== "Standard" datasets: ==<br />
<br />
<br />
{{StateOfTheArtTable}}<br />
<br />
| SystemName || How does it work? || Author and Article [http://www.bla.com] || Software? || 98% according to... || Any extra comments? <br />
|-<br />
| textcat || n-gram matching || Cavnar, W. B. and J. M. Trenkle (1994) "[http://www.nonlineardynamics.com/trenkle/papers/sdr94ps.gz N-Gram-Based Text Categorization]" || Yes || - || -<br />
|-<br />
<br />
|}<br />
<br />
==See also==<br />
* [[Language Identification Tools]]<br />
<br />
[[Category:State of the art]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_(State_of_the_art)&diff=9839Language Identification (State of the art)2012-12-06T08:55:54Z<p>Kiwibird: </p>
<hr />
<div>== "Standard" measure: ==<br />
<br />
== "Standard" datasets: ==<br />
<br />
<br />
{{StateOfTheArtTable}}<br />
<br />
| SystemName || How does it work? || Author and Article [http://www.bla.com] || Software? || 98% according to... || Any extra comments? <br />
|-<br />
| textcat || n-gram matching || Cavnar, W. B. and J. M. Trenkle (1994) "[http://www.nonlineardynamics.com/trenkle/papers/sdr94ps.gz N-Gram-Based Text Categorization]" || Yes || - || -<br />
|-<br />
<br />
|}<br />
<br />
==See also==<br />
[[Language Identification Tools]]<br />
<br />
[[Category:State of the art]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Detection&diff=9838Language Detection2012-12-06T08:55:30Z<p>Kiwibird: Redirected page to Language Identification Tools</p>
<hr />
<div>#REDIRECT [[Language Identification Tools]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9837Language Identification Tools2012-12-06T08:54:28Z<p>Kiwibird: /* Free Software */</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ (BSD license)<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat<br />
** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java implementation<br />
** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat<br />
** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier (Apache license)<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9836Language Identification Tools2012-12-06T08:52:18Z<p>Kiwibird: /* Free Software */</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* LibTextCat http://software.wise-guys.nl/libtextcat/ (BSD license)<br />
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat<br />
** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes<br />
** http://olivo.net/software/lc4j/ – a java implementation<br />
** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier (Apache license)<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9835Language Identification Tools2012-12-06T08:49:32Z<p>Kiwibird: </p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[List of resources by language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
==Free Software==<br />
* TextCat<br />
** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat/LM - language models for the perl version<br />
** http://olivo.net/software/lc4j/ - a java implementation<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier (Apache license)<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Language_Identification_Tools&diff=9834Language Identification Tools2012-12-06T08:48:45Z<p>Kiwibird: Created page with "A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch)..."</p>
<hr />
<div>A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).<br />
<br />
Most of these tools require training on a big corpus (see [[Category:Resources_by_language]] for corpora per language), but many come with some prebuilt language models.<br />
<br />
<br />
==Free Software==<br />
* TextCat<br />
** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat/LM - language models for the perl version<br />
** http://olivo.net/software/lc4j/ - a java implementation<br />
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier (Apache license)<br />
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)<br />
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google<br />
<br />
==Proprietary==<br />
* Google Language Identification API<br />
* Lingua-Systems lid http://www.lingua-systems.com/language-identifier/<br />
<br />
==See also==<br />
* [[Language Identification (State of the art)]]<br />
* [https://en.wikipedia.org/wiki/Language_detection English Wikipedia on Language detection]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Polish&diff=9623Resources for Polish2012-08-06T10:41:19Z<p>Kiwibird: /* Free/Open Source Software */</p>
<hr />
<div>==Corpora==<br />
* [http://korpus.pl/en/ IPI PAN Corpus] - The IPI PAN Corpus is a large (currently over 250 million segments), morphosyntactically annotated, publicly available corpus of Polish, developed by the Linguistic Engineering Group at the Institute of Computer Science, Polish Academy of Sciences (ICS PAS)<br />
* [http://korpus.pwn.pl/index_en.php PWN Corpus] - PWN has prepared and made available an online version of the Corpus of Polish consisting of 40 million words. The samples were taken from 386 books, 977 editions selected from 185 different press publications, 84 transcribed spoken texts, 207 web sites and several hundred advertising leaflets and other ephemera. The full version of the corpus is available on payment for access, while a demonstration version of over 7.5 million words is available free of charge.<br />
<br />
==Taggers, parsers, morphology analysers==<br />
<br />
==Free/Open Source Software==<br />
* [http://morfologik.blogspot.com/ Morfologik] -- morphological dictionary by Marcin Miłkowski (of LanguageTool), licensed under CC-SA / GNU LGPL<br />
** [http://nlp.pwr.wroc.pl/redmine/projects/libpltagger/wiki/Morfologik_converted Morfologik converted to the IKIPI tagset] (the tagset of the IPI PAN Corpus)<br />
* [http://nlp.pwr.wroc.pl/en/tools-and-resources/narzedzia-przetwarzania-morfosyntaktycznego Morphosyntactic Toolchain] by WrocUT Language Technology Group G4.19, licensed under GNU LGPL (some optional addons are GNU GPL). Command-line utilities providing tokenisation, morphological analysis, morphosyntactic tagging, shallow parsing (chunking), WCCL feature vectors for machine learning.<br />
<br />
==Unknown license==<br />
* [http://nlp.ipipan.waw.pl/~wolinski/morfeusz/ "Morfeusz"] - morphological analyser of Polish (Wolinski, 2005), <br />
** [http://www.springerlink.com/content/l101v8823391j568/ main reference] Morfeusz — a Practical Tool for the Morphological Analysis of Polish <br />
* "AMOR" - morphology analyser of Polish (Joanna Rabiega, 2000), <br />
** [http://members.chello.pl/jrw/doc/jr_ma.pdf/ main reference] Podstawy lingwistyczne automatycznego analizatora morfologicznego AMOR <br />
* [http://duch.mimuw.edu.pl/~kszafran/index.php?option=com_docman&task=cat_view&gid=33&Itemid=43 "SAM"] - morphological analyser of Polish (Krzysztof Szafran, 1994), <br />
* [http://sourceforge.net/project/showfiles.php?group_id=166344 Morfologik] - Polish morphological analyzer based on current ispell dictionaries, and Java libraries interfacing it. First completely open-source and comprehensive morphological tools for Polish. Will be used for grammar correction tools (to be included in the future) <br />
* [http://nlp.ipipan.waw.pl/Spejd/ Spejd - Shallow Parsing and Disambiguation Engine] <br />
* [http://www.cs.put.poznan.pl/dweiss/xml/projects/lametyzator/index.xml lemmatizer] - Dawid Weiss<br />
<br />
==Lexical resources==<br />
<br />
<br />
==Bibliography==<br />
<br />
<br />
==External links==<br />
* [http://bach.ipipan.waw.pl/mailman/listinfo/ling Polish linguistics mailing list] - mainly in Polish<br />
<br />
<br />
[[Category:Resources by language|Polish]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Talk:Resources_for_German&diff=9580Talk:Resources for German2012-07-15T08:40:06Z<p>Kiwibird: Created page with "More resources discussed at http://thread.gmane.org/gmane.science.linguistics.corpora/16011/focus=16025"</p>
<hr />
<div>More resources discussed at http://thread.gmane.org/gmane.science.linguistics.corpora/16011/focus=16025</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=POS_Tagging_(State_of_the_art)&diff=9383POS Tagging (State of the art)2012-06-15T14:03:49Z<p>Kiwibird: /* WSJ */</p>
<hr />
<div>==Test collections==<br />
* '''Performance measure:''' per token accuracy. (The convention is for this to be measured on all tokens, including punctuation tokens and other unambiguous tokens.)<br />
* '''English'''<br />
** '''Penn Treebank''' ''Wall Street Journal'' (WSJ) release 3 (LDC99T42). The splits of data for this task were not standardized early on (unlike for parsing) and early work uses various data splits defined by counts of tokens or by sections. Most work from 2002 on adopts the following data splits, introduced by Collins (2002):<br />
*** '''Training data:''' sections 0-18<br />
*** '''Development test data:''' sections 19-21<br />
*** '''Testing data:''' sections 22-24<br />
<br />
* '''French'''<br />
** '''French TreeBank''' (FTB, Abeillé et al; 2003) ''Le Monde'', December 2007 version, 28-tag tagset (CC tagset, Crabbé and Candito, 2008). Classical data split (10-10-80):<br />
*** '''Training data:''' sentences 2471 to 12351<br />
*** '''Development test data:''' sentences 1236 to 2470<br />
*** '''Testing data:''' sentences 1 to 1235<br />
<br />
<br />
== Tables of results ==<br />
<br />
===WSJ===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
| License<br />
|-<br />
| TnT*<br />
| Hidden markov model<br />
| Brants (2000)<br />
| [http://www.coli.uni-saarland.de/~thorsten/tnt/ TnT]<br />
| No<br />
| 96.46%<br />
| 85.86%<br />
| Unknown<br />
|-<br />
| MElt<br />
| MEMM with external lexical information<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 96.96%<br />
| 91.29%<br />
| Unknown<br />
|-<br />
| GENiA Tagger**<br />
| Maximum entropy cyclic dependency network<br />
| Tsuruoka, et al (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/GENIA/tagger/ GENiA]<br />
| No<br />
| 97.05%<br />
| Not available<br />
| Gratis for non-commercial usage<br />
|-<br />
| Averaged Perceptron<br />
| Averaged Perception discriminative sequence model<br />
| Collins (2002)<br />
| Not available<br />
| No<br />
| 97.11%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Maxent easiest-first<br />
| Maximum entropy bidirectional easiest-first inference<br />
| Tsuruoka and Tsujii (2005)<br />
| [http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/postagger/ Easiest-first]<br />
| No<br />
| 97.15%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SVMTool<br />
| SVM-based tagger and tagger generator<br />
| Giménez and Márquez (2004)<br />
| [http://www.lsi.upc.es/~nlp/SVMTool/ SVMTool]<br />
| No<br />
| 97.16%<br />
| 89.01%<br />
| Unknown<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost]<br />
| No<br />
| 97.23%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Stanford Tagger 1.0<br />
| Maximum entropy cyclic dependency network<br />
| Toutanova et al. (2003)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.24%<br />
| 89.04%<br />
| Unknown<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| No<br />
| 97.29%<br />
| 89.70%<br />
| Unknown<br />
|-<br />
| Stanford Tagger 2.0<br />
| Maximum entropy cyclic dependency network<br />
| Manning (2011)<br />
| [http://nlp.stanford.edu/software/tagger.shtml Stanford Tagger]<br />
| Yes<br />
| 97.32%<br />
| 90.79%<br />
| Unknown<br />
|-<br />
| LTAG-spinal<br />
| Bidirectional perceptron learning<br />
| Shen et al. (2007)<br />
| [http://www.cis.upenn.edu/~xtag/spinal/ LTAG-spinal]<br />
| No<br />
| 97.33%<br />
| Not available<br />
| Unknown<br />
|-<br />
| Morče/COMPOST<br />
| Averaged Perceptron<br />
| Spoustová et al. (2009)<br />
| [http://ufal.mff.cuni.cz/compost]<br />
| Yes<br />
| 97.44%<br />
| Not available<br />
| Unknown<br />
|-<br />
| SCCN<br />
| Semi-supervised condensed nearest neighbor<br />
| Søgaard (2011)<br />
| [http://cst.dk/anders/scnn/ SCCN]<br />
| Yes<br />
| 97.50%<br />
| Not available<br />
| Unknown<br />
|}<br />
<br />
(*) TnT: Accuracy is as reported by Giménez and Márquez (2004) for the given test collection. Brants (2000) reports 96.7% token accuracy and 85.5% unknown word accuracy on a 10-fold cross-validation of the Penn WSJ corpus.<br />
<br />
(**) GENiA: Results are for models trained and tested on the given corpora (to be comparable to other results). The distributed GENiA tagger is trained on a mixed training corpus and gets 96.94% on WSJ, and 98.26% on GENiA biomedical English.<br />
<br />
(***) Extra data: Whether system training exploited (usually large amounts of) extra unlabeled text, such as by semi-supervised learning, self-training, or using distributional similarity features, beyond the standard supervised training data.<br />
<br />
===FTB===<br />
<br />
{| border="1" cellpadding="5" cellspacing="1" width="100%"<br />
|-<br />
! System name<br />
! Short description<br />
! Main publication<br />
! Software<br />
! Extra Data?***<br />
! All tokens<br />
! Unknown words<br />
|-<br />
| Morfette<br />
| Perceptron with external lexical information*<br />
| Chrupała et al. (2008), Seddah et al. (2010)<br />
| [http://sites.google.com/site/morfetteweb/ Morfette]<br />
| No<br />
| 97.68%<br />
| 90.52%<br />
|-<br />
| SEM<br />
| CRF with external lexical information*<br />
| Constant et al. (2011)<br />
| [http://www.univ-orleans.fr/lifo/Members/Isabelle.Tellier/SEM.html SEM]<br />
| No<br />
| 97.7%<br />
| Not available<br />
|-<br />
| MElt<br />
| MEMM with external lexical information*<br />
| Denis and Sagot (2009)<br />
| [https://gforge.inria.fr/projects/lingwb/ Alpage linguistic workbench]<br />
| No<br />
| 97.80%<br />
| 91.77%<br />
|}<br />
<br />
(*) External lexical information from the Lefff lexicon (Sagot 2010, [https://gforge.inria.fr/frs/?group_id=482 Alexina project])<br />
<br />
<br />
== References ==<br />
<br />
* Brants, Thorsten. 2000. [http://acl.ldc.upenn.edu/A/A00/A00-1031.pdf TnT -- A Statistical Part-of-Speech Tagger]. "6th Applied Natural Language Processing Conference".<br />
<br />
* Chrupała, Grzegorz, Dinu, Georgiana and van Genabith, Josef. 2008. [http://www.lrec-conf.org/proceedings/lrec2008/pdf/594_paper.pdf Learning Morphology with Morfette]. "LREC 2008". <br />
<br />
* Collins, Michael. 2002. [http://people.csail.mit.edu/mcollins/papers/tagperc.pdf Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms]. ''EMNLP 2002''.<br />
<br />
* Constant, Matthieu, Tellier, Isabelle, Duchier, Denys, Dupont, Yoann, Sigogne, Anthony, and Billot, Sylvie. [http://www.lirmm.fr/~lopez/TALN2011/Longs-TALN+RECITAL/Tellier_taln11_submission_54.pdf Intégrer des connaissances linguistiques dans un CRF : application à l'apprentissage d'un segmenteur-étiqueteur du français]. "TALN'11"<br />
<br />
* Denis, Pascal and Sagot, Benoît. 2009. [http://alpage.inria.fr/~sagot/pub/paclic09tagging.pdf Coupling an annotated corpus and a morphosyntactic lexicon for state-of-the-art POS tagging with less human effort]. "PACLIC 2009"<br />
<br />
* Giménez, J., and Márquez, L. 2004. [http://www.lsi.upc.es/~nlp/SVMTool/lrec2004-gm.pdf SVMTool: A general POS tagger generator based on Support Vector Machines]. ''Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC'04)''. Lisbon, Portugal. <br />
<br />
* Manning, Christopher D. 2011. Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In Alexander Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing, 12th International Conference, CICLing 2011, Proceedings, Part I. Lecture Notes in Computer Science 6608, pp. 171--189. Springer.<br />
<br />
* Seddah, Djamé, Chrupała, Grzegorz, Çetinoglu, Özlem and Candito, Marie. 2010. [http://aclweb.org/anthology-new/W/W10/W10-1410.pdf Lemmatization and Lexicalized Statistical Parsing of Morphologically Rich Languages: the Case of French] "SPMRL 2010 (NAACL 2010 workshop)"<br />
<br />
* Shen, L., Satta, G., and Joshi, A. 2007. [http://acl.ldc.upenn.edu/P/P07/P07-1096.pdf Guided learning for bidirectional sequence classification]. ''Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL 2007)'', pages 760-767.<br />
<br />
* Søgaard, Anders. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). Portland, Oregon.<br />
<br />
* Spoustová, Drahomíra "Johanka", Jan Hajič, Jan Raab and Miroslav Spousta. 2009. Semi-supervised Training for the Averaged Perceptron POS Tagger. Proceedings of the 12 EACL, pages 763-771.<br />
<br />
* Toutanova, K., Klein, D., Manning, C.D., Yoram Singer, Y. 2003. [http://nlp.stanford.edu/kristina/papers/tagging.pdf Feature-rich part-of-speech tagging with a cyclic dependency network]. ''Proceedings of HLT-NAACL 2003'', pages 252-259.<br />
<br />
* Tsuruoka, Yoshimasa, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/pci05.pdf Developing a Robust Part-of-Speech Tagger for Biomedical Text, Advances in Informatics]" - ''10th Panhellenic Conference on Informatics'', '''LNCS 3746''', pp. 382-392, 2005 <br />
<br />
* Tsuruoka, Yoshimasa and Jun'ichi Tsujii. 2005. "[http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/papers/emnlp05bidir.pdf Bidirectional Inference with the Easiest-First Strategy for Tagging Sequence Data]", ''Proceedings of HLT/EMNLP 2005'', pp. 467-474.<br />
<br />
== See also ==<br />
* [[POS Induction (State of the art)]]<br />
* [[Part-of-speech tagging]]<br />
* [[State of the art]]<br />
<br />
<br />
[[Category:State of the art]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Slovenian&diff=9377Resources for Slovenian2012-05-30T12:36:47Z<p>Kiwibird: /* Corpora */</p>
<hr />
<div>==Corpora==<br />
* [http://nl.ijs.si/elan/ IJS - ELAN] Slovene-English Parallel Corpus<br />
** License: "freely available for downloading, but please acknowledge in any publications"<br />
<br />
* [http://nl.ijs.si/ME/ Multext EAST] lexica, annotated "1984" corpus, parallel and comparable text and speech corpora. <br />
** License: "research use only"<br />
** Languages involved: Bulgarian, Croatian, Czech, English, Estonian, Hungarian, Lithuanian, Macedonian, Persian, Polish, Resian, Romanian, Russian, Serbian, Slovak, Slovene, and Ukrainian<br />
<br />
* [http://langtech.jrc.it/JRC-Acquis.html JRC Acquis] parallel texts. <br />
** License: Public domain.<br />
** Languages involved: Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovene and Swedish. <br />
<br />
[[Category:Resources by language|Solvenian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Slovenian&diff=9376Resources for Slovenian2012-05-30T12:34:42Z<p>Kiwibird: /* Corpora */</p>
<hr />
<div>==Corpora==<br />
* [http://nl.ijs.si/elan/ Slovene-English IJS - ELAN Parallel Corpus] License: "freely available for downloading, but please acknowledge in any publications"<br />
<br />
* [http://nl.ijs.si/ME/ Multext EAST] License: "research use only"<br />
<br />
* [http://langtech.jrc.it/JRC-Acquis.html JRC Acquis] parallel texts in the following 22 languages: Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovene and Swedish. License: Public domain.<br />
<br />
[[Category:Resources by language|Solvenian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Slovenian&diff=9375Resources for Slovenian2012-05-30T12:31:55Z<p>Kiwibird: /* Corpora */</p>
<hr />
<div>==Corpora==<br />
* [http://nl.ijs.si/elan/ Slovene-English IJS - ELAN Parallel Corpus] License: "freely available for downloading, but please acknowledge in any publications"<br />
* [http://langtech.jrc.it/JRC-Acquis.html JRC Acquis] parallel texts in the following 22 languages: Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovene and Swedish. License: Public domain.<br />
<br />
[[Category:Resources by language|Solvenian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Slovenian&diff=9374Resources for Slovenian2012-05-30T12:30:40Z<p>Kiwibird: /* Corpora */</p>
<hr />
<div>==Corpora==<br />
* [http://nl.ijs.si/elan/ Slovene-English Parallel Corpus] License: "freely available for downloading, but please acknowledge in any publications"<br />
* [http://langtech.jrc.it/JRC-Acquis.html JRC Acquis] parallel texts in the following 22 languages: Bulgarian, Czech, Danish, German, Greek, English, Spanish, Estonian, Finnish, French, Hungarian, Italian, Lithuanian, Latvian, Maltese, Dutch, Polish, Portuguese, Romanian, Slovak, Slovene and Swedish. License: Public domain.<br />
<br />
[[Category:Resources by language|Solvenian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Norwegian&diff=9231Resources for Norwegian2012-02-23T14:10:16Z<p>Kiwibird: /* Free software */</p>
<hr />
<div>==Corpora==<br />
===Free software===<br />
<br />
===Proprietary===<br />
* [http://corpora.informatik.uni-leipzig.de/ Norwegian plain text and Co-occurrences at LCC] ("the corpora may be used for scientific purposes only and not passed on to third parties")<br />
<br />
==Timeline Analysis==<br />
* [http://wortschatz.uni-leipzig.de/wdtno/ Ord I Dag]<br />
<br />
==Machine translation systems==<br />
<br />
===Free software===<br />
<br />
* [http://www.apertium.org Apertium] Norwegian Nynorsk<->Norwegian Bokmål, GPL v2<br />
** [http://wiki.apertium.org/wiki/Apertium-nn-nb wiki] with installation information etc.<br />
<br />
===Proprietary===<br />
<br />
==Lexical resources==<br />
===Free software===<br />
* [http://svn.emmtee.net/tags/topp/parc/pargram/norwegian/bokmal/bokmal-nkllex.lfg Bokmål LFG lexicon] with POS and count/mass, GPL<br />
* [http://www.edd.uio.no/prosjekt/ordbanken/ Norsk ordbank], full form dictionaries for Nynorsk (106,789 lemmata) and Bokmål (142,899 lemmata), GPL <br />
** [http://savannah.nongnu.org/projects/ordbanken/ alternative download with cli lookup interface]<br />
* [http://www.nb.no/spraakbanken/tilgjengelege-ressursar/leksikalske-databasar SCARRIE, Bokmål full form dictionary], XML, about 75,000 lemmata, CC-BY unported<br />
<br />
===Unknown license===<br />
* [http://www.nb.no/spraakbanken/tilgjengelege-ressursar/leksikalske-databasar "Leksikalsk database for norsk, opphavleg produsert av NST"], lexical database with SAMPA transcriptions, meant for speech technology<br />
<br />
==Parsing/disambiguation==<br />
===Free software===<br />
* [http://www.hf.uio.no/tekstlab/tagger.html Oslo-Bergen-taggeren], [[Constraint Grammar]] disambiguator, GPL<br />
** [https://github.com/noklesta/The-Oslo-Bergen-Tagger source and packages on github]<br />
** [http://maximos.aksis.uib.no/Aksis-wiki/Oslo-Bergen_Tagger older alternative download site]<br />
** [http://apertium.svn.sourceforge.net/viewvc/apertium/trunk/apertium-nn-nb/ the version used in Apertium]<br />
** [https://github.com/ogrim/clj-obt Clojure bindings]<br />
<br />
<br />
* [http://www.hf.ntnu.no/hf/isk/Ansatte/petter.haugereid/norsyg.html Norsyg], [[HPSG]] grammar for Norwegian bokmål, LGPL. Implemented in [[LKB]], works with the full ''Norsk ordbank'' lexicon.<br />
<br />
[[Category:Resources by language|Norwegian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Norwegian&diff=9230Resources for Norwegian2012-02-23T14:08:58Z<p>Kiwibird: /* Free software */</p>
<hr />
<div>==Corpora==<br />
===Free software===<br />
<br />
===Proprietary===<br />
* [http://corpora.informatik.uni-leipzig.de/ Norwegian plain text and Co-occurrences at LCC] ("the corpora may be used for scientific purposes only and not passed on to third parties")<br />
<br />
==Timeline Analysis==<br />
* [http://wortschatz.uni-leipzig.de/wdtno/ Ord I Dag]<br />
<br />
==Machine translation systems==<br />
<br />
===Free software===<br />
<br />
* [http://www.apertium.org Apertium] Norwegian Nynorsk<->Norwegian Bokmål, GPL v2<br />
** [http://wiki.apertium.org/wiki/Apertium-nn-nb wiki] with installation information etc.<br />
<br />
===Proprietary===<br />
<br />
==Lexical resources==<br />
===Free software===<br />
* [http://svn.emmtee.net/tags/topp/parc/pargram/norwegian/bokmal/bokmal-nkllex.lfg Bokmål LFG lexicon] with POS and count/mass, GPL<br />
* [http://www.edd.uio.no/prosjekt/ordbanken/ Norsk ordbank], full form dictionaries for Nynorsk (106,789 lemmata) and Bokmål (142,899 lemmata), GPL <br />
** [http://savannah.nongnu.org/projects/ordbanken/ alternative download with cli lookup interface]<br />
* [http://www.nb.no/spraakbanken/tilgjengelege-ressursar/leksikalske-databasar SCARRIE, Bokmål full form dictionary], XML, about 75,000 lemmata, CC-BY unported<br />
<br />
===Unknown license===<br />
* [http://www.nb.no/spraakbanken/tilgjengelege-ressursar/leksikalske-databasar "Leksikalsk database for norsk, opphavleg produsert av NST"], lexical database with SAMPA transcriptions, meant for speech technology<br />
<br />
==Parsing/disambiguation==<br />
===Free software===<br />
* [http://www.hf.uio.no/tekstlab/tagger.html Oslo-Bergen-taggeren], [[Constraint Grammar]] disambiguator, GPL<br />
** [http://maximos.aksis.uib.no/Aksis-wiki/Oslo-Bergen_Tagger download], [http://apertium.svn.sourceforge.net/viewvc/apertium/trunk/apertium-nn-nb/ the version used in Apertium]<br />
<br />
* [http://www.hf.ntnu.no/hf/isk/Ansatte/petter.haugereid/norsyg.html Norsyg], [[HPSG]] grammar for Norwegian bokmål, LGPL. Implemented in [[LKB]], works with the full ''Norsk ordbank'' lexicon.<br />
<br />
[[Category:Resources by language|Norwegian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Norwegian&diff=9229Resources for Norwegian2012-02-23T14:06:58Z<p>Kiwibird: /* Free software */</p>
<hr />
<div>==Corpora==<br />
===Free software===<br />
<br />
===Proprietary===<br />
* [http://corpora.informatik.uni-leipzig.de/ Norwegian plain text and Co-occurrences at LCC] ("the corpora may be used for scientific purposes only and not passed on to third parties")<br />
<br />
==Timeline Analysis==<br />
* [http://wortschatz.uni-leipzig.de/wdtno/ Ord I Dag]<br />
<br />
==Machine translation systems==<br />
<br />
===Free software===<br />
<br />
* [http://www.apertium.org Apertium] Norwegian Nynorsk<->Norwegian Bokmål, GPL v2<br />
** [http://wiki.apertium.org/wiki/Apertium-nn-nb wiki] with installation information etc.<br />
<br />
===Proprietary===<br />
<br />
==Lexical resources==<br />
===Free software===<br />
* [http://svn.emmtee.net/tags/topp/parc/pargram/norwegian/bokmal/bokmal-nkllex.lfg Bokmål LFG lexicon] with POS and count/mass, GPL<br />
* [http://www.edd.uio.no/prosjekt/ordbanken/ Norsk ordbank], full form dictionaries for Nynorsk (106,789 lemmata) and Bokmål (142,899 lemmata), GPL <br />
** [http://savannah.nongnu.org/projects/ordbanken/ alternative download with cli lookup interface]<br />
* [http://www.nb.no/spraakbanken/tilgjengelege-ressursar/leksikalske-databasar SCARRIE, Bokmål full form dictionary], XML, about 75,000 lemmata, CC-BY unported<br />
<br />
==Parsing/disambiguation==<br />
===Free software===<br />
* [http://www.hf.uio.no/tekstlab/tagger.html Oslo-Bergen-taggeren], [[Constraint Grammar]] disambiguator, GPL<br />
** [http://maximos.aksis.uib.no/Aksis-wiki/Oslo-Bergen_Tagger download], [http://apertium.svn.sourceforge.net/viewvc/apertium/trunk/apertium-nn-nb/ the version used in Apertium]<br />
<br />
* [http://www.hf.ntnu.no/hf/isk/Ansatte/petter.haugereid/norsyg.html Norsyg], [[HPSG]] grammar for Norwegian bokmål, LGPL. Implemented in [[LKB]], works with the full ''Norsk ordbank'' lexicon.<br />
<br />
[[Category:Resources by language|Norwegian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Norwegian&diff=9228Resources for Norwegian2012-02-23T14:06:47Z<p>Kiwibird: /* Free software */</p>
<hr />
<div>==Corpora==<br />
===Free software===<br />
<br />
===Proprietary===<br />
* [http://corpora.informatik.uni-leipzig.de/ Norwegian plain text and Co-occurrences at LCC] ("the corpora may be used for scientific purposes only and not passed on to third parties")<br />
<br />
==Timeline Analysis==<br />
* [http://wortschatz.uni-leipzig.de/wdtno/ Ord I Dag]<br />
<br />
==Machine translation systems==<br />
<br />
===Free software===<br />
<br />
* [http://www.apertium.org Apertium] Norwegian Nynorsk<->Norwegian Bokmål, GPL v2<br />
** [http://wiki.apertium.org/wiki/Apertium-nn-nb wiki] with installation information etc.<br />
<br />
===Proprietary===<br />
<br />
==Lexical resources==<br />
===Free software===<br />
* [http://svn.emmtee.net/tags/topp/parc/pargram/norwegian/bokmal/bokmal-nkllex.lfg Bokmål LFG lexicon] with POS and count/mass, GPL<br />
* [http://www.edd.uio.no/prosjekt/ordbanken/ Norsk ordbank], full form dictionaries for Nynorsk (106,789 lemmata) and Bokmål (142,899 lemmata), GPL <br />
** [http://savannah.nongnu.org/projects/ordbanken/ alternative download with cli lookup interface]<br />
* [http://www.nb.no/spraakbanken/tilgjengelege-ressursar/leksikalske-databasar SCARRIE, Bokmål full form dictionary), XML, about 75,000 lemmata, CC-BY unported<br />
<br />
==Parsing/disambiguation==<br />
===Free software===<br />
* [http://www.hf.uio.no/tekstlab/tagger.html Oslo-Bergen-taggeren], [[Constraint Grammar]] disambiguator, GPL<br />
** [http://maximos.aksis.uib.no/Aksis-wiki/Oslo-Bergen_Tagger download], [http://apertium.svn.sourceforge.net/viewvc/apertium/trunk/apertium-nn-nb/ the version used in Apertium]<br />
<br />
* [http://www.hf.ntnu.no/hf/isk/Ansatte/petter.haugereid/norsyg.html Norsyg], [[HPSG]] grammar for Norwegian bokmål, LGPL. Implemented in [[LKB]], works with the full ''Norsk ordbank'' lexicon.<br />
<br />
[[Category:Resources by language|Norwegian]]</div>Kiwibirdhttps://aclweb.org/aclwiki/index.php?title=Resources_for_Japanese&diff=9082Resources for Japanese2011-12-02T07:56:07Z<p>Kiwibird: /* Multilingual */</p>
<hr />
<div>There is a very good list at Kyoto University: [http://www-lab25.kuee.kyoto-u.ac.jp/NLP_Portal/lr-cat-e.html Catalogue of Language Resources and Tools in Japan]<br />
<br />
==Corpora==<br />
===Proprietary===<br />
* [http://corpora.informatik.uni-leipzig.de/ Japanese plain text and Co-occurrences at LCC] (downloadable and web-searchable, but only for non-commercial use)<br />
* [http://www.ninjal.ac.jp/english/products/bccwj/ Balanced Corpus of Contemporary Written Japanese (BCCWJ)] (subset is web searchable at Kotonoha)<br />
<br />
===Free/Open Licence===<br />
====Multilingual====<br />
* [http://www.edrdg.org/projects/tanaka/tanakacorpus.html Tanaka Corpus] by Jim Breen, under a CC-BY-SA 3.0 licence<br />
** [http://tatoeba.org/eng/home Tatoeba] Updated version of the Tanaka Corpus; ≈150,000 sentence pairs (CC-BY)<br />
* [http://alaginrc.nict.go.jp/WikiCorpus/index_E.html Japanese-English Bilingual Corpus of Wikipedia's Kyoto Articles] ≈500,000 pairs of manually-translated sentences (CC-BY 3.0)<br />
* [http://id.ndl.go.jp/auth/ndlsh National Diet Library Subject Headers] Japanese Subject Headers, with paraphrases including English Translations([http://id.ndl.go.jp/auth/docs/about-ndlsh#03 non-commercial attribution])<br />
* [http://mastarpj.nict.go.jp/~mutiyama/align/index.html English-Japanese Translation Alignment Data] aligned by [http://mastarpj.nict.go.jp/~mutiyama/ Masao Utiyama] (GFDL, CC-by-nc 1.0)<br />
* [http://nlpwww.nict.go.jp/wn-ja/index.en.html WordNet Definitions and Glosses] ≈180,000 sentence/phrase pairs (WordNet license, similar to BSD)<br />
* [http://www.phontron.com/kftt/#alignments The Kyoto Free Translation Task (KFTT)] by Graham Neubig, 1235 sentences of Japanese-English manually word-aligned<br />
<br />
====Monolingual====<br />
* [http://www-lab25.kuee.kyoto-u.ac.jp/NLP_Portal/lr-cat-e.html#jp:knb_corpus Kyoto University and NTT Blog Corpus]<br />
<br />
== Grammars ==<br />
===Free/Open Licence===<br />
* [http://wiki.delph-in.net/moin/JacyTop Jacy HPSG grammar] MIT Licence<br />
===Unknown licence===<br />
* [[Generation grammars|KPML generation grammar]] (downloadable)<br />
<br />
==Dictionaries==<br />
===Free/Open Licence===<br />
* [http://www.csse.monash.edu.au/~jwb/edict.html EDICT] Japanese-English dictionary, by Jim Breen, (CC-BY-SA 3.0 licence)<br />
* [http://www.csse.monash.edu.au/~jwb/enamdict_doc.html ENAMDICT/JMnedict] proper name dictionary, by Jim Breen, (CC-BY-SA 3.0 licence)<br />
* [http://nlpwww.nict.go.jp/wn-ja/index.en.html Japanese version of WordNet] by NICT, (WordNet license, like BSD)<br />
<br />
===Unknown licence===<br />
* [http://www.csse.monash.edu.au/~jwb/afaq/jitadoushi.html List of Japanese transitive/intransitive verb pairs] (dead link?)<br />
<br />
[[Category:Resources by language|Japanese]]</div>Kiwibird