Difference between revisions of "Language Identification Tools"

From ACL Wiki
Jump to: navigation, search
(Free Software)
m (Free Software)
(2 intermediate revisions by one user not shown)
Line 4: Line 4:
  
 
==Free Software==
 
==Free Software==
* LibTextCat http://software.wise-guys.nl/libtextcat/ (BSD license)
+
* LibTextCat http://software.wise-guys.nl/libtextcat/ C library (BSD license)
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat
+
** Interfaces to the C library libtextcat:
** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes
+
*** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat
** http://olivo.net/software/lc4j/ – a java implementation
+
*** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat
** http://www.jedi.be/pages/JTextCat/ – a java interface to libtextcat
+
*** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat
** https://github.com/crodas/PHPTextCat/ – a php module for libtextcat
+
** http://odur.let.rug.nl/~vannoord/TextCat/ – original perl TextCat implementation
** https://launchpad.net/pylibtextcat python2 / https://github.com/bbqsrc/pylibtextcat/ python3 interface to libtextcat
+
*** http://opus.lingfil.uu.se/tools/public/language_guesser/textcat – perl version with more language models, encoding fixes
 +
** http://olivo.net/software/lc4j/ – a java reimplementation
 +
** http://thomas.mangin.com//content/texcat-in-python.html – a python implementation by Thomas Mangin
 +
** http://www.mnogosearch.org/guesser/ – another C reimplementation
 +
 
 +
 
 
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier Java (Apache 2.0 license)
 
* Nutch Language Identifier https://wiki.apache.org/nutch/LanguageIdentifier Java (Apache 2.0 license)
 
** https://code.google.com/p/language-detection/ source code, data for 53 languages
 
** https://code.google.com/p/language-detection/ source code, data for 53 languages
 
** https://code.google.com/p/lang-guess/ lang-guess is a fork of language-detection
 
** https://code.google.com/p/lang-guess/ lang-guess is a fork of language-detection
 +
 +
 
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)
 
* Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)
 
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google
 
** doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google
 +
 +
 
* LID http://www.cavar.me/damir/LID/ Python and Scheme (GPL3)
 
* LID http://www.cavar.me/damir/LID/ Python and Scheme (GPL3)
  

Revision as of 02:25, 6 December 2012

A listing of language identification tools. Language identification can mean both identifiying text type (e.g. news vs literature) and language (e.g. English vs Frisian vs Dutch).

Most of these tools require training on a big corpus (see List of resources by language for corpora per language), but many come with some prebuilt language models.

Free Software



  • Compact Language Detector for Javascript https://github.com/jaukia/cld-js (3-clause license)
    • doesn't seem to include a method to add new languages, the existing ones were presumably generated by Google


Proprietary

See also