Link Search Menu Expand Document

Language identification

Identifying the language of a text


Language identification tools identify the language of an unknown text. In machine translation, language identification tools are used in building monolingual training data from large collections of unidentified texts such as webpages. Language identification tools are also used to automatically identify the source language in many machine translation systems when the source language is unspecified.

As of 2023, language identification is typically implemented as a text classifier over character n-grams. Alternative approaches include dictionary lookup, word n-grams, statistical models, and rule-based identification.

Challenges

  • Similar languages may be mistaken for one another, for example Dutch and Afrikaans.
  • Language identification is harder on shorter texts because there may not be enough language-specific information. Longer texts have more chances of having language-specific characters and words making it easier to identify.
  • If a particular locale is not supported in the language identification tool, it may be misclassified as another, similar locale. For example, if a language identification tool does not support Serbian with the Cyrillic script, it may be misclassified as Macedonian.

↑

Edit this article β†’

Machine Translate is created and edited by contributors like you!

Learn more about contributing β†’

Licensed under CC-BY-SA-4.0.

Cite this article β†’