(人类)文档语言 [英] (human) Language of a document

问题描述

是否有一种方法(程序，库)可以大致了解文档所用的语言?

Is there a way (a program, a library) to approximately know which language a document is written in?

我有一堆混合语言的文本文档(〜500K)要导入支持i18n的CMS(Drupal)中.

I have a bunch of text documents (~500K) in mixed languages to import in a i18n enabled CMS (Drupal)..

我不需要完美的搭配，只有一些猜测.

I don't need perfect matches, only some guess.

推荐答案

鉴于您拥有需要识别的所有不同语言的语料库数据，因此有一种非常简单的方法.这就是所谓的n-gram建模.我认为 Lingua :: Identify 确实可以不过，这已经是您最好的选择，而不是自己实施.

There is a pretty easy way to do this, given that you have corpus data in all the different languages you'll need to identify. It's called n-gram modeling. I think Lingua::Identify does this already, though, so that is your best bet rather than implementing your own.

这篇关于(人类)文档语言的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！