(人类)文档的语言 [英] (human) Language of a document

问题描述

有没有办法(程序、库)大致知道文档是用哪种语言编写的?

Is there a way (a program, a library) to approximately know which language a document is written in?

我有一堆混合语言的文本文档(~500K)要导入支持 i18n 的 CMS(Drupal)..

I have a bunch of text documents (~500K) in mixed languages to import in a i18n enabled CMS (Drupal)..

我不需要完美匹配，只需要一些猜测.

I don't need perfect matches, only some guess.

推荐答案

有一种非常简单的方法可以做到这一点，因为您拥有需要识别的所有不同语言的语料库数据.这称为 n-gram 建模.我认为 Lingua::Identify 确实如此不过，这已经是最好的选择，而不是实现自己的.

There is a pretty easy way to do this, given that you have corpus data in all the different languages you'll need to identify. It's called n-gram modeling. I think Lingua::Identify does this already, though, so that is your best bet rather than implementing your own.

这篇关于(人类)文档的语言的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！