确定文字是否为英文? [英] Determine if text is in English?

问题描述

我同时使用 Nltk 和

I am using both Nltk and Scikit Learn to do some text processing. However, within my list of documents I have some documents that are not in English. For example, the following could be true:

[ "this is some text written in English", 
  "this is some more text written in English", 
  "Ce n'est pas en anglais" ] 

出于分析目的，我希望将所有非英语的句子作为预处理的一部分删除.但是，有什么好方法吗?我一直在使用Google搜索，但是找不到任何可以让我识别字符串是否为英文的特定内容.这是Nltk或Scikit learn中未提供的功能吗? 编辑我见过像这个，但两者都是针对单个单词的...不是文档".我是否必须遍历句子中的每个单词以检查整个句子是否为英语?

For the purposes of my analysis, I want all sentences that are not in English to be removed as part of pre-processing. However, is there a good way to do this? I have been Googling, but cannot find anything specific that will let me recognize if strings are in English or not. Is this something that is not offered as functionality in either Nltk or Scikit learn? EDIT I've seen questions both like this and this but both are for individual words... Not a "document". Would I have to loop through every word in a sentence to check if the whole sentence is in English?

我使用的是Python，因此使用Python的库会更可取，但是我可以根据需要切换语言，只是认为Python会是最好的选择.

I'm using Python, so libraries that are in Python would be preferable, but I can switch languages if needed, just thought that Python would be the best for this.

推荐答案

有一个名为langdetect的库.它是从可从以下位置获得的Google语言检测功能移植而来的:

There is a library called langdetect. It is ported from Google's language-detection available here:

https://pypi.python.org/pypi/langdetect

开箱即用，它支持55种语言.

It supports 55 languages out of the box.

这篇关于确定文字是否为英文?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！