如何自动标记所需的内容，算法和建议 [英] How to auto-tag content, algorithms and suggestions needed

问题描述

我正在使用一些非常大型的报纸文章数据库，将它们保存在MySQL数据库中，并且可以查询所有内容.

I am working with some really large databases of newspaper articles, I have them in a MySQL database, and I can query them all.

我现在正在寻找方法来帮助我用一些描述性标签来标记这些文章.

I am now searching for ways to help me tag these articles with somewhat descriptive tags.

所有这些文章都可以通过如下网址访问:

All these articles is accessible from a URL that looks like this:

http://web.site/CATEGORY/this-is-the-title-slug

因此，至少我可以使用类别来确定我们正在使用的内容类型.但是，我也想基于文章文本进行标记.

So at least I can use the category to figure what type of content that we are working with. However, I also want to tag based on the article-text.

我最初的方法是这样做:

My initial approach was doing this:

1. 获取所有文章
2. 获取所有单词，删除所有标点符号，按空格分隔，然后按出现次数进行计数
3. 分析它们，并过滤掉常见的非描述性词，例如他们"，我"，此"，这些"，他们的"等.
4. 过滤掉所有常用词后，剩下的就是具有标签价值的词.

但事实证明这是一项相当手动的任务，而不是一种非常漂亮或有用的方法.

But this turned out to be a rather manual task, and not a very pretty or helpful approach.

这也存在单词或名称被空格分隔的问题，例如，如果1.000篇文章包含名称"John Doe"，而1.000篇文章包含名称"John Hanson"，那么我只会得到该单词是约翰"，不是他的名字和姓氏.

This also suffered from the problem of words or names that are split by space, for example if 1.000 articles contains the name "John Doe", and 1.000 articles contains the name of "John Hanson", I would only get the word "John" out of it, not his first name, and last name.

推荐答案

自动标记文章确实是一个研究问题，当其他人已经完成许多工作时，您可能会花费大量时间重新发明轮子.我建议使用现有的自然语言处理工具包之一，例如 NLTK .

Automatically tagging articles is really a research problem and you can spend a lot of time re-inventing the wheel when others have already done much of the work. I'd advise using one of the existing natural language processing toolkits like NLTK.

一开始，我建议您考虑实施一个合适的Tokeniser(比用空格分割要好得多)，然后研究Chunking和Stemming算法.

To get started, I would suggest looking at implementing a proper Tokeniser (much better than splitting by whitespace), and then take a look at Chunking and Stemming algorithms.

您可能还希望计算 n-gram 的频率，即单词，而不是单个单词.这样可以解决用空格分隔的单词"的问题. NLTK之类的工具包具有为此内置的功能.

You might also want to count frequencies for n-grams, i.e. a sequences of words, instead of individual words. This would take care of "words split by a space". Toolkits like NLTK have functions in-built for this.

最后，随着迭代地改进算法，您可能希望对数据库的随机子集进行训练，然后尝试使用该算法标记其余文章集，以查看其效果如何.

Finally, as you iteratively improve your algorithm, you might want to train on a random subset of the database and then try how the algorithm tags the remaining set of articles to see how well it works.

这篇关于如何自动标记所需的内容，算法和建议的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！