使用scikit-learn分为多个类别 [英] Use scikit-learn to classify into multiple categories

问题描述

我正在尝试使用scikit-learn的一种有监督的学习方法，将文本片段分类为一个或多个类别。我尝试过的所有算法的预测函数只会返回一个匹配项。

I'm trying to use one of scikit-learn's supervised learning methods to classify pieces of text into one or more categories. The predict function of all the algorithms I tried just returns one match.

例如，我有一段文字：

"Theaters in New York compared to those in London"

并且我已经训练算法可以为我输入的每个文本片段选择一个地方。

And I have trained the algorithm to pick a place for every text snippet I feed it.

在上面的示例中，我希望它返回纽约和伦敦，但只返回纽约。

In the above example I would want it to return New York and London, but it only returns New York.

是否可以使用scikit-learn返回多个结果？甚至以下一个最高概率返回标签？

Is it possible to use scikit-learn to return multiple results? Or even return the label with the next highest probability?

感谢您的帮助。

-更新

我尝试使用 OneVsRestClassifier ，但每条文本我仍然只能得到一个选择。下面是我正在使用的示例代码

I tried using OneVsRestClassifier but I still only get one option back per piece of text. Below is the sample code I am using

y_train = ('New York','London')


train_set = ("new york nyc big apple", "london uk great britain")
vocab = {'new york' :0,'nyc':1,'big apple':2,'london' : 3, 'uk': 4, 'great britain' : 5}
count = CountVectorizer(analyzer=WordNGramAnalyzer(min_n=1, max_n=2),vocabulary=vocab)
test_set = ('nice day in nyc','london town','hello welcome to the big apple. enjoy it here and london too')

X_vectorized = count.transform(train_set).todense()
smatrix2  = count.transform(test_set).todense()


base_clf = MultinomialNB(alpha=1)

clf = OneVsRestClassifier(base_clf).fit(X_vectorized, y_train)
Y_pred = clf.predict(smatrix2)
print Y_pred

结果：['纽约''伦敦''伦敦']

Result: ['New York' 'London' 'London']

推荐答案

您想要的称为多标签分类。 Scikits-learn可以做到这一点。参见此处： http://scikit-learn.org/dev/modules/multiclass.html 。

What you want is called multi-label classification. Scikits-learn can do that. See here: http://scikit-learn.org/dev/modules/multiclass.html.

我不确定您的示例出了什么问题，我的sklearn版本显然没有WordNGramAnalyzer。也许这是使用更多训练示例或尝试使用其他分类器的问题？尽管请注意，多标签分类器希望目标是元组/标签列表。

I'm not sure what's going wrong in your example, my version of sklearn apparently doesn't have WordNGramAnalyzer. Perhaps it's a question of using more training examples or trying a different classifier? Though note that the multi-label classifier expects the target to be a list of tuples/lists of labels.

以下对我有用：

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier

X_train = np.array(["new york is a hell of a town",
                    "new york was originally dutch",
                    "the big apple is great",
                    "new york is also called the big apple",
                    "nyc is nice",
                    "people abbreviate new york city as nyc",
                    "the capital of great britain is london",
                    "london is in the uk",
                    "london is in england",
                    "london is in great britain",
                    "it rains a lot in london",
                    "london hosts the british museum",
                    "new york is great and so is london",
                    "i like london better than new york"])
y_train = [[0],[0],[0],[0],[0],[0],[1],[1],[1],[1],[1],[1],[0,1],[0,1]]
X_test = np.array(['nice day in nyc',
                   'welcome to london',
                   'hello welcome to new york. enjoy it here and london too'])   
target_names = ['New York', 'London']

classifier = Pipeline([
    ('vectorizer', CountVectorizer(min_n=1,max_n=2)),
    ('tfidf', TfidfTransformer()),
    ('clf', OneVsRestClassifier(LinearSVC()))])
classifier.fit(X_train, y_train)
predicted = classifier.predict(X_test)
for item, labels in zip(X_test, predicted):
    print '%s => %s' % (item, ', '.join(target_names[x] for x in labels))

对我来说，这将产生输出：

For me, this produces the output:

nice day in nyc => New York
welcome to london => London
hello welcome to new york. enjoy it here and london too => New York, London

希望这会有所帮助。

这篇关于使用scikit-learn分为多个类别的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！