Sklearn Pipeline:如何为kmeans构建文本文本? [英] Sklearn Pipeline: How to build for kmeans, clustering text?
问题描述
我的文字如下所示:
list1 = ["My name is xyz", "My name is pqr", "I work in abc"]
以上内容是使用kmeans对文本进行聚类的训练集.
The above will be training set for clustering text using kmeans.
list2 = ["My name is xyz", "I work in abc"]
以上是我的测试集.
我已经建立了矢量化器和模型,如下所示:
I have built a vectorizer and the model as shown below:
vectorizer = TfidfVectorizer(min_df = 0, max_df=0.5, stop_words = "english", charset_error = "ignore", ngram_range = (1,3))
vectorized = vectorizer.fit_transform(list1)
km=KMeans(n_clusters=2, init='k-means++', n_init=10, max_iter=1000, tol=0.0001, precompute_distances=True, verbose=0, random_state=None, copy_x=True, n_jobs=1)
km.fit(vectorized)
如果我尝试为"list2"的测试集预测聚类,则:
If I try to predict the cluster for my test set of "list2":
km.predict(list2)
我收到以下错误:
ValueError: Incorrect number of features. Got 2 features, expected 5
有人告诉我要使用Pipeline
解决此问题.所以我写了下面的代码:
I was told to use Pipeline
to solve this issue. So I wrote the code below:
pipe = Pipeline([('vect', vectorizer), ('vectorized', vectorized), ('kmeans',km )])
但是我得到了错误:
TypeError Traceback (most recent call last)
/mnt/folder/Text_Mining/<ipython-input-14-321cabc3bf35> in <module>()
----> 1 pipe = Pipeline([('vect', vectorizer), ('vectorized', vectorized), ('kmeans',km )])
/usr/local/lib/python2.7/dist-packages/scikit_learn-0.13-py2.7-linux-x86_64.egg/sklearn/pipeline.pyc in __init__(self, steps)
87 raise TypeError("All intermediate steps a the chain should "
88 "be transforms and implement fit and transform"
---> 89 "'%s' (type %s) doesn't)" % (t, type(t)))
90
91 if not hasattr(estimator, "fit"):
TypeError: All intermediate steps a the chain should be transforms and implement fit and transform' (0, 2) 1.0
(1, 4) 0.57735026919
(1, 3) 0.57735026919
(1, 1) 0.57735026919
(2, 0) 1.0' (type <class 'scipy.sparse.csr.csr_matrix'>) doesn't)
我认为也许vectorized
的输出未实现拟合和变换,但是在这种特殊情况下我该怎么做?我是机器学习的新手.
另外,如何从kmeans模型获取标签?运行kmeans时,可以使用km.labels_
访问群集标签.如何在管道中做类似的事情?
I think that maybe the output of vectorized
does not implement a fit and transform, but how do I do that in this particular case? I'm new to Machine Learning.
Also, how to get the labels from the kmeans model? When I run kmeans, I can access the cluster labels by using km.labels_
. How to do something similar in Pipeline?
推荐答案
您需要做的是使用list1
然后使用相同的 vectorizer ,transform
训练vectorizer
list1
和list2
.这样可以解决问题.演示:
What you need to do is to train a vectorizer
using list1
, then with the same vectorizer, transform
list1
and list2
. This will solve the issue. Demo:
>>> list1 = ["My name is xyz", "My name is pqr", "I work in abc"]
>>> list2 = ["My name is xyz", "I work in abc"]
>>> vectorizer = TfidfVectorizer(min_df = 0, max_df=0.5, stop_words = "english", charset_error = "ignore", ngram_range = (1,3))
>>> vec=vectorizer.fit(list1) # train vec using list1
>>> vectorized = vec.transform(list1) # transform list1 using vec
>>> km=KMeans(n_clusters=2, init='k-means++', n_init=10, max_iter=1000, tol=0.0001, precompute_distances=True, verbose=0, random_state=None, cpy_x=True, n_jobs=1)
>>> km.fit(vectorized)
>>> list2Vec=vec.transform(list2) # transform list2 using vec
>>> km.predict(list2Vec)
array([0, 0], dtype=int32)
这篇关于Sklearn Pipeline:如何为kmeans构建文本文本?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!