如何将 sklearn CountVectorizer 与“word"和“char"分析器一起使用?- Python [英] How do I use sklearn CountVectorizer with both 'word' and 'char' analyzer? - python

问题描述

如何将 sklearn CountVectorizer 与word"和char"分析器一起使用?http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html

我可以分别按单词或字符提取文本特征，但如何创建charword_vectorizer?有没有办法组合矢量化器?或使用多个分析器?

<预><代码>>>>从 sklearn.feature_extraction.text 导入 CountVectorizer>>>word_vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 2), min_df=1)>>>char_vectorizer = CountVectorizer(analyzer='char', ngram_range=(1, 2), min_df=1)>>>x = ['这是一个 foo bar', '你是一个 foo bar 黑羊']>>>word_vectorizer.fit_transform(x)<2x15 稀疏矩阵的类型 '<type 'numpy.int64'>'以压缩稀疏列格式存储 18 个元素>>>>char_vectorizer.fit_transform(x)<2x47 类型的稀疏矩阵 '<type 'numpy.int64'>'以压缩稀疏列格式存储 64 个元素>>>>char_vectorizer.get_feature_names()[u' ', u' a', u' b', u' f', u' i', u' s', u'a', u'a', u'ac', u'ar',u'b', u'ba', u'bl', u'c', u'ck', u'e', u'e ', u'ee', u'ep', u'f',u'fo', u'h', u'he', u'hi', u'i', u'is', u'k', u'k', u'l', u'la',u'o', u'o ', u'oo', u'ou', u'p', u'r', u'r ', u're', u's', u's ', u'sh', u't', u'th', u'u', u'u ', u'y', u'yo']>>>word_vectorizer.get_feature_names()[u'are', u'are foo', u'bar', u'bar black', u'black', u'black山羊', u'foo', u'foo bar', u'is',你是傻瓜，你是绵羊，你是这个，你是这个，你是，你是’]

解决方案

您可以将可调用对象作为 analyzer 参数传递以完全控制标记化，例如

<预><代码>>>>从 pprint 导入 pprint>>>进口重新>>>x = ['这是一个 foo bar', '你是一个 foo bar 黑羊']>>>def words_and_char_bigrams(text):... words = re.findall(r'w{3,}', text)... 对于 w 的话:... 产量 w...对于范围内的 i(len(w) - 2):...产生 w[i:i+2]...>>>v = CountVectorizer(分析器=words_and_char_bigrams)>>>pprint(v.fit(x).vocabulary_){'ac':0，'ar': 1,'是':2，'巴': 3,'酒吧':4，'bl': 5,'黑色':6，'ee': 7,'福':8，'富':9，'他':10，'你好':11，'拉': 12,'sh': 13,'羊':14，'th': 15,'这个':16，'哟':17，你":18}

How do I use sklearn CountVectorizer with both 'word' and 'char' analyzer? http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html

I could extract the text features by word or char separately but how do i create a charword_vectorizer? Is there a way to combine the vectorizers? or use more than one analyzer?

>>> from sklearn.feature_extraction.text import CountVectorizer
>>> word_vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 2), min_df=1)
>>> char_vectorizer = CountVectorizer(analyzer='char', ngram_range=(1, 2), min_df=1)
>>> x = ['this is a foo bar', 'you are a foo bar black sheep']
>>> word_vectorizer.fit_transform(x)
<2x15 sparse matrix of type '<type 'numpy.int64'>'
    with 18 stored elements in Compressed Sparse Column format>
>>> char_vectorizer.fit_transform(x)
<2x47 sparse matrix of type '<type 'numpy.int64'>'
    with 64 stored elements in Compressed Sparse Column format>
>>> char_vectorizer.get_feature_names()
[u' ', u' a', u' b', u' f', u' i', u' s', u'a', u'a ', u'ac', u'ar', u'b', u'ba', u'bl', u'c', u'ck', u'e', u'e ', u'ee', u'ep', u'f', u'fo', u'h', u'he', u'hi', u'i', u'is', u'k', u'k ', u'l', u'la', u'o', u'o ', u'oo', u'ou', u'p', u'r', u'r ', u're', u's', u's ', u'sh', u't', u'th', u'u', u'u ', u'y', u'yo']
>>> word_vectorizer.get_feature_names()
[u'are', u'are foo', u'bar', u'bar black', u'black', u'black sheep', u'foo', u'foo bar', u'is', u'is foo', u'sheep', u'this', u'this is', u'you', u'you are']

解决方案

You can pass a callable as the analyzer argument to get full control over the tokenization, e.g.

>>> from pprint import pprint
>>> import re
>>> x = ['this is a foo bar', 'you are a foo bar black sheep']
>>> def words_and_char_bigrams(text):
...     words = re.findall(r'w{3,}', text)
...     for w in words:
...         yield w
...         for i in range(len(w) - 2):
...             yield w[i:i+2]
...             
>>> v = CountVectorizer(analyzer=words_and_char_bigrams)
>>> pprint(v.fit(x).vocabulary_)
{'ac': 0,
 'ar': 1,
 'are': 2,
 'ba': 3,
 'bar': 4,
 'bl': 5,
 'black': 6,
 'ee': 7,
 'fo': 8,
 'foo': 9,
 'he': 10,
 'hi': 11,
 'la': 12,
 'sh': 13,
 'sheep': 14,
 'th': 15,
 'this': 16,
 'yo': 17,
 'you': 18}

这篇关于如何将 sklearn CountVectorizer 与“word"和“char"分析器一起使用?- Python的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！