sklearn TfidfVectorizer:通过不删除其中的停用词来生成自定义NGram [英] sklearn TfidfVectorizer : Generate Custom NGrams by not removing stopword in them

问题描述

以下是我的代码:

sklearn_tfidf = TfidfVectorizer(ngram_range= (3,3),stop_words=stopwordslist, norm='l2',min_df=0, use_idf=True, smooth_idf=False, sublinear_tf=True)
sklearn_representation = sklearn_tfidf.fit_transform(documents)

它通过删除所有停用词来生成三字组.

It generates tri gram by removing all the stopwords.

我希望它允许那些TRIGRAM中途带有停用词的人(而不是在开头和结尾)

是否需要为此编写处理器. 需要建议.

Is there processor needs to be written for this. Need suggestions.

推荐答案

是的，您需要提供自己的分析器功能，该功能将根据您的要求将文档转换为功能.

Yes, you need to supply your own analyzer function which will convert the documents to the features as per your requirements.

根据文档:

analyzer:字符串，{'word'，'char'，'char_wb'}或可调用

analyzer : string, {‘word’, ‘char’, ‘char_wb’} or callable

....
....
If a callable is passed it is used to extract the sequence of 
features out of the raw, unprocessed input.

在该自定义可调用项中，您需要首先将句子拆分为不同的部分，删除特殊字符(例如逗号，大括号，符号等)，将其转换为小写，然后将其转换为n_grams.

In that custom callable you need to take care of first splitting the sentence into different parts, removing special chars like comma, braces, symbols etc, convert them to lower case, then convert them to n_grams.

默认实现按以下顺序处理单个句子:

The default implementation works on a single sentences in the following order:

1. 解码:根据给定编码的句子(默认为'utf-8')
2. 预处理:将句子转换为小写
3. 标记化:从句子中获取单个单词标记(默认正则表达式选择2个或更多字母数字字符的标记)
4. 停止单词删除:从上述步骤中删除出现在停用词中的单个单词标记
5. 创建N_gram:删除停用词后，其余令牌将按所需的n_gram排列
6. 删除太少或太常见的功能:删除频率大于max_df或小于min_df的单词.

1. Decoding: the sentence according to given encoding (default 'utf-8')
2. Preprocessing: convert the sentence to lower case
3. Tokenizing: get single word tokens from the sentence (The default regexp selects tokens of 2 or more alphanumeric characters)
4. Stop word removal: remove the single word tokens from the above step which are present in stop words
5. N_gram creation: After stop word removal, the remaining tokens are then arranged in the required n_grams
6. Remove too rare or too common features: Remove words which have frequency greater than max_df or lower than min_df.

如果要向TfidfVectorizer中的analyzer参数传递自定义可调用对象，则需要处理所有这一切.

You need to handle all this if you want to pass a custom callable to the analyzer param in the TfidfVectorizer.

OR

您可以扩展TfidfVectorizer类，并且仅覆盖最后2个步骤.像这样:

You can extend the TfidfVectorizer class and only override the last 2 steps. Something like this:

from sklearn.feature_extraction.text import TfidfVectorizer
class NewTfidfVectorizer(TfidfVectorizer):
    def _word_ngrams(self, tokens, stop_words=None):

        # First get tokens without stop words
        tokens = super(TfidfVectorizer, self)._word_ngrams(tokens, None)
        if stop_words is not None:
            new_tokens=[]
            for token in tokens:
                split_words = token.split(' ')

                # Only check the first and last word for stop words
                if split_words[0] not in stop_words and split_words[-1] not in stop_words:
                    new_tokens.append(token)
            return new_tokens

        return tokens

然后，像这样使用它:

vectorizer = NewTfidfVectorizer(stop_words='english', ngram_range=(3,3))
vectorizer.fit(data)

这篇关于sklearn TfidfVectorizer:通过不删除其中的停用词来生成自定义NGram的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！