NameError: 名称 'tokenize' 未定义 [英] NameError: name 'tokenize' is not defined

问题描述

到情感分析分类:我们训练了各种机器学习模型和深度神经网络，将评论文本的情感分为 3 个主要类别(正面、中性、负面).

To to Sentiment Analysis Classification : We trained various machine learning models and deep neural networks to classify the sentiment of review text into 3 main categories (Positive, Neutral, Negative).

# TfidfVectorizer transform train and test
    con_vec = TfidfVectorizer(stop_words='english',tokenizer=tokenize,max_features=20000,ngram_range=(1,2))
    X_train_tfidf = con_vec.fit_transform(X_train)
    # with open("tfidf_vectorizer.pkl", 'wb') as handle:
    #                     pickle.dump(con_vec, handle)
    y_train_tfidf = y_train
    X_test_tfidf = con_vec.transform(X_test)
    y_test_tfidf = y_test

**ERROR CODE :** 
    **Pandas Version is pandas 1.1.4**
   ---------------------------------------------------------------------------

      1 from nltk import word_tokenize # Generate table of words with their counts
      2 from nltk.tokenize import word_tokenize # TfidfVectorizer transform train and test
----> 3 con_vec = TfidfVectorizer(stop_words='english',tokenizer=tokenize,max_features=20000,ngram_range=(1,2))
      4 X_train_tfidf = con_vec.fit_transform(X_train)
    
  
    **NameError: name 'tokenize' is not defined**

Any solution how to remove this error?
Also tried from **nltk import word_tokenize** but still same error 
    
      [1]: https://i.stack.imgur.com/lmoSp.png

推荐答案

尝试导入:

from nltk.tokenize import tokenize

这篇关于NameError: 名称 'tokenize' 未定义的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！