SGDClassifier每次为文本分类提供不同的准确性 [英] SGDClassifier giving different accuracy each time for text classification

问题描述

我正在使用SVM分类器将文本分类为好文本和乱码.我正在使用python的scikit-learn并按如下方式进行操作:

I'm using the SVM Classifier for classifying text as good text and gibberish. I'm using python's scikit-learn and doing it as follows:

'''
Created on May 5, 2017
'''

import re
import random
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import SGDClassifier
from sklearn import metrics

# Prepare data

def prepare_data(data):
    """
    data is expected to be a list of tuples of category and texts.
    Returns a tuple of a list of lables and a list of texts
    """
    random.shuffle(data)
    return zip(*data)

# Format training data

training_data = [
    ("good", "rain a lot the packs maybe damage."),
    ("good", "15107 Lane Pflugerville, TX customer called me and his phone number and my phone numbers were not masked. thank you customer has had a stroke and items were missing from his delivery the cleaning supplies for his wet vacuum steam cleaner.  he needs a call back from customer support "),
    ("gibber", "wh. screen"),
    ("gibber", "How will I know if I"),
    ("good", "I have problems scheduling blocks they are never any available.  Can I do full time?  Can I get scheduled more than one day a month?"),
    ("good", "Suggestion: easier way to sign in due alleviate the tediousness of periodically having to sign back in to the app to check for blocks."),
    ("good", "I am so glad to hear from you. "),
    ("good", "loading on today's itinerary takes ages!!!!!! time consuming when you have 150+ packages to deliver!!!!!"),
    ("good", "due to the new update that makes hours available at 10 pm. if you worked 8 hours that day you can't see next day hours due to 8 hour limit. please fix this"),
    ("good", "omg, PLEASE make it so we don't have to sign in every time we need to go into the app. At least make it good for a week. Thanks."),
    ("good", "Constantly being logged out of app, if we could have a continuous login so we could receive notifications if blocks are available that would be ideal."),
    ("good", "I am having problems  with the App. Every time I exit the App and reopen it asks for my login info."),
    ("good", "15 minute service time due to 33rd floor and  20l lbs of cargo"),
    ("good", "I have been sceduled 1 block in 3 weeks. I check for new block availability multiple times a day and have not seen 1 available in three weeks. is there any way to get more blocks."),
    ("good", "When will delivery jobs be available? Everytime I open this app, it says nothing is available. Have deliveries in Cincinnati started yet?"),
    ("good", "During delivery had to call customer support and after 10 minutes support person couldn't find my pick up location Kirkland /Bellevue and told me to hang up and call different support team.  Support person were unprofessional and rude, which is not acceptable."),
    ("good", "can you please remove the pick up from my phone"),
    ("good", "Dear friends: I'm very very happy it's a big oportunitt"),
    ("good", "THANK YOU so much for the block you assigned me for next week.   If you have an additional 5 blocks please go ahead and assign them to me for next week.  My availability is updated and current.  You guys are awesome!!!"),
    ("good", "after update every time I open app I have too log in! I used to be able to stay logged in unless I logged out, can you return stay logged in option."),
    ("good", "It looks like my app is not installed properly on my android phone, Note 5. I cannot access or do not see the tab to swipe to start delivering and the map or help button that should be visible for me to work today 5/6 at rpm"),
    ("gibber", "AF0000"),
    ("good", "awesome app, awesome hiring process, awesome delivery warehouse , awesome team and help in the field! lets deliver I would like more more more delivers , looking forward to the future ! I just bought a new delivery vehical !"),
    ("good", "I will like to ask why I can't get more delivery's only one in two weeks"),
    ("good", "device too slow software crashing all day"),
    ("good", "it doesn't work sometimes."),
    ("good", "can you please remove the old sprouts pick up from my phone"),
    ("good", "They ability to zoom in on text screens would be very helpful. Am example would be customer notes when viewing in certain lighting conditions can be difficult."),
    ("good", "I missed out on a delivery day when I clicked check in and waited for my turn to get an order only to find out that not only did my check in not register but the gps showed me down the street. I encountered this issue again when one of the warehouse employees placed an order for that location and the app wanted me to drive in a big circle to get back to where I was standing."),
    ("good", "i am a little concerned that i didn't receive any blocks of time for this coming week, even though i had a perfect delivery score from this past pay period. Did the Cincinnati market over hire drivers where there are many people being shut completely out of any delivery blocks for an entire week? i really enjoy this type of work and the app makes it quite convenient."),
    ("good", "I've arrived at the pick up restaurant but the staff did not have the barecode for me to scan, however I pick up the package and deliver but my is still not let me move on"),
    ("good", "might want to check my assigned hours for next week.  5am to 1pm??"),
    ("good", "hi team--just want to give some positive feedback.  I have had nothing but positive feedback from customers. Great support when calling help line. Thank you for this opportunity and if there is ever a situation where you need drivers immediately I will drop what I'm doing and help. You guys are the best."),
    ("good", "Allow days or blocks throughout the day to be modified after General availability is set up for time off like doctors appointments."),
    ("gibber", "AL001234"),
    ("good", "Please, enlight me."),
    ("good", "it only shows my schedule starting in two weeks. when will we be able to start work"),
    ("good", "include more packages for one block, if the packages can be fitted into the car, so driver don't have to come back and pickup every two hours. 25% of the time is wasted coming back for pick up."),
    ("gibber", "BBB h"),
    ("gibber", "AG0003006033SDgCJ12344"),
    ("gibber", "How will I know if I"),
    ("good", "please bring back some sort of hours cap! or possibly stagger the hour drops from 1200 to 1203 so that people with slower internet/slower phone arent at a disadvantage!"),
    ("good", "when the hours released tonight all of the people who didn't have 40 hours could see them.    however the drivers that are capped at 40 were unable to see them due to a flawed system.  please fix the system so that we are not continually treated unfairly like all of the drivers that whined so much and got us in to this mess.  the cap system is unfair to people that want to work and it caused problems with a lack of drivers  to deliver today at the hub.  obviously this is not a good system and benefits no one."),
    ("good", "You have seriously messed up the whole scheduling process. Why can't I get any blocks at 10 even if I wait exactly until 10? Midnight was much better. So now that scheduling is a huge random pain in the ass, why would people want to keep doing this? I haven't been able to schedule work for three days now, it's quite frustrating when I don't get a chance to sign up, even when I'm diligent with timing."),
    ("good", "Seriously, that's all I'm going to get is one lousy day? Tell me again why you need drivers if all we get is one day. I'm not sure this is gonna work out for me. I waited forever to get my background check back and this is what I get? smh"),
    ("good", "doesn't save updated access codes"),
    ("good", "the scheduling of my route is nor done very accurately. it keeps me driving back and forth"),
    ("good", "can't understand how to pick up a block. my availability is wide open. when you guys send the alerts about blocks available I open it real quick and there is nothing there. I do it in a matter of seconds"),
    ("good", "My availability keeps disappearing from my calender.   I set my availability for three weeks in advance. The gray dots are visible  but disappear on Wednesday or Thursday.  This makes it impossible for me to see and choose available blocks for the upcoming week. How can I get it fix.   Mike"),
    ("good", "GPS blank screen"),
    ("gibber", "sea swq"),
    ("gibber", "hiw o"),
    ("gibber", "Dr a"),
    ("gibber", "quick to quick to u uhu wu just us"),
    ("gibber", "Awa what's"),
    ("gibber", "wxdfcs"),
    ("gibber", "7k9opu"),
    ("gibber", "o.m.day day"),
    ("gibber", "GGT part his h"),
    ("gibber", "aawfhg"),
    ("gibber", "seesaw 2s"),
    ("gibber", "wawaa"),
    ("gibber", "of ll"),
    ("gibber", "rewards"),
    ("gibber", "mmqqm5my"),
    ("gibber", ".in w"),
    ("gibber", "play r"),
    ("gibber", "was wwnw www www n"),
    ("gibber", "wqq2fwqq2fz22"),
    ("gibber", "not"),
    ("gibber", "I by yu I"),
    ("gibber", "Hi just wanted to let you know that it's bee"),
    ("gibber", "I erroneously v"),
    ("gibber", "I find it"),
    ("gibber", "bqyyx I a"),
    ("gibber", "are are"),
    ("gibber", "wawi waarnnnkwn"),
    ("gibber", "t Petey ueteu he"),
    ("gibber", "ews ri"),
    ("gibber", "bd xd"),
    ("gibber", "hatpa"),
    ("gibber", "se wests tasgt"),
    ("gibber", "wa vgcx azc Jo of"),
    ("gibber", "2w222"),
    ("gibber", "her u t b"),
    ("gibber", "ddddedc"),
    ("gibber", "just juju in hiking"),
    ("gibber", "wew2ww2wwwew2i2wkkk"),
    ("gibber", "meleeee"),
    ("gibber", "Aaq wqXD"),


]
training_labels, training_texts = prepare_data(training_data)


# Format test data

test_data = [

("gibber", "an quality"),
    ("good", "Can't check in.   Time was 4:06.  I didn't drive out here for no reason."),
    ("good", "can you do view all full address including postal code how it's in old app that helps do correctly delivery and not waist customer time"),
    ("good", "i am available again starting at 10am to 10pm. thanks"),
    ("gibber", "Hello, I encountered"),
    ("good", "I want to know how we are notified if there is a block I have been signed in and haven't been given a block yet"),
    ("gibber", "aawaaw"),
    ("gibber", "eeeeeeeeene"),
    ("good", "I am not getting enough shifts"),
    ("gibber", "hey e75k"),
    ("good", "my screen had went black or inverted"),
    ("good", "maps packed up again in sr20ls"),
    ("good", "how to clear my itinerary from old pickup address ?"),
    ("good", "keep signing me out."),
    ("good", "For alcohol delivery,  where does customer sign?"),
    ("gibber", "t Petey ueteu he"),
    ("good", "can't get blocks.  too many drivers ??"),
    ("good", "got a new phone how do i download to new phone")



]
test_labels, test_texts = prepare_data(test_data)


# Create feature vectors

"""
Convert a collection of text documents to a matrix of token counts.
See: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
"""
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(training_texts)
y = training_labels


# Train the classifier


clf = SGDClassifier()
clf.fit(X, y)


# Test performance

X_test = vectorizer.transform(test_texts)
y_test = test_labels

# Generates a list of labels corresponding to the samples
test_predictions = clf.predict(X_test)

# Convert back to the usual format
annotated_test_data = list(zip(test_predictions, test_texts))
print(annotated_test_data)

# evaluate predictions
y_test = np.array(test_labels)
print(metrics.classification_report(y_test, test_predictions))
print("Accuracy: %0.4f" % metrics.accuracy_score(y_test, test_predictions))

但是，每次运行时，我都会获得不同的准确性.为什么会这样?

But, I keep getting different accuracy each time I run it. Why is this happening?

更新: 因此，我将training_data移到了一个文本文件中，并在上面的代码中像这样读取它:

UPDATE: So I moved the training_data to a text file and I'm reading it in the above code like this:

lines = [line.rstrip('\n') for line in open("file.txt")]
training_data=[]
for i in lines:
    result = i.rstrip(',')
    l = literal_eval(result)
    training_data.append(l)

training_labels, training_texts = prepare_data(training_data)

我也在上面的代码中对此进行了更改:

And I also changed this in my above code:

clf = SGDClassifier(random_state=5000)

因此，现在random_state不为None.但是，每次我的精确度仍然不同！

So, now the random_state is not None. But, I'm still getting different accuracies each time!!

推荐答案

所有对数据进行拆分或混洗的分类器(从h/t到Vivek)在构造函数中均具有可选的random_state变量，其默认值为None.传递random_state时，将由内部 check_random_state 函数.

All classifiers which split or shuffle data (h/t to Vivek) have an optional random_state variable in the constructor with a default value of None. When the random_state is passed, it is checked by an internal check_random_state function.

从文档中:

check_random_state:从 参数random_state.如果random_state是None或np.random，则a 返回随机初始化的RandomState对象.如果random_state 是一个整数，则用于播种新的RandomState对象.如果 random_state是RandomState对象，然后将其传递.

check_random_state: create a np.random.RandomState object from a parameter random_state. If random_state is None or np.random, then a randomly-initialized RandomState object is returned. If random_state is an integer, then it is used to seed a new RandomState object. If random_state is a RandomState object, then it is passed through.

由于使用的是默认值None，因此代码中存在一些不受控制的随机噪声.

Because you're using the default None, you have some uncontrolled stochastic noise in your code.

传递可重复性的种子.

这篇关于SGDClassifier每次为文本分类提供不同的准确性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！