凌乱的散点图回归线:Python [英] messy scatter plot regression line: Python

问题描述

在python 2.7.6、matlablib、scikit learn 0.17.0中，当我在散点图上做多项式回归线时，多项式曲线会像这样:

In python 2.7.6, matlablib, scikit learn 0.17.0, When I make a polynomial regression lines on a scatter plot, the polynomial curve will be really messy like this:

脚本是这样的:读取两列浮动数据，做散点图和回归

The script is like this: it will read two columns of floating data and make a scatter plot and regression

import pandas as pd
import scipy.stats as stats
import pylab 
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
import pylab as pl
import sklearn
from sklearn import preprocessing
from sklearn.cross_validation import train_test_split
from sklearn import datasets, linear_model
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import Ridge

df=pd.read_csv("boston_real_estate_market_clean.csv")

LSTAT = df['LSTAT'].as_matrix()

LSTAT=LSTAT.reshape(LSTAT.shape[0], 1)

MEDV=df['MEDV'].as_matrix()

MEDV=MEDV.reshape(MEDV.shape[0], 1)

# Train test set split
X_train1, X_test1, y_train1, y_test1 =                train_test_split(LSTAT,MEDV,test_size=0.3,random_state=1)

# Ploynomial Regression-nst order

plt.scatter(X_test1, y_test1, s=10, alpha=0.3)

for degree in [1,2,3,4,5]:
    model = make_pipeline(PolynomialFeatures(degree), Ridge())
    model.fit(X_train1,y_train1)
    y_plot = model.predict(X_test1)
    plt.plot(X_test1, y_plot, label="degree %d" % degree
             +'; $q^2$: %.2f' % model.score(X_train1, y_train1)
             +'; $R^2$: %.2f' % model.score(X_test1, y_test1))


plt.legend(loc='upper right')

plt.show()

我猜原因是因为X_test1，y_plot"没有正确排序?

I guess the reason is because the "X_test1, y_plot" are not sorted properly?

X_test1 是一个像这样的 numpy 数组:

X_test1 is a numpy array like this:

[[  5.49]
 [ 16.65]
 [ 17.09]
 ....
 [ 25.68]
 [ 24.39]]

yplot 是一个像这样的 numpy 数组:

yplot is a numpy array like this:

[[ 29.78517812]
 [ 17.16759833]
 [ 16.86462359]
 [ 23.18680265]
...[ 37.7631725 ]]

我尝试与此排序:

 [X_test1, y_plot] = zip(*sorted(zip(X_test1, y_plot), key=lambda y_plot: y_plot[0]))

     plt.plot(X_test1, y_plot, label="degree %d" % degree
              +'; $q^2$: %.2f' % model.score(X_train1, y_train1)
              +'; $R^2$: %.2f' % model.score(X_test1, y_test1))

曲线现在看起来很正常，但是结果很奇怪，R ^ 2为负.

The curve looks normal now but the result is weird with a negative R^2.

任何大师都可以告诉我真正的问题是或如何正确排序吗?谢谢！

Could any guru show me the real issue is or how to sort here properly? Thank you!

推荐答案

虽然情节现在是正确的，但您在排序时弄乱了 X_test1 与 y_test1 的配对，因为您忘记以同样的方式对 y_test1 进行排序.最好的解决方案是在拆分后立即排序.然后，稍后计算的y_plot将自动正确:(此处未经测试的示例使用numpy作为np)

While the plot is now correct, you messed up the pairing of X_test1 to y_test1 while sorting because you forgot to also sort y_test1 in the same way. The best solution is to sort right after the split. Then y_plot, which is computed later, will be automatically correct: (Here untested example using numpy as np)

X_train1, X_test1, y_train1, y_test1 =             train_test_split(LSTAT,MEDV,test_size=0.3,random_state=1)

sorted_index = np.argsort(X_test1)
X_test1 = X_test1[sorted_index]
y_test1 = y_test1[sorted_index]

这篇关于凌乱的散点图回归线:Python的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！