具有统计显着性的 pandas 列相关性 [英] pandas columns correlation with statistical significance
问题描述
给定一个 Pandas 数据框 df,获取其列 df.1
和 df.2
之间的相关性的最佳方法是什么?
What is the best way, given a pandas dataframe, df, to get the correlation between its columns df.1
and df.2
?
我不希望输出使用 NaN
计算行,而 pandas
内置相关性会这样做.但我也希望它输出一个 pvalue
或一个标准错误,而内置的则没有.
I do not want the output to count rows with NaN
, which pandas
built-in correlation does. But I also want it to output a pvalue
or a standard error, which the built-in does not.
SciPy
似乎被 NaN 赶上了,尽管我相信它确实报告了重要性.
SciPy
seems to get caught up by the NaNs, though I believe it does report significance.
数据示例:
1 2
0 2 NaN
1 NaN 1
2 1 2
3 -4 3
4 1.3 1
5 NaN NaN
推荐答案
@Shashank 提供的答案很好.然而,如果你想要一个纯 pandas
的解决方案,你可能会喜欢这个:
Answer provided by @Shashank is nice. However, if you want a solution in pure pandas
, you may like this:
import pandas as pd
from pandas.io.data import DataReader
from datetime import datetime
import scipy.stats as stats
gdp = pd.DataFrame(DataReader("GDP", "fred", start=datetime(1990, 1, 1)))
vix = pd.DataFrame(DataReader("VIXCLS", "fred", start=datetime(1990, 1, 1)))
#Do it with a pandas regression to get the p value from the F-test
df = gdp.merge(vix,left_index=True, right_index=True, how='left')
vix_on_gdp = pd.ols(y=df['VIXCLS'], x=df['GDP'], intercept=True)
print(df['VIXCLS'].corr(df['GDP']), vix_on_gdp.f_stat['p-value'])
结果:
-0.0422917932738 0.851762475093
与 stats 函数的结果相同:
Same results as stats function:
#Do it with stats functions.
df_clean = df.dropna()
stats.pearsonr(df_clean['VIXCLS'], df_clean['GDP'])
结果:
(-0.042291793273791969, 0.85176247509284908)
为了扩展到更多变量,我给你一个丑陋的基于循环的方法:
To extend to more vairables I give you an ugly loop based approach:
#Add a third field
oil = pd.DataFrame(DataReader("DCOILWTICO", "fred", start=datetime(1990, 1, 1)))
df = df.merge(oil,left_index=True, right_index=True, how='left')
#construct two arrays, one of the correlation and the other of the p-vals
rho = df.corr()
pval = np.zeros([df.shape[1],df.shape[1]])
for i in range(df.shape[1]): # rows are the number of rows in the matrix.
for j in range(df.shape[1]):
JonI = pd.ols(y=df.icol(i), x=df.icol(j), intercept=True)
pval[i,j] = JonI.f_stat['p-value']
rho 的结果:
GDP VIXCLS DCOILWTICO
GDP 1.000000 -0.042292 0.870251
VIXCLS -0.042292 1.000000 -0.004612
DCOILWTICO 0.870251 -0.004612 1.000000
pval 的结果:
[[ 0.00000000e+00 8.51762475e-01 1.11022302e-16]
[ 8.51762475e-01 0.00000000e+00 9.83747425e-01]
[ 1.11022302e-16 9.83747425e-01 0.00000000e+00]]
这篇关于具有统计显着性的 pandas 列相关性的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!