如何使用Nans将zscore归一化 pandas 列? [英] how to zscore normalize pandas column with nans?
问题描述
我有一个熊猫数据框,其中有一列我想zscore归一化的实数值:
I have a pandas dataframe with a column of real values that I want to zscore normalize:
>> a
array([ nan, 0.0767, 0.4383, 0.7866, 0.8091, 0.1954, 0.6307,
0.6599, 0.1065, 0.0508])
>> df = pandas.DataFrame({"a": a})
问题是单个nan
值使所有数组nan
:
The problem is that a single nan
value makes all the array nan
:
>> from scipy.stats import zscore
>> zscore(df["a"])
array([ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan])
将zscore
(或不是scipy的等效功能)应用于熊猫数据框的列并使其忽略nan
值的正确方法是什么?我希望它与np.nan
的原始列具有相同的尺寸,用于无法归一化的值
What's the correct way to apply zscore
(or an equivalent function not from scipy) to a column of a pandas dataframe and have it ignore the nan
values? I'd like it to be same dimension as original column with np.nan
for values that can't be normalized
编辑:也许最好的解决方案是使用scipy.stats.nanmean
和scipy.stats.nanstd
?我不明白为什么为此目的需要更改std
的自由度:
edit: maybe the best solution is to use scipy.stats.nanmean
and scipy.stats.nanstd
? I don't see why the degrees of freedom need to be changed for std
for this purpose:
zscore = lambda x: (x - scipy.stats.nanmean(x)) / scipy.stats.nanstd(x)
推荐答案
mean
和std
的pandas'
版本将处理Nan
,因此您可以采用这种方式进行计算(获得与scipy zscore我认为您需要在std
上使用ddof = 0:
Well the pandas'
versions of mean
and std
will hand the Nan
so you could just compute that way (to get the same as scipy zscore I think you need to use ddof=0 on std
):
df['zscore'] = (df.a - df.a.mean())/df.a.std(ddof=0)
print df
a zscore
0 NaN NaN
1 0.0767 -1.148329
2 0.4383 0.071478
3 0.7866 1.246419
4 0.8091 1.322320
5 0.1954 -0.747912
6 0.6307 0.720512
7 0.6599 0.819014
8 0.1065 -1.047803
9 0.0508 -1.235699
这篇关于如何使用Nans将zscore归一化 pandas 列?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!