Pandas 滚动 std 产生不一致的结果并且与 values.std 不同 [英] Pandas rolling std yields inconsistent results and differs from values.std

问题描述

使用pandas v1.0.1 和numpy 1.18.1，我想计算时间序列上具有不同窗口大小的滚动均值和标准差.在我正在处理的数据中，某些后续点的值可以是恒定的，这样 - 取决于窗口大小 - 滚动平均值可能等于窗口中的所有值，并且相应的 std 预计为 0.

Using pandas v1.0.1 and numpy 1.18.1, I want to calculate the rolling mean and std with different window sizes on a time series. In the data I am working with, the values can be constant for some subsequent points such that - depending on the window size - the rolling mean might be equal to all the values in the window and the corresponding std is expected to be 0.

但是，根据窗口大小，我看到使用相同 df 的不同行为.

However, I see a different behavior using the same df depending on the window size.

MWE:

for window in [3,5]:
    values = [1234.0, 4567.0, 6800.0, 6810.0, 6821.0, 6820.0, 6820.0, 6820.0, 6820.0, 6820.0, 6820.0]
    df = pd.DataFrame(values, columns=['values'])
    df.loc[:, 'mean'] = df.rolling(window, min_periods=1).mean()
    df.loc[:, 'std'] = df.rolling(window, min_periods=1).std(ddof=0)
    print(df.info())
    print(f'window: {window}')
    print(df)
    print('non-rolling result:', df['values'].iloc[len(df.index)-window:].values.std())
    print('')

输出:

window: 3
    values         mean          std
0   1234.0  1234.000000     0.000000
1   4567.0  2900.500000  1666.500000
2   6800.0  4200.333333  2287.053757
3   6810.0  6059.000000  1055.011216
4   6821.0  6810.333333     8.576454
5   6820.0  6817.000000     4.966555
6   6820.0  6820.333333     0.471405
7   6820.0  6820.000000     0.000000
8   6820.0  6820.000000     0.000000
9   6820.0  6820.000000     0.000000
10  6820.0  6820.000000     0.000000
non-rolling result: 0.0

window: 5
    values         mean          std
0   1234.0  1234.000000     0.000000
1   4567.0  2900.500000  1666.500000
2   6800.0  4200.333333  2287.053757
3   6810.0  4852.750000  2280.329732
4   6821.0  5246.400000  2186.267193
5   6820.0  6363.600000   898.332366
6   6820.0  6814.200000     8.158431
7   6820.0  6818.200000     4.118252
8   6820.0  6820.200000     0.400000
9   6820.0  6820.000000     0.000021
10  6820.0  6820.000000     0.000021
non-rolling result: 0.0

正如预期的那样，对于 idx 7,8,9,10 使用 3 的窗口大小，std 是 0.对于窗口大小 5，我希望 idx 9 和 10 产生 0.但是，结果是不同的从 0.

As expected, the std is 0 for idx 7,8,9,10 using a window size of 3. For a window size of 5, I would expect idx 9 and 10 to yield 0. However, the result is different from 0.

如果我手动"计算每个窗口大小的最后一个窗口的 std(分别使用 idxs 8,9,10 和 6,7,8,9,10)，我得到的结果都是 0案例.

If I calculate the std 'manually' for the last window of each window size (using idxs 8,9,10 and 6,7,8,9,10, respectively), I get the expected result of 0 for both cases.

有人知道这里可能是什么问题吗?任何数字警告?

Does anybody have an idea what could be the issue here? Any numerical caveats?

推荐答案

pd.rolling 中 std() 的实现似乎更喜欢高性能而不是数值精度.但是您可以应用 np 版本的标准偏差:

It seems that implementation of std() in pd.rolling prefers high performance over numerical accuracy. However You can apply np version of standard deviation:

df.loc[:, 'std'] = df.rolling(window, min_periods=1).apply(np.std)

结果:

    values          std
0   1234.0     0.000000
1   4567.0  1666.500000
2   6800.0  2287.053757
3   6810.0  2280.329732
4   6821.0  2186.267193
5   6820.0   898.332366
6   6820.0     8.158431
7   6820.0     4.118252
8   6820.0     0.400000
9   6820.0     0.000000
10  6820.0     0.000000

现在精度更好了.

这篇关于Pandas 滚动 std 产生不一致的结果并且与 values.std 不同的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！