如何在滚动运算符中访问多列? [英] How to access multi columns in the rolling operator?

问题描述

我想在熊猫中做一些滚动窗口计算，这需要同时处理两列.我将举一个简单的例子来清楚地表达问题:

I want to do some rolling window calculation in pandas which need to deal with two columns at the same time. I'll take an simple instance to express the problem clearly:

import pandas as pd

df = pd.DataFrame({
    'x': [1, 2, 3, 2, 1, 5, 4, 6, 7, 9],
    'y': [4, 3, 4, 6, 5, 9, 1, 3, 1, 2]
})

windowSize = 4
result = []

for i in range(1, len(df)+1):
    if i < windowSize:
        result.append(None)
    else:
        x = df.x.iloc[i-windowSize:i]
        y = df.y.iloc[i-windowSize:i]
        m = y.mean()
        r = sum(x[y > m]) / sum(x[y <= m])
        result.append(r)

print(result)

有没有办法在熊猫中没有for循环来解决问题?感谢您的帮助

Is there any way without for loop in pandas to solve the problem? Any help is appreciated

推荐答案

这是使用NumPy工具的一种矢量化方法-

Here's one vectorized approach using NumPy tools -

windowSize = 4
a = df.values
X = strided_app(a[:,0],windowSize,1)
Y = strided_app(a[:,1],windowSize,1)
M = Y.mean(1)
mask = Y>M[:,None]
sums = np.einsum('ij,ij->i',X,mask)
rest_sums = X.sum(1) - sums
out = sums/rest_sums

strided_app来自 here .

运行时测试-

方法-

# @kazemakase's solution
def rolling_window_sum(df, windowSize=4):
    rw = rolling_window(df.values.T, windowSize)
    m = np.mean(rw[1], axis=-1, keepdims=True)
    a = np.sum(rw[0] * (rw[1] > m), axis=-1)
    b = np.sum(rw[0] * (rw[1] <= m), axis=-1)
    result = a / b
    return result    

# Proposed in this post    
def strided_einsum(df, windowSize=4):
    a = df.values
    X = strided_app(a[:,0],windowSize,1)
    Y = strided_app(a[:,1],windowSize,1)
    M = Y.mean(1)
    mask = Y>M[:,None]
    sums = np.einsum('ij,ij->i',X,mask)
    rest_sums = X.sum(1) - sums
    out = sums/rest_sums
    return out

时间-

In [46]: df = pd.DataFrame(np.random.randint(0,9,(1000000,2)))

In [47]: %timeit rolling_window_sum(df)
10 loops, best of 3: 90.4 ms per loop

In [48]: %timeit strided_einsum(df)
10 loops, best of 3: 62.2 ms per loop

要压缩更多性能，我们可以计算Y.mean(1)部分，该部分基本上是带有

To squeeze in more performance, we can compute the Y.mean(1) part, which is basically a windowed summation with Scipy's 1D uniform filter. Thus, M could be alternatively computed for windowSize=4 as -

from scipy.ndimage.filters import uniform_filter1d as unif1d

M = unif1d(a[:,1].astype(float),windowSize)[2:-1]

性能提升显着-

In [65]: %timeit strided_einsum(df)
10 loops, best of 3: 61.5 ms per loop

In [66]: %timeit strided_einsum_unif_filter(df)
10 loops, best of 3: 49.4 ms per loop

这篇关于如何在滚动运算符中访问多列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！