通过递增/递减最后找到的值来向前/向后填充na? [英] Forward/Backward fill na by incrementing/decrementing last found value?
问题描述
给出以下熊猫数据框(可以在此处找到它的副本).如何用递增/递减行数nr填充单独的列中的na，直到下一个信号值和前/后信号值? 信号值只有:1； -1或np.na
Given the following pandas data frame (a copy of it can by found here). How to fill na in a separate column with incrementing/decrementing nr of rows until the next signal value and the forward/backward signal value? The signal value is only: 1; -1 or np.na
+----+---------+--------+
| | Values | Signal |
+----+---------+--------+
| 0 | 1420.49 | |
| 1 | 1421.12 | |
| 2 | 1418.95 | |
| 3 | 1419.04 | 1 |
| 4 | 1419.04 | |
| 5 | 1417.51 | |
| 6 | 1416.97 | |
| 7 | 1413.21 | -1 |
| 8 | 1411.49 | |
| 9 | 1412.57 | |
| 10 | 1408.55 | 1 |
| 11 | 1409.16 | |
| 12 | 1413.38 | |
| 13 | 1413.38 | 1 |
| 14 | 1402.35 | |
| 15 | 1397.8 | |
| 16 | 1398.36 | |
| 17 | 1397.62 | |
| 18 | 1394.58 | -1 |
| 19 | 1399.05 | |
| 20 | 1399.9 | |
| 21 | 1398.96 | -1 |
| 22 | 1398.96 | |
| 23 | 1393.69 | |
| 24 | 1398.13 | |
| 25 | 1398.66 | |
| 26 | 1398.02 | 1 |
| 27 | 1397.97 | |
| 28 | 1396.05 | |
| 29 | 1398.13 | |
+----+---------+--------+
最后的结果应该是这样的(此处是它的副本):
The result should be something like this in the end (here is a copy of it):
+----+---------+--------+------------------------+----------------------+-----------------+
| | Values | Signal | forward signal rows nr | backward signal rows | value at signal |
+----+---------+--------+------------------------+----------------------+-----------------+
| 0 | 1420.49 | | | | |
| 1 | 1421.12 | | | | |
| 2 | 1418.95 | | | | |
| 3 | 1419.04 | 1 | 1 | 4 | 1416.97 |
| 4 | 1419.04 | | 2 | 3 | 1416.97 |
| 5 | 1417.51 | | 3 | 2 | 1416.97 |
| 6 | 1416.97 | | 4 | 1 | 1416.97 |
| 7 | 1413.21 | -1 | -1 | -3 | 1412.57 |
| 8 | 1411.49 | | -2 | -2 | 1412.57 |
| 9 | 1412.57 | | -3 | -1 | 1412.57 |
| 10 | 1408.55 | 1 | 1 | 3 | 1413.38 |
| 11 | 1409.16 | | 2 | 2 | 1413.38 |
| 12 | 1413.38 | | 3 | 1 | 1413.38 |
| 13 | 1413.38 | 1 | 1 | 5 | 1397.62 |
| 14 | 1402.35 | | 2 | 4 | 1397.62 |
| 15 | 1397.8 | | 3 | 3 | 1397.62 |
| 16 | 1398.36 | | 4 | 2 | 1397.62 |
| 17 | 1397.62 | | 5 | 1 | 1397.62 |
| 18 | 1394.58 | -1 | -1 | -3 | 1399.9 |
| 19 | 1399.05 | | -2 | -2 | 1399.9 |
| 20 | 1399.9 | | -3 | -1 | 1399.9 |
| 21 | 1398.96 | -1 | -1 | -5 | 1398.66 |
| 22 | 1398.96 | | -2 | -4 | 1398.66 |
| 23 | 1393.69 | | -3 | -3 | 1398.66 |
| 24 | 1398.13 | | -4 | -2 | 1398.66 |
| 25 | 1398.66 | | -5 | -1 | 1398.66 |
| 26 | 1398.02 | 1 | 1 | 4 | 1398.13 |
| 27 | 1397.97 | | 2 | 3 | 1398.13 |
| 28 | 1396.05 | | 3 | 2 | 1398.13 |
| 29 | 1398.13 | | 4 | 1 | 1398.13 |
+----+---------+--------+------------------------+----------------------+-----------------+
我通过几个嵌套循环获得了最终结果，但是问题是它们在几百万行的较大数据帧上效率很低.
I achieved the final result with a few nested loops but the problem is that they are very inefficient on larger data frames of a few million rows.
推荐答案
基于信号的分组的常用方法(IMHO应该对此提供更好的本机支持)使用compare-cumsum-groupby模式.这里的比较是确定信号条目是否为空，之后我们进行累加和，以便每个信号组都有自己的ID(组ID或GID).剩下的只是算术.
The usual approach to signal-based groupings (which we should really have better native support for, IMHO) to use the compare-cumsum-groupby pattern. Here the comparison is to determine whether a signal entry is null or not, after which we do a cumulative sum so that each signal group has its own id (group id, or gid). The rest is just arithmetic.
虽然这里有些重复，我们可以重构，但我感到很懒，所以:
While there's some duplication here we could refactor away, I'm feeling lazy, and so:
gid = df["Signal"].notnull().cumsum()
dg = df.groupby(gid)
sign = dg["Signal"].transform("first")
df["forward signal rows"] = (dg.cumcount() + 1) * sign
df["backward signal rows"] = (dg["Signal"].transform("size") - dg.cumcount()) * sign
df["value at signal"] = dg["Values"].transform("last")
df.loc[gid == 0, "value at signal"] = np.nan
为我提供了一个与您的目标相匹配的框架.
gives me a frame matching your target one.
这篇关于通过递增/递减最后找到的值来向前/向后填充na?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
