使用由列变量确定的块大小加载 Pandas 数据框 [英] Load pandas dataframe with chunksize determined by column variable

问题描述

如果我有一个 csv 文件太大而无法使用 Pandas(在本例中为 35gb)加载到内存中，我知道可以使用 chunksize 分块处理文件.

If I have a csv file that's too large to load into memory with pandas (in this case 35gb), I know it's possible to process the file in chunks, with chunksize.

但是我想知道是否可以根据列中的值更改块大小.

However I want to know if it's possible to change chunksize based on values in a column.

我有一个 ID 列，然后每个 ID 有几行包含信息，如下所示:

I have an ID column, and then several rows for each ID with information, like this:

ID,   Time,  x, y
sasd, 10:12, 1, 3
sasd, 10:14, 1, 4
sasd, 10:32, 1, 2
cgfb, 10:02, 1, 6
cgfb, 10:13, 1, 3
aenr, 11:54, 2, 5
tory, 10:27, 1, 3
tory, 10:48, 3, 5
ect...

我不想将 ID 分成不同的块.例如，将处理大小为 4 的块:

I don't want to separate IDs into different chunks. for example chunks of size 4 would be processed:

ID,   Time,  x, y
sasd, 10:12, 1, 3
sasd, 10:14, 1, 4
sasd, 10:32, 1, 2
cgfb, 10:02, 1, 6
cgfb, 10:13, 1, 3 <--this extra line is included in the 4 chunk

ID,   Time,  x, y
aenr, 11:54, 2, 5
tory, 10:27, 1, 3
tory, 10:48, 3, 5
...

有可能吗?

如果不是，可能使用带有 for 循环的 csv 库:

If not perhaps using the csv library with a for loop along the lines of:

for line in file:
    x += 1
    if x > 1000000 and curid != line[0]:
        break
    curid = line[0]
    #code to append line to a dataframe

虽然我知道这只会创建一个块，而且 for 循环需要很长时间来处理.

although I know this would only create one chunk, and for loops take a long time to process.

推荐答案

如果您逐行遍历 csv 文件，您可以使用依赖于任何列的生成器yield 块.

If you iterate through the csv file line by line, you can yield chunks with a generator dependent on any column.

工作示例:

import pandas as pd

def iter_chunk_by_id(file):
    csv_reader = pd.read_csv(file, iterator=True, chunksize=1, header=None)
    first_chunk = csv_reader.get_chunk()
    id = first_chunk.iloc[0,0]
    chunk = pd.DataFrame(first_chunk)
    for l in csv_reader:
        if id == l.iloc[0,0]:
            id = l.iloc[0,0]
            chunk = chunk.append(l)
            continue
        id = l.iloc[0,0]
        yield chunk
        chunk = pd.DataFrame(l)
    yield chunk

## data.csv ##
# 1, foo, bla
# 1, off, aff
# 2, roo, laa
# 3, asd, fds
# 3, qwe, tre
# 3, tre, yxc   

chunk_iter = iter_chunk_by_id("data.csv")

for chunk in chunk_iter:
    print(chunk)
    print("_____")

输出:

   0     1     2
0  1   foo   bla
1  1   off   aff
_____
   0     1     2
2  2   roo   laa
3  2   jkl   xds
_____
   0     1     2
4  3   asd   fds
5  3   qwe   tre
6  3   tre   yxc
_____

这篇关于使用由列变量确定的块大小加载 Pandas 数据框的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！