如何在Python数据框中读取大块数据? [英] How to read data in chunks in Python dataframe?

问题描述

我想将文件f逐块读取到数据帧.这是我使用的代码的一部分.

I want to read the file f in chunks to a dataframe. Here is part of a code that I used.

for i in range(0, maxline, chunksize):
df = pandas.read_csv(f,sep=',', nrows=chunksize, skiprows=i)
df.to_sql(member, engine, if_exists='append',index= False, index_label=None, chunksize=chunksize)

我得到了错误:

pandas.io.common.EmptyDataError:没有要从文件中解析的列

pandas.io.common.EmptyDataError: No columns to parse from file

仅当chunksize> = maxline(这是文件f中的总行)时，该代码才有效.但是，就我而言，chunksize< = maxline.

The code works only when the chunksize >= maxline (which is total lines in file f). However, in my case, the chunksize<=maxline.

请告知解决方法.

推荐答案

我认为最好在 concat 与参数，因为需要避免在index中重复:

I think it is better to use the parameter chunksize in read_csv. Also, use concat with the parameter ignore_index, because of the need to avoid duplicates in index:

chunksize = 5
TextFileReader = pd.read_csv(f, chunksize=chunksize)

df = pd.concat(TextFileReader, ignore_index=True)

请参阅pandas docs .

See pandas docs.

这篇关于如何在Python数据框中读取大块数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！