Pandas 数据框 to_csv - 拆分为多个输出文件 [英] Pandas dataframe to_csv - split into multiple output files

问题描述

将非常大的数据帧 (50GB) 拆分为多个输出(水平)的最佳/最简单的方法是什么?

What is the best /easiest way to split a very large data frame (50GB) into multiple outputs (horizontally)?

我想过做这样的事情:

stepsize = int(1e8)
for id, i in enumerate(range(0,df.size,stepsize)): 
    start = i 
    end = i + stepsize-1 #neglect last row ...
    df.ix[start:end].to_csv('/data/bs_'+str(id)+'.csv.out')

但我敢打赌有更聪明的解决方案吗?

But I bet there is a smarter solution out there?

如 jakevdp 所述，HDF5 是存储大量数值数据的更好方法，但它不符合我的业务需求.

As noted by jakevdp, HDF5 is a better way to store huge amounts of numerical data, however it doesn't meet my business requirements.

推荐答案

在文件名中使用 id 否则将不起作用.您错过了 id，如果没有 id，则会出现错误.

Use id in the filename else it will not work. You missed id, and without id, it gives an error.

for id, df_i in  enumerate(np.array_split(df, number_of_chunks)):
    df_i.to_csv('/data/bs_{id}.csv'.format(id=id))

这篇关于Pandas 数据框 to_csv - 拆分为多个输出文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！