将二进制值的文本转换为numpy文件 [英] convert a text of binary values to numpy file

问题描述

如何将包含二进制值字符(0和1)的巨大文本文件(> 16G)转换为numpy数组文件而又不消耗python中的内存?假设我们在计算机上有足够的存储空间，但没有足够的RAM用于转换.

How can one convert a huge text file (>16G) containing binary-valued characters (0 and 1) to a numpy array file without blowing up the memory in python? Assuming we have enough storage on the machine but not enough RAM for the conversion.

样本数据:

0,0,0,0,0,1,0,0,0 
1,0,0,1,0,0,0,0,0
...

示例代码:

converted_data = [ map(int,line.split(',')) for line in f ]

推荐答案

您使用pickle创建了许多bin文件，并且有一些代码可以加载和卸载数据的不同部分.

You create many bin-files with pickle and you have some code that loads and unloads the different part of your data.

假设您有一个16GB的文件，则可以创建16个1GB的pickle文件.

Say you have a file that is 16GB, you can create 16 1GB pickle files.

如果您说有足够的RAM，则在完成pickle文件之后，应该可以将其全部加载到内存中.

If you say you have enough RAM, after the pickle files are done, you should be able to load it all in memory.

这篇关于将二进制值的文本转换为numpy文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！