Python 3-pickle可以处理大于4GB的字节对象吗? [英] Python 3 - Can pickle handle byte objects larger than 4GB?
问题描述
基于此注释和参考文档,Python 3.4+中的Pickle 4.0+应该能够对字节对象进行pickle.大于4 GB.
Based on this comment and the referenced documentation, Pickle 4.0+ from Python 3.4+ should be able to pickle byte objects larger than 4 GB.
但是,在Mac OS X 10.10.4上使用python 3.4.3或python 3.5.0b2时,当我尝试腌制一个大字节数组时出现错误:
However, using python 3.4.3 or python 3.5.0b2 on Mac OS X 10.10.4, I get an error when I try to pickle a large byte array:
>>> import pickle
>>> x = bytearray(8 * 1000 * 1000 * 1000)
>>> fp = open("x.dat", "wb")
>>> pickle.dump(x, fp, protocol = 4)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument
我的代码中是否存在错误或我误解了文档?
Is there a bug in my code or am I misunderstanding the documentation?
推荐答案
以下是 issue 24658 .使用pickle.loads
或pickle.dumps
并将字节对象分成大小为2**31 - 1
的块,以将其放入文件或从文件中取出.
Here is a simple workaround for issue 24658. Use pickle.loads
or pickle.dumps
and break the bytes object into chunks of size 2**31 - 1
to get it in or out of the file.
import pickle
import os.path
file_path = "pkl.pkl"
n_bytes = 2**31
max_bytes = 2**31 - 1
data = bytearray(n_bytes)
## write
bytes_out = pickle.dumps(data)
with open(file_path, 'wb') as f_out:
for idx in range(0, len(bytes_out), max_bytes):
f_out.write(bytes_out[idx:idx+max_bytes])
## read
bytes_in = bytearray(0)
input_size = os.path.getsize(file_path)
with open(file_path, 'rb') as f_in:
for _ in range(0, input_size, max_bytes):
bytes_in += f_in.read(max_bytes)
data2 = pickle.loads(bytes_in)
assert(data == data2)
这篇关于Python 3-pickle可以处理大于4GB的字节对象吗?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!