gzip引发OverflowError:大小不适合未签名的int [英] gzip raised OverflowError: Size does not fit in an unsigned int

问题描述

环境:Windows，Python 3.4.1、64位版本.

Environment: Windows, Python 3.4.1, 64-bit version.

我试图用pickle和gzip保存数据，就像这样:

I tried to save data with pickle and gzip, simply like this:

with gzip.open(filename, 'rb') as f:
    pickle.dump(data,f)

在没有gzip的情况下可以成功转储数据，但是在使用gzip的情况下，异常引发为:

The data can be successfully dumped without gzip, but with gzip, exception raised as:

File "C:\Python34\lib\gzip.py", line 344, in write
  self.fileobj.write( self.compress.compress(data) )
OverflowError: Size does not fit in an unsigned int

我追溯了代码，发现gzip实际上是建立在zlib之上的.在搜寻了这个问题之后，我遇到了此页面 http://bugs.python.org/file32715/zlib_64bit-4.patch .似乎对unsigned int类型的长度施加了限制.

I traced back the code, and found that gzip is actually built upon zlib. And after googling this problem, I came across this page http://bugs.python.org/file32715/zlib_64bit-4.patch. There seems a length limit of unsigned int type is imposed.

所以，我的问题是，有什么办法可以弥补这个错误或通过它吗?

So, my question is, is there any way to make up for this bug or pass around it?

推荐答案

您可以尝试将gzip文件包装在写入器中，该写入器将数据分成给定最大大小的数据块.这是一个草图:

You could try wrapping the gzip file in a writer that splits the data into chunks of a given maximum size. Here is a sketch:

class ChunkedWriter(object):
    def __init__(self, file, chunksize=65536):
        self.file = file
        self.chunksize = chunksize

    def write(self, data):
        mdata = memoryview(data)
        for i in range(0, len(mdata), self.chunksize):
            self.file.write(bytes(mdata[i:i+self.chunksize]))

我不确定这是否能真正解决您的问题，因为我无法在自己的计算机上复制它.

I'm not sure whether this will actually solve your problem since I was unable to reproduce it on my own computer.

这篇关于gzip引发OverflowError:大小不适合未签名的int的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！