IOError:[Errno 22]读/写大字节串时参数无效 [英] IOError: [Errno 22] Invalid argument when reading/writing large bytestring

问题描述

我要

IOError: [Errno 22] Invalid argument

当我尝试使用f.write()向磁盘写入大字节串时，其中f是用模式wb打开的.

when I try to write a large bytestring to disk with f.write(), where f was opened with mode wb.

我已经看到很多人在使用Windows网络驱动器时出现此错误，但是我使用的是OSX(最初问这个问题时是10.7，现在是10.8，使用标准的HFS +本地文件系统).我正在使用Python 3.2.2(发生在python.org二进制文件和自制安装程序上).我在系统Python 2.7.2中看不到此问题.

I've seen lots of people online getting this error when using a Windows network drive, but I'm on OSX (10.7 when I originally asked the question but 10.8 now, with a standard HFS+ local filesystem). I'm using Python 3.2.2 (happens on both a python.org binary and a homebrew install). I don't see this problem with the system Python 2.7.2.

我还根据此Windows错误解决方法尝试了模式w+b ，但这当然没有帮助.

I also tried mode w+b based on this Windows bug workaround, but of course that didn't help.

数据来自大型numpy数组(将近4GB的浮点数).如果我手动循环遍历字符串并将其分块写出，则效果很好.但是因为我不能一口气写出来，所以np.save和np.savez失败了-因为它们只使用f.write(ary.tostring()).当我尝试使用h5py将其保存到现有的HDF5文件中时，出现类似的错误.

The data is coming from a large numpy array (almost 4GB of floats). It works fine if I manually loop over the string and write it out in chunks. But because I can't write it all in one pass, np.save and np.savez fail -- since they just use f.write(ary.tostring()). I get a similar error when I try to save it into an existing HDF5 file with h5py.

请注意，读取用file(filename, 'rb')打开的文件时会遇到相同的问题:f.read()给出此IOError，而f.read(chunk_size)表示合理的chunk_size起作用.

Note that I get the same problem when reading a file opened with file(filename, 'rb'): f.read() gives this IOError, while f.read(chunk_size) for reasonable chunk_size works.

有什么想法吗?

推荐答案

这似乎是带有fread/fwrite的一般OSX错误，因此Python用户无法真正修复.参见 numpy#3858 ，此问题/答案，....

This appears to be a general OSX bug with fread / fwrite and so isn't really fixable by a Python user. See numpy #3858, this torch7 commit, this SO question/answer, ....

据说它已经在Mavericks中修复了，但是我仍然看到了这个问题.

Supposedly it's been fixed in Mavericks, but I'm still seeing the issue.

Python 2可能已经解决了这个问题，或者它的io模块可能一直缓冲着较大的读取/写入操作；我还没有彻底调查.

Python 2 may have worked around this or its io module may have always buffered large reads/writes; I haven't investigated thoroughly.

这篇关于IOError:[Errno 22]读/写大字节串时参数无效的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！