np.fromfile 文件大小的最大限制? [英] Max limit for file size with np.fromfile?
问题描述
np.fromfile() 是否有最大文件大小?我正在尝试读取一个 48GB 的文件,大约有 2.1e9 条记录(每条记录有 5 个值).
If there a max file size for np.fromfile()? I'm trying to read a 48GB file, with about 2.1e9 records (each record has 5 values).
当我将计数限制为 8e7 时它有效:
It works when I limit the count to 8e7:
fromfile(filename, dtp, int(8e7))
Out[69]:
array([(1, 244025.0, 1.1666666269302368, 360.1666564941406, 50.0),
(2, 244025.0, 1.1666666269302368, 360.5, 50.0),
(3, 244025.0, 1.1666666269302368, 360.8333435058594, 50.0), ...,
(255138, 244131.0, 1128.9346923828125, 461.38494873046875, 49.5),
(255139, 244131.0, 1143.77783203125, 473.2532958984375, 49.5),
(255140, 244131.0, 1150.4803466796875, 464.6799011230469, 49.5)],
dtype=[('ntrac', '<i4'), ('ints', '<f8'), ('x', '<f4'), ('y', '<f4'), ('z', '<f4')])
但是当计数设置为 9e7 时,所有值都被读取为零:
But all values are read as zeros when the count is set to 9e7:
In [70]: fromfile(filename, dtp, int(9e7))
Out[70]:
array([(0, 0.0, 0.0, 0.0, 0.0), (0, 0.0, 0.0, 0.0, 0.0),
(0, 0.0, 0.0, 0.0, 0.0), ..., (0, 0.0, 0.0, 0.0, 0.0),
(0, 0.0, 0.0, 0.0, 0.0), (0, 0.0, 0.0, 0.0, 0.0)],
dtype=[('ntrac', '<i4'), ('ints', '<f8'), ('x', '<f4'), ('y', '<f4'), ('z', '<f4')])
推荐答案
看来确实是 Mac OS X 的 bug.
It looks like it is indeed a bug with Mac OS X.
在 numpy 中看到以下错误:https://github.com/numpy/numpy/issues/2806
See the following bug in numpy: https://github.com/numpy/numpy/issues/2806
我在分支的分支中有一个建议的修复程序,可以帮助某人进行验证,请参阅:https://github.com/sauliusl/numpy/tree/bug_large_save
I have a proposed fix in my fork of the branch, would help for someone to verify it, see: https://github.com/sauliusl/numpy/tree/bug_large_save
这篇关于np.fromfile 文件大小的最大限制?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!