python内存不足错误的解决方法是什么? [英] What are the workaround options for python out of memory error?

问题描述

我正在将x，y，z点文件(LAS)读入python，并遇到内存错误.我正在为我正在研究的项目在已知点之间插入未知点.我开始处理小文件(<5,000,000点)，并且能够毫无问题地读取/写入numpy数组和python列表.我收到了更多要处理的数据(> 50,000,000点)，现在我的代码因MemoryError而失败.

I am reading a x,y,z point file (LAS) into python and have run into memory errors. I am interpolating unknown points between known points for a project I am working on. I began working with small files (< 5,000,000 points) and was able to read/write to a numpy array and python lists with no problem. I have received more data to work with (> 50,000,000 points) and now my code fails with a MemoryError.

处理如此大量的数据有哪些选择?我不必一次将所有数据加载到内存中，但是我需要使用

What are some options for handling such large amounts of data? I do not have to load all data into memory at once, but I will need to look at neighboring points using scipy kd-tree I am using Python 2.7 32 bit on a 64 bit Windows XP OS.

谢谢.

代码发布在下面.我拿出了用于长计算和变量定义的代码.

Code is posted below. I took out code for long calculations and variable definitions.

from liblas import file
import numpy as np

f = file.File(las_file, mode='r')
num_points = int(f.__len__())
dt = [('x', 'f4'), ('y', 'f4'), ('z', 'f4'), ('i', 'u2'), ('c', 'u1'), ('t', 'datetime64[us]')]
xyzict = np.empty(shape=(num_points,), dtype = dt)
counter = 0
for p in f:
    newrow = (p.x, p.y, p.z, p.intensity, p.classification, p.time)
    xyzict[counter] = newrow    
    counter += 1

dropoutList = []
counter = 0
for i in np.nditer(xyzict):
    # code to define P1x, P1y, P1z, P1t
    if counter != 0:
        # code to calculate n, tDiff, and seconds 
        if n > 1 and n < scanN:
            # code to find v and vD
            for d in range(1, int(n-1)):
                # Code to interpolate x, y, z for points between P0 and P1
                # Append tuple of x, y, and z to dropoutList
                dropoutList.append(vD)
    # code to set x, y, z, t for next iteration
    counter += 1

推荐答案

无论系统中有多少RAM，如果您运行的是32位python，实际的内存限制为2 GB应用. SO上还有许多其他问题可以解决此问题(例如，请参见这里).由于您在ndarray中使用的结构为23个字节，并且您正在读取超过50,000,000个点，因此已经使您的内存大约为1 GB.您尚未包括其余代码，因此不清楚程序其他部分正在消耗多少额外的内存.

Regardless of the amount of RAM in your system, if you are running 32-bit python, you will have a practical limit of about 2 GB of RAM for your application. There are a number of other questions on SO that address this (e.g., see here). Since the structure you are using in your ndarray is 23 bytes and you are reading over 50,000,000 points, that already puts you at about 1 GB. You haven't included the rest of your code so it isn't clear how much additional memory is being consumed by other parts of your program.

如果系统中的RAM超过2 GB，并且您将继续处理大型数据集，则应安装64位python来解决大约2 GB的限制.

If you have well over 2 GB of RAM in your system and you will continue to work on large data sets, you should install 64-bit python to get around this ~ 2 GB limit.

这篇关于python内存不足错误的解决方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！