for循环内的pickle上的内存泄漏强制出现内存错误 [英] Memory leak on pickle inside a for loop forcing a memory error

问题描述

我有大量用python pickler腌制的数组对象.

I have huge array objects that are pickled with the python pickler.

我试图解开它们，并在for循环中读取数据. 每次阅读和评估完之后，我都会删除对这些对象的所有引用.

I am trying to unpickle them and reading out the data in a for loop. Every time I am done reading and assesing, I delete all the references to those objects.

删除后，我什至同时调用gc.collect()和time.sleep()来查看堆内存是否减少.

After deletion, I even call gc.collect() along with time.sleep() to see if the heap memory reduces.

堆内存并没有减少指向以下事实的事实，即仍然在pickle加载中的某处引用了数据. 15个数据文件(我要处理250+个文件，每个文件1.6GB)后，出现内存错误.

The heap memory doesn't reduce pointing to the fact that, the data is still referenced somewhere within the pickle loading. After 15 datafiles(I got 250+ files to process, 1.6GB each) I hit the memory error.

我在这里看到了许多其他问题，指出了应该解决的内存泄漏问题. 我不了解我的情况到底是怎么回事.

I have seen many other questions here, pointing out a memory leak issue which was supposedly solved. I don't understand what is exactly happening in my case.

推荐答案

在进程运行之前，Python内存管理不会向OS释放内存.

Python memory management does not free memory to OS till the process is running.

运行带有子进程的for循环来调用脚本帮助我解决了该问题. 感谢您的反馈.

Running the for loop with a subprocess to call the script helped me solved the issue. Thanks for the feedbacks.

这篇关于for循环内的pickle上的内存泄漏强制出现内存错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！