如何避免[Errno 12]无法分配由于使用子进程模块而导致的内存错误 [英] How to avoid [Errno 12] Cannot allocate memory errors caused by using subprocess module
问题描述
完整的工作测试用例
当然，根据您在本地和远程计算机上的内存，阵列大小会有所不同.
Of course depending on your memory on the local and remote machines your array sizes will be different.
z1 = numpy.random.rand(300000000,2);
for i in range(1000):
print('*******************************************\n');
direct_output = subprocess.check_output('ssh blah@blah "ls /"', shell=True);
direct_output = 'a'*1200000;
a2 = direct_output*10;
print(len(direct_output));
当前用例
以防我的用例如下:
我发出数据库查询，然后将结果表存储在远程计算机上.然后，我想跨网络传输它们并进行分析.到目前为止，我一直在python中执行以下操作:
I issue db queries then store the resulting tables on the remote machine. I then want to transfer them across a network and do analysis. Thus far I have been doing something like the following in python:
#run a bunch of queries before hand with the results in remote files
....
counter = 0
mergedDataFrame = None
while NotDone:
output = subprocess.check_output('ssh blah@blah cat /data/file%08d'%(counter))
data = pandas.read_csv(...)
#do lots of analysis, append, merge, numpy stuff etc...
mergedDataFrame = pandas.merge(...)
counter += 1
在某些时候，我在check_output命令中收到以下错误:[Errno 12]无法分配内存
At some point I receive the following error at the check_output command: [Errno 12] Cannot allocate memory
背景
由于以下问题，我想我对什么地方有错了.已经发布了许多解决方案，我正在尝试确定哪种解决方案可以避免[Errno 12]无法分配与使用fork/clone的子流程实现相关的内存错误.
Thanks to the below questions I think I have an idea of what is wrong. There are a number of solutions posted, and I am trying to determine which of the solutions will avoid the [Errno 12] Cannot allocate memory error associated with the subprocess implementation using fork/clone.
Python子进程.Popen" OSError:[Errno 12 ]无法分配内存".这给出了基本的诊断，并提出了一些解决方法，例如生成单独的脚本等...
Python subprocess.Popen "OSError: [Errno 12] Cannot allocate memory" This gives the underlying diagnosis and suggests some workaround like spawning separate script etc...
了解Python派生和内存分配错误建议使用rfoo来规避fork/clone的子进程限制以及产生子进程和复制内存等...这似乎暗示着客户端-服务器模型
Understanding Python fork and memory allocation errors Suggests using rfoo to circumvent the subprocess limitation of fork/clone and spawning child process and copy memory etc... This seems to imply a client-server model
什么是最简单的SSH方式使用Python?，但是由于内存限制和fork/clone实现，我还有其他约束不能使用子进程?解决方案建议使用paramiko或基于其的东西，其他建议使用子流程(我发现在我的情况下将不起作用).
What is the simplest way to SSH using Python? , but I have the additional constraints that I cannot use subprocess due to memory limitations and fork/clone implementation? The solutions suggests using paramiko or something built on top of it, others suggest subprocess (which I have found will not work in my case).
还有其他类似的问题，但是答案经常是关于文件描述符是罪魁祸首(在这种情况下不是)，为系统增加了更多RAM(我不能这样做)，升级到x64(我已经在x64上了) ).一些提示有关ENOMEM的问题.一些答案提到试图确定subprocess.Popen(在我的情况下为check_output)是否没有正确清理进程，但看起来S. Lott和其他人都同意子进程代码本身正在正确清理.
There were other similar questions but the answers often talked about file descriptors being the culprit (in this case they are not), adding more RAM to the system ( I cannot do this), upgrading to x64 ( I already am on x64). Some hint at the problem of ENOMEM. A few answers mention trying to determine if the subprocess.Popen (in my case check_output) is not properly cleaning the processes, but it looks like S. Lott and others agree that the subprocess code itself is properly cleaning up.
- 使用subprocess.Popen的Python内存分配错误
- 尽管有很多，Python IOError也无法分配内存
- 无法在Popen命令上分配内存
- Python子进程.Popen出现OSError错误:[Errno 12]在一段时间后无法分配内存
- Python memory allocation error using subprocess.Popen
- Python IOError cannot allocate memory although there is plenty
- Cannot allocate memory on Popen commands
- Python subprocess.Popen erroring with OSError: [Errno 12] Cannot allocate memory after period of time
我在github https://github上搜索了源代码.com/paramiko/paramiko/search?q = Popen& type = Code ，它似乎在proxy.py文件中使用了子进程.
I have searched through the source code on github https://github.com/paramiko/paramiko/search?q=Popen&type=Code and it appears to use subprocess in the proxy.py file.
实际问题
这是否意味着paramiko最终将使用上述Popen解决方案，当python内存占用量增加并且由于克隆/叉实现而导致重复进行Popen调用时，将会出现问题?
Does this mean that ultimately paramiko is using the Popen solution described above that will have problems when the python memory footprint grows and repeated Popen calls are made due to the clone/fork implementation?
如果paramiko无法正常工作，是否还有另一种方法可以解决我仅使用客户端的解决方案?还是需要一个客户端/服务器/套接字解决方案?如果是这样，那么rfoo，tornado或zeromq，http传输中的任何一种都可以在这里工作?
If paramiko will not work is there another way to do what I am looking for with a client side only solution? Or will a client/server/socket solution be needed? If so will any of rfoo, tornado, or zeromq, http transfers work here?
注释 我正在运行64位linux 8GB主内存.我不想追求购买更多RAM的选择.
Notes I am running 64bit linux 8GB main memory. I do not want to pursue the options of buying more RAM.
推荐答案
如果内存不足，则可能需要增加交换内存.否则，您可能根本没有启用任何交换.在Ubuntu中(它也应适用于其他发行版)，您可以通过以下方式检查交换情况:
If you are running out of memory, you may want to increase your swap memory. Or you might have no swap enabled at all. In Ubuntu (it should work for other distributions as well) you can check your swap by:
$sudo swapon -s
如果为空，则表示您没有启用任何交换.要添加1GB的交换空间，请执行以下操作:
if it is empty it means you don't have any swap enabled. To add a 1GB swap:
$sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024k
$sudo mkswap /swapfile
$sudo swapon /swapfile
将以下行添加到fstab以使交换永久生效.
Add the following line to the fstab to make the swap permanent.
$sudo vim /etc/fstab
/swapfile none swap sw 0 0
可在此处找到来源和更多信息. >.
Source and more information can be found here.
这篇关于如何避免[Errno 12]无法分配由于使用子进程模块而导致的内存错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
