使用maxtasksperchild进行python多重处理 [英] python multiprocessing with maxtasksperchild

问题描述

我确认这是Python中的错误.这是错误 http://bugs.python.org/issue10332 (作为回应，我提交了一个新的错误维护人员将我指向10332).我将多处理目录从Python源库中复制到了我的项目目录中，并且测试用例现在可以正常工作了.

这个看似简单的程序对我不起作用，除非删除了maxtasksperchild参数.我在做什么错了?

from multiprocessing import Pool
import os
import sys

def f(x):
  print "pid: ", os.getpid(), " got: ", x
  sys.stdout.flush()
  return [x, x+1]

def cb(r):
  print "got result: ", r

if __name__ == '__main__':
  pool = Pool(processes=1, maxtasksperchild=9)
  keys = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  result = pool.map_async(f, keys, chunksize=1, callback=cb)
  pool.close()
  pool.join()

运行它，我得到:

$ python doit.py
pid:  6409  got:  1
pid:  6409  got:  2
pid:  6409  got:  3
pid:  6409  got:  4
pid:  6409  got:  5
pid:  6409  got:  6
pid:  6409  got:  7
pid:  6409  got:  8
pid:  6409  got:  9

它挂了.也就是说，没有生成处理第10个元素的新工作者.

在另一个终端，我看到:

$ ps -C python
  PID TTY          TIME CMD
 6408 pts/11   00:00:00 python
 6409 pts/11   00:00:00 python <defunct>

这是在运行python 2.7.2+(从ubuntu软件包安装)的Ubuntu 11.10上完成的.

解决方案

此问题已在python3中修复.

pid:  18316  got:  1
pid:  18316  got:  2
pid:  18316  got:  3
pid:  18316  got:  4
pid:  18316  got:  5
pid:  18316  got:  6
pid:  18316  got:  7
pid:  18316  got:  8
pid:  18316  got:  9
pid:  18317  got:  10
got result:  [[1, 2], [2, 3], [3, 4], [4, 5], [5, 6], [6, 7], [7, 8], [8, 9], [9, 10], [10, 11]]

EDIT: I confirmed this to be bug in Python. It is bug http://bugs.python.org/issue10332 (I filed a new bug, in response to which the maintainer pointed me to 10332). I copied the multiprocessing directory from Python source repo into my project directory, and the testcase works properly now.

This seemingly-simple program isn't working for me unless I remove the maxtasksperchild parameter. What am I doing wrong?

from multiprocessing import Pool
import os
import sys

def f(x):
  print "pid: ", os.getpid(), " got: ", x
  sys.stdout.flush()
  return [x, x+1]

def cb(r):
  print "got result: ", r

if __name__ == '__main__':
  pool = Pool(processes=1, maxtasksperchild=9)
  keys = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
  result = pool.map_async(f, keys, chunksize=1, callback=cb)
  pool.close()
  pool.join()

When I run it, I get:

$ python doit.py
pid:  6409  got:  1
pid:  6409  got:  2
pid:  6409  got:  3
pid:  6409  got:  4
pid:  6409  got:  5
pid:  6409  got:  6
pid:  6409  got:  7
pid:  6409  got:  8
pid:  6409  got:  9

And it hangs. That is, the new worker to process the 10th element didn't get spawned.

In another terminal, I see:

$ ps -C python
  PID TTY          TIME CMD
 6408 pts/11   00:00:00 python
 6409 pts/11   00:00:00 python <defunct>

This is done on Ubuntu 11.10 running python 2.7.2+ (installed from ubuntu packages).

解决方案

This issue has been fixed in python3.

pid:  18316  got:  1
pid:  18316  got:  2
pid:  18316  got:  3
pid:  18316  got:  4
pid:  18316  got:  5
pid:  18316  got:  6
pid:  18316  got:  7
pid:  18316  got:  8
pid:  18316  got:  9
pid:  18317  got:  10
got result:  [[1, 2], [2, 3], [3, 4], [4, 5], [5, 6], [6, 7], [7, 8], [8, 9], [9, 10], [10, 11]]

这篇关于使用maxtasksperchild进行python多重处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！