Python的multiprocessing.Pool是否支持远程子进程? [英] Does Python's multiprocessing.Pool support remote subprocesses?

问题描述

我最近发现Pool.map()对于简单的多处理任务有多么出色，但是我的本地计算机只有两个处理器，我希望利用一些远程计算资源.

I have recently discovered how awesome Pool.map() is for simple multiprocessing tasks, but my local machine only has two processors and I was hoping to take advantage of some remote computing resources.

是否可以构建处理池以透明方式使用远程计算机和本地计算机?

Can a processing pool be constructed to use remote machines as well as the local machines in a transparent way?

我需要在远程计算机上运行什么python代码，如何在本地(主)计算机上设置池?

What python code would I need to run on the remote machines, How would I set up the pool on the local (master) machine?

在用于多处理的文档页面上，它表示管理器将用于共享对象，但是我不清楚如何将它们用于远程池.除了Managers之外，似乎没有其他任何东西可以引用端口和地址，因此我猜想所有远程活动都将通过此类进行.

On the documentation page for multiprocessing, It says Managers are to be used for shared objects, but it's unclear to me how they should be used for a remote Pool. Nothing other than Managers seems make any reference to ports and addresses, so I'm guessing all remote activity happens through this class.

推荐答案

multiprocessing.Pool似乎不是为分布式处理而设计的-如果您要这样做，建议您查看

multiprocessing.Pool seems not designed for distributed processing -- if that's what you want to do, I suggest you check out the execnet package instead. That works over SSH pipes and sends code along to the remote Python interpreter if necessary.

这篇关于Python的multiprocessing.Pool是否支持远程子进程?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！