Python，使用远程管理器和多处理 [英] Python, Using Remote Managers and Multiprocessing

问题描述

我想使用multiprocessing模块中的远程管理器功能在许多机器之间分配工作.我知道有第3方模块，但是我想尽可能地坚持使用核心.我知道对于台式机(单机)，您可以使用multiprocessing.Pool类来限制CPU的数量，但是远程管理器会有一些问题.

I want to use the remote manager functions in the multiprocessing module to distribute work among many machines. I know there are 3rd party modules, but I want to stick with core as much as possible. I know for desktop (single machine), you can use the multiprocessing.Pool class to limit the number of CPUs, but have a couple of questions with remote managers.

我为远程管理器提供以下代码:

I have the following code for the remote manager:

   from multiprocessing.managers import BaseManager
   import Queue
   queue = Queue.Queue()
   class QueueManager(BaseManager): pass
   QueueManager.register('get_queue', callable=lambda:queue)
   m = QueueManager(address=('', 50000), authkey='abracadabra')
   s = m.get_server()
   s.serve_forever()

这很好用，我什至可以使用以下代码将作业提交到队列中:

This works great, and I can even submit a job into the Queue using the following code:

QueueManager.register('get_queue')
m = QueueManager(address=('machinename', 50000), authkey='abracadabra')
m.connect()
queue = m.get_queue()
queue.put('hello')

您也可以queue.get()在队列中获得单个条目.

You can also the queue.get() to get a single entry in the queue.

1. 如何获取队列中的项目?当我尝试遍历队列时，我进入了无限循环.
2. 在工人上，您可以将每台机器限制为每台机器1个工作吗?
3. 由于此方法似乎是pull方法，因此工人需要检查作业是否存在，是否可以使用push方法来触发多处理服务器?

推荐答案

遍历队列与执行操作相同:

Iterating over a queue is the same as doing:

while True:
    elem = queue.get()  #queue empty -> it blocks!!!

一种在队列上迭代"并在没有更多要执行的作业时阻止您的工作进程的优雅方法是将None(或其他方式)用作哨兵并使用iter(callable, sentinel):

An elegant way to "iterate" over a queue and block your worker process when there are no more jobs to execute is to use None(or something else) as a sentinel and use iter(callable, sentinel):

for job in iter(queue.get, None):
    # execute the calculation
    output_queue.put(result)

#shutdown the worker process

等同于:

while True:
    job = queue.get()
    if job is None:
        break
    #execute the calculation
    output_queue.put(result)
#shutdown the worker process

请注意，您必须在队列中为每个辅助子进程插入一个前哨，否则将有子进程在等待它.

Note that you have to insert in the queu a sentinel for each worker subprocess, otherwise there will be subprocesses waiting for it.

关于第二个问题，我不明白您在问什么. BaseManager提供一个服务器，该服务器执行来自客户端的调用，因此，显然，所有请求都由同一台计算机满足. 还是说允许每个客户只做一个请求?即使可以手动"实现，我也看不到任何选择.

Regarding your second question, I don't understand what you are asking. The BaseManager provides one server that executes the calls from the clients, so, obviously, all requests are satisfied by the same machine. Or do you mean allow each client to do only a request? I don't see any option for this, even though it could be implemented "by hand".

我不明白你的问题. 什么就像拉方法?您能否以可以触发多处理服务器的推送方法"的含义来详细说明一下您的问题?

I don't understand your question. What is like a pull method? Can you rephrase your question with a bit more details on what you mean by "a push method where the multiprocessing server can be triggered"?

这篇关于Python，使用远程管理器和多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！