Python multiprocessing.managers.BaseManager依次运行已注册的可调用函数 [英] Python multiprocessing.managers.BaseManager running registered callable function sequentially

问题描述

我正在使用Python的多处理库提供的远程管理器.我已经使用BaseManager设置了一个远程服务器，多个客户端同时连接到该服务器.不幸的是，我的服务器正在按顺序为每个客户端提供请求.我的服务器应该通过网络调用Google的Directions API来返回距离和时间.

I am working with a remote manager provided by Python's multiprocessing library. I have setup a remote server using BaseManager, to which multiple clients connect simultaneously. Unfortunately, my server is serving requests sequentially for each client. My server is supposed to make a network call to Google's directions API to return distance and time.

我的理解是，将为每个连接的客户端生成一个新线程，因此我不会遇到这个问题.

My understanding was that a new thread would be spawned for each client that connects, so i would not face this problem.

我以简化的方式提供了代码示例.

I have provided a sample of my code in a simplified manner.

这是服务器代码:

import time
from multiprocessing.managers import BaseManager
import threading

class DistanceTime:

    def get_distance_time(self):
        print('started by thread %s'%(threading.get_ident()))
        # assume that network request was made here
        time.sleep(2)
        print('ended by thread %s'%(threading.get_ident()))

def server():
    distance_time=DistanceTime()
    BaseManager.register('get_distance_time', callable=distance_time.get_distance_time)
    manager = BaseManager(address=('localhost', 5000), authkey=b'abracadabra')
    server = manager.get_server()
    print('server running')
    server.serve_forever()

server()

这是客户端代码:

from multiprocessing.managers import BaseManager
from concurrent.futures import ThreadPoolExecutor
import time

def client():
    BaseManager.register('get_distance_time')
    manager = BaseManager(address=('localhost', 5000), authkey=b'abracadabra')
    manager.connect()
    executor = ThreadPoolExecutor(max_workers=3)
    # client mades three simultaneous requests to the server
    b=executor.submit(manager.get_distance_time)
    b=executor.submit(manager.get_distance_time)
    c=executor.submit(manager.get_distance_time)
    print('done')
    time.sleep(5)

client()

即使客户端立即发送所有三个请求，服务器也会打印以下内容:

Even though the client sends all three requests immediately, the server prints the following:

server running
started by thread 16740
ended by thread 16740
started by thread 4712
ended by thread 4712
started by thread 7132
ended by thread 7132

理想情况下，所有开始打印的照片都应该放在一起.这是我的应用程序的主要瓶颈.

Ideally, all started prints should come together. This is a major bottleneck for my application.

推荐答案

您正在register调用的可调用对象是一种创建"方法，它们是

the callable you're registering is a 'creation' method and these are always run in a locked context, but the object it returns is automatically proxied and any methods invoked on it aren't automatically locked

在您的演示代码中，我将进行以下更改:

in your demo code, I'd change:

def server():
    distance_time=DistanceTime()
    BaseManager.register('get_distance_time', callable=distance_time.get_distance_time)

成为:

def server():
    distance_time = DistanceTime()
    BaseManager.register('DistanceTime', lambda: distance_time)

，然后将其用作:

distance_time = manager.DistanceTime()
a = executor.submit(distance_time.get_distance_time)
b = executor.submit(distance_time.get_distance_time)
c = executor.submit(distance_time.get_distance_time)

应该允许所有内容并行进行.我没有实际测试过，但是如果您说这行不通的话……

which should allow everything go in parallel. I've not actually tested this, but will if you say this doesn't work...

这并不重要，但是我通常认为最好将这些东西记录在单独的/派生的Manager

not that it matters here, but I generally feel it's better to register these sorts of things in a separate/derived Manager

这篇关于Python multiprocessing.managers.BaseManager依次运行已注册的可调用函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！