使用Python Bottle，Multiprocessing和gevent进行流式连接 [英] Streaming Connection Using Python Bottle, Multiprocessing, and gevent

问题描述

我有一个Bottle应用程序，该应用程序使用子流程来完成请求的大部分工作.对于返回单个响应的路由，我将执行以下操作.

I have a Bottle application that uses subprocesses to do most of the work for requests. For routes that return a single response, I do something like what's below.

@route('/index')
def index():
    worker = getWorker()
    return worker.doStuff()

我的路线之一必须是数据流.我想不出一种聪明的方法让工作人员返回响应流.下面的示例类似于我想要做的，只是没有工人.

One of my routes needs to be a data stream. I can't figure out a smart way to have the worker return a response stream. The example below is similar to what I want to do, only without a worker.

@route('/stream')
def stream():
    yield 'START'
    sleep(3)
    yield 'MIDDLE'
    sleep(5)
    yield 'END'

我希望能够做以下类似的事情.由于我无法产生/返回发电机，因此这种方式是不可能的.

I'd like to be able to do something like below. Since I can't yield/return a generator this isn't possible this way.

@route('/stream')
def stream():
    worker = getWorker()
    yield worker.doStuff()
class worker:
    # Remember, this is run in a subprocess in real life.
    def doStuff():
        yield 'START'
        sleep(3)
        yield 'MIDDLE'
        sleep(5)
        yield 'END'

这是一个大型项目，我做事的方式没有太多灵活性.我知道有时候最简单的答案是您的设计是错误的".但是在这种情况下，我有一些无法控制的约束(路由必须是数据流，而工作必须由子流程完成).

This is for a large project and I don't have much flexibility in the way that I do things. I know sometimes the easiest answer is "your design is wrong." In this case though, I have some constraints that are beyond my control (the route has to be a data stream and the work has to be done by a subprocess).

编辑 我也不能有doStuff()块.我希望能够创建类似我返回的gevent队列并具有工作进程的东西.现在的问题是，似乎我无法同时使用gevent.queue和Process.

EDIT I also can't have doStuff() block. I'd like to be able to create something like a gevent queue that I return and have the worker process. The problem now is that it doesn't seem like I can use gevent.queue and Process together.

@route('/stream')
def index():
    body = gevent.queue.Queue()
    worker = multiprocessing.Process(target=do_stuff, args=body)
    worker.start()
    return body()

def do_stuff(body):
    while True:
        gevent.sleep(5)
        body.put("data")

推荐答案

经过大量研究和实验，我确定gevent队列不能以这种方式与Python多处理一起使用.除了使用这种方式进行处理之外，还可以使用redis之类的方法来允许进程和gevent greenlet进行通信.

After a lot of research and experimenting, I've determined that a gevent queue can't be used with Python multiprocessing in this way. Instead of doing things this way, something like redis can be used to allow the processes and gevent greenlets communicate.

@route('/stream')
def index():
    worker = multiprocessing.Process(target=do_stuff)
    worker.start()
    yield redis_server.lpop()

def do_stuff(body):
    while True:
        gevent.sleep(5)
        redis_server.lpush("data")

这篇关于使用Python Bottle，Multiprocessing和gevent进行流式连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！