将Twisted Python与多处理功能混合在一起? [英] Mix Python Twisted with multiprocessing?

问题描述

我需要用Python编写类似程序的代理，其工作流程与网络代理非常相似.该程序位于客户端和服务器之间，接受客户端向服务器发送的请求，处理该请求，然后将其发送到原始服务器.当然，使用的协议是使用TCP的专用协议.

I need to write a proxy like program in Python, the work flow is very similar to a web proxy. The program sits in between the client and the server, incept requests sent by the client to the server, process the request, then send it to the original server. Of course the protocol used is a private protocol uses TCP.

为了最大程度地减少工作量，我想使用Twisted Python处理请求的接收(该部分充当服务器)和重新发送(该部分充当客户端).

To minimize the effort, I want to use Python Twisted to handle the request receiving (the part acts as a server) and resending (the part acts as a client).

为了最大化性能，我想使用python多处理(线程具有GIL限制)将程序分为三个部分(进程).第一个进程运行Twisted来接收请求，将请求放入队列，然后立即将成功返回给原始客户端.第二个进程从队列中获取请求，进一步处理该请求并将其放入另一个队列.第三进程从第二队列获取请求，并将其发送到原始服务器.

To maximum the performance, I want to use python multiprocessing (threading has the GIL limit) to separate the program into three parts (processes). The first process runs Twisted to receive requests, put the request in a queue, and return success immediately to the original client. The second process take request from the queue, process the request further and put it to another queue. The 3rd process take request from the 2nd queue and send it to the original server.

我是Python Twisted的新手，我知道它是事件驱动的，我还听说最好不要将Twisted与线程或多处理混合使用.因此，我不知道这种方法是否合适，还是仅通过使用Twisted会有更优雅的方法吗?

I was a new comer to Python Twisted, I know it is event driven, I also heard it's better to not mix Twisted with threading or multiprocessing. So I don't know whether this way is appropriate or is there a more elegant way by just using Twisted?

推荐答案

Twisted具有其自己的事件驱动方式来运行子流程，这种方式(按我的看法，很正确，但我认为是正确的)比multiprocessing模块更好.核心API是 spawnProcess ，但类似

Twisted has its own event-driven way of running subprocesses which is (in my humble, but correct, opinion) better than the multiprocessing module. The core API is spawnProcess, but tools like ampoule provide higher-level wrappers over it.

如果您使用 spawnProcess ，您将能够以与处理Twisted中任何其他事件相同的方式处理子流程的输出；如果使用multiprocessing，则需要开发自己的基于队列的方法，以某种方式将子进程的输出获取到Twisted主循环中，因为通常

If you use spawnProcess, you will be able to handle output from subprocesses in the same way you'd handle any other event in Twisted; if you use multiprocessing, you'll need to develop your own queue-based way of getting output from a subprocess into the Twisted mainloop somehow, since the normal callFromThread API that a thread might use won't work from another process. Depending on how you call it, it will either try to pickle the reactor, or just use a different non-working reactor in the subprocess; either way it will lose your call forever.

这篇关于将Twisted Python与多处理功能混合在一起?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！