uwsgi工人之间的沟通 [英] Communication between workers in uwsgi
问题描述
我是Python的新手，来自.Net背景.
I'm relatively new to Python, coming from a .Net background.
简短版本:如何创建应用程序范围内的单例或某种其他机制，以允许多个线程/进程相互通信?
The short version: How can I create an application-wide singleton or some other mechanism to allow multiple threads/processes to communicate with each other?
也许我被宠坏了，但是在.Net中，我只是在App_Start中创建某些内容，或者在应用程序级别"创建其他内容.我该如何在python/uwsgi中做同样的事情?
Maybe I'm spoilt but in .Net, I'd just create something in App_Start or something else at the "application level". How can I do the same in python/uwsgi?
长版:
我们已经使用Django编写了一个宁静的API.
We've got a restful API written using Django.
某些调用需要进行一些预处理，然后传递到执行长时间运行操作的后端系统.
Some of the calls require some pre-processing and then passing to a back-end system which performs long-running operations.
此刻的流程看起来像...
The flow at the moment looks something like...
- 接收请求以处理所有符合给定条件的文档
- Api确定哪些文档符合这些条件(大约100,000，耗时15-20s)
- Api为此批处理请求生成一个uuid
- Api将引用批次ID的消息发布到每个文档的后端队列.
- Api在不同的队列上侦听已完成"消息，并为每个批次ID(〜1-15分钟)计算成功/失败
- 正在进行处理时，UI可以请求更新特定批次ID
We need to listen to the response queue using a different thread to that which is used to serve pages since it's in a wait spin-loop...
while True:
self.channel.wait()
我通过获取对QueueManager的引用来处理此问题，该c1是单例.管理器触发初始请求，记录批次ID，然后在第二个线程中监视队列并更新本地状态.
I was handling this by getting a reference to a QueueManager which is a singleton. The manager fires off the initial request, records the batch id and then, in a second thread, monitors the queue and updates local state.
我们实际上并不关心长期保存状态-如果消息在队列中，则处理将由后端完成，并且状态监视仅是提示用户事情正在进行中.如果他们浏览，他们也将无法访问状态(批次ID存储在JS的内存中)
We don't actually care about preserving the state long-term - If the messages are on the queue, the processing will be done by the back-end and the state monitoring is only a cue to the user that things are underway. If they browse away, they also lose access to the status (batch id is stored in-memory in JS)
这有很多好处-我们避免使用数据库来同步信息(以及相关的清理).我们能够将单个线程用作消息使用者，而不必担心并发问题，因为只有一个线程可以收集消息/更新本地状态.
This had a couple of benefits - we avoided using a database for syncing information (and the associated cleanup). We were able to use a single thread as a message consumer and didn't have to worry about concurrency issues as only one thread will ever collect messages/update the local state.
所以...现在是时候使用uwsgi运行它了，我发现了一个主要问题.如果我将进程数设置为1，则单例可以按预期工作，但是在api正在处理数据的15-20秒钟内，所有请求都被阻止.显然，这是不可接受的.相反，如果我增加了多个工作人员，则每个工作人员都有自己的单身人士和自己的消息侦听器-因此，如果发布者和使用者是同一过程，则几乎是随机的.即使是，更新状态的请求也可能不会在相同的过程中结束.
So... Now it's time to run it using uwsgi I've found a major issue. If I set the number of processes to 1, the the singleton works as expected, but all requests are blocked during the 15-20s that the api is processing data. Clearly that's unacceptable. Conversely, if I spin up multiple workers, each has its own singleton and its own message listener - so it's pretty much random if the publisher and consumer are the same process. And even if they are, the request for a status update probably won't end up at that same process.
如何在多个工作程序之间交换状态信息?有没有办法使用多个线程而不是多个工作线程?
How can I swap state information between multiple workers? Is there a way to use multiple threads instead of multiple workers?
似乎我真的需要:
- n个线程，每个服务请求
- 1个线程正在队列中侦听
- 它们之间进行通信的内存方式
请注意，我已经拥有--enable-threads，但这似乎仅适用于我产生的新线程(不知道为什么默认情况下不会启用该功能)
Note I've already got --enable-threads but that only seems to apply to new threads I spawn (no idea why that wouldn't be enabled by default)
推荐答案
要生成多个线程，只需添加--threads N，其中N是要生成的线程数.
To spawn multiple threads just add --threads N where N is the number of threads to spawn.
请注意，仅当您有单个工作程序/流程时，此方法才有效.另一种常见的方法是使用uWSGI缓存框架(名称具有误导性，实际上是共享字典).它将允许您在辅助线程和线程之间共享数据.
Pay attention, it will works only if you have a single worker/process. Another common approach is using the uWSGI caching framework (the name is misleading, infact is a shared dictionary). It will allows you to share data between workers and threads.
这篇关于uwsgi工人之间的沟通的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
