在不同模块之间共享队列实例 [英] Sharing a Queue instance between different modules

问题描述

我是Python的新手，我想在不同模块中创建的线程/进程之间创建一个全局静态变量"，即我的线程安全和进程安全队列.我从doc ，它是使用第三个模块(称为cfg)创建的，它定义并初始化了我的全局队列，该概念是使用第三个模块创建的.我在模块之间共享此对象的实例时遇到问题，因为我尝试在从cfg模块导入的共享队列中，在导入该对象的其他模块中打印 repr ()函数.表明它们是不同的实例.似乎每次我尝试导入模块时，都会创建一个新实例并将其传递给导入它的模块.

I am new to Python and I would like to create what is a 'global static variable', my thread-safe and process-safe queue, between threads/processes created in different modules. I read from the doc that the concept of a global variable is created using a third module, that I will call as cfg, which defines and initializes my global queue. I have a problem sharing the instance of this objects between my modules because I tried to print the repr() function over the shared queue imported from the cfg module, inside my other modules that import it, and it shows that they are different instances. It seems that every time I try to import a module a new instance is created and passed to the module who imports it.

Main.py:

import GatewayManager
if __name__ == '__main__':
    GatewayManager.initialize()
    doSomething()

GatewayManager.py:

GatewayManager.py:

import multiprocessing
import queue
import threading

def initialize():
    # Multiprocessing or Threading
    global isMonoCPU
    isMonoCPU = multiprocessing.cpu_count() == 1

    global sharedQueue
    sharedQueue = multiprocessing.Queue() if not isMonoCPU else queue.Queue()
    print("gateway: ", sharedQueue.__repr__())

otherModules.py:

otherModules.py:

import GatewayManager

# Some module write on the queue
GatewayManager.sharedQueue.put(variable)

# Some read from the queue
GatewayManager.sharedQueue.get()
print("driver: ", GatewayManager.sharedQueue.__repr__())

推荐答案

此处:

# GatewayManager.py:

...

def initialize():
    global sharedQueue
    # ...
    sharedQueue = multiprocessing.Queue()
    # ...

您的 GatewayManager 模块在调用 initialize()函数之前没有 sharedQueue 属性，因此，如果有其他模块尝试在调用 GatewayManager.initialize()之前使用 GatewayManager.sharedQueue ，那么您当然会得到此错误.而且由于 GatewayManager.initialize()在每次调用时盲目地重新绑定 sharedQueue ，因此，如果您从另一个模块再次调用它，则会丢失已经创建的队列并得到一个新的队列.

Your GatewayManager module don't have a sharedQueue attribute until it's initialize() function is called., so if any other module tries to use GatewayManager.sharedQueue before GatewayManager.initialize() has been called then of course you'll get this error. And since GatewayManager.initialize() blindly rebinds sharedQueue on each call, if you call it again from another module then you lose the already created queue and gets a new one.

您想要的是确保您的共享队列仅创建一次并且仅创建一次，并且无论发生什么情况都将创建该共享队列.这里的解决方案(至少是一个解决方案-但这是一个已知的有效解决方案)是代理所有 GatewayManager.sharedQueue.whatever 访问功能，这些功能将在需要时以及需要时初始化队列.

What you want is to make sure your sharedqueue is created once and only once, and that it will be created whatever happens. The solution (well, one solution at least - but it's a known working solution) here is to proxy all GatewayManager.sharedQueue.whatever access thru functions that will take care of initializing the queue if and when needed.

# gateway_manager.py

class _QueueProxy(object):
    def __init__(self):
        self._queueimp = None

    @property
    def _queue(self):
        if self._queueimp is None:
            isMonoCPU = multiprocessing.cpu_count() == 1
            self._queueimp = queue.Queue() if isMonoCPU else multiprocessing.Queue() 

        return self._queueimp

    def get(self, *args, **kw):
        return self._queue.get(*args, **kw)

    def put(self, *args, **kw):
        return self._queue.put(*args, **kw)

   # etc... only expose public methods and attributes of course


# and now our `shared_queue` instance    
shared_queue = _QueueProxy()

现在，您可以安全地(几乎-队列创建不是原子的，因此您可能会遇到竞争条件)从任何模块使用 gateway_manager.shared_queue ，而不必担心初始化.

And now you can safely (well almost - the queue creation is not atomic so you may have race conditions) use gateway_manager.shared_queue from any module without having to care about initialization.

当然，如果您有两个不同的进程(在这里我不是在谈论 multiprocessing.Process )，您仍然会有两个不同的队列，但是我想您已经了解了这一点(如果没有，请阅读让·保罗的答案).

Of course if you have two distinct processes (I'm not talking about multiprocessing.Process here) you will still have two distinct queues but I assume you already understood this (and if not please read Jean-Paul's answer).

这篇关于在不同模块之间共享队列实例的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！