使用 PyZMQ 限制队列长度 [英] Limiting queue length with PyZMQ

问题描述

我想限制 Python 应用程序中 ZeroMQ 消息队列消耗的内存量.我知道设置高水位会限制发送方排队的数量，但是有没有办法控制接收方排队的数量?Python ZeroMQ 绑定似乎将其设置为无限制.

I want to limit the amount of memory consumed by my ZeroMQ message queues in a Python application. I know that setting the high-water mark will limit the amount that will be queued on the sender side, but is there a way to control how much will be queued on the receiver side? The Python ZeroMQ binding seems to have it set at unlimited.

我的测试场景:我有两个用于测试的 Python 终端.一个是接收方:

My test scenario: I have two python terminals that I am using for testing. One is the receiver:

Python 2.5.1 (r251:54863, Aug 25 2008, 20:50:04) 
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import zmq
>>> context = zmq.Context()
>>> socket = context.socket(zmq.PULL)
>>> socket.setsockopt(zmq.RCVBUF, 256)
>>> socket.bind("tcp://127.0.0.1:12345")

另一个是发件人:

Python 2.5.1 (r251:54863, Aug 25 2008, 20:50:04) 
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import zmq
>>> context=zmq.Context()
>>> socket = context.socket(zmq.PUSH)
>>> socket.setsockopt(zmq.SNDBUF, 2048)
>>> socket.setsockopt(zmq.HWM, 1)
>>> socket.connect("tcp://127.0.0.1:12345")
>>> num = 0
>>> while True:
...  print num
...  socket.send(str(num))
...  num = num + 1
... 

我在接收端运行了几次 socket.recv() 以确保队列正常工作，但除此之外，让两个终端坐在那里.发送循环似乎从不阻塞，接收提示似乎有越来越多的内存占用.

I run socket.recv() on the receiver side a couple times to make sure that the queue works, but other than that, let the two terminals just sit there. The send loop seems to never block and the receive prompt seems to have a growing memory footprint.

推荐答案

与 ZeroMQ 的文档矛盾，PUSH高水位> 侧和 PULL 侧.一旦我更改了 PULL，效果会更好.新的 PULL 代码是:

In contradiction to the documentation of ZeroMQ, the high water mark needs to be set on both the PUSH side and the PULL side. Once I changed the PULL, it worked better. The new PULL code is:

Python 2.5.1 (r251:54863, Aug 25 2008, 20:50:04) 
[GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import zmq
>>> context=zmq.Context()
>>> socket = context.socket(zmq.PULL)
>>> socket.setsockopt(zmq.RCVBUF, 256)
>>> socket.setsockopt(zmq.HWM, 1)
>>> socket.bind("tcp://127.0.0.1:12345")

这篇关于使用 PyZMQ 限制队列长度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！