ZMQ PUB 发送文件 [英] ZMQ PUB Send file

问题描述

我第一次尝试 (PY)ZMQ，想知道是否可以使用 PUB/SUB 发送完整的 FILE(二进制)?我需要向许多订阅者发送数据库更新.我看到了短消息的例子，但没有看到文件.可能吗?

I'm trying (PY)ZMQ for the first time, and wonder if it's possible to send a complete FILE (binary) using PUB/SUB? I need to send database updates to many subscribers. I see examples of short messages but not files. Is it possible?

出版商:

import zmq
import time
import os
import sys

while True:

print 'loop'
msg = 'C:\TEMP\personnel.db'

# Prepare context & publisher
context = zmq.Context()
publisher = context.socket(zmq.PUB)
publisher.bind("tcp://*:2002")
time.sleep(1)

curFile = 'C:/TEMP/personnel.db'
size = os.stat(curFile).st_size
print 'File size:',size

target = open(curFile, 'rb')
file = target.read(size)
if file:
    publisher.send(file)

publisher.close()
context.term()
target.close()
time.sleep(10)

订阅者:

'''always listening'''

import zmq
import os
import time
import sys

while True:

path = 'C:/TEST'
filename = 'personnel.db'
destfile = path + '/' + filename

if os.path.isfile(destfile):
    os.remove(destfile)
    time.sleep(2)

context = zmq.Context()
subscriber = context.socket(zmq.SUB)
subscriber.connect("tcp://127.0.0.1:2002")
subscriber.setsockopt(zmq.SUBSCRIBE,'')

msg = subscriber.recv(313344)
if msg:
    f = open(destfile, 'wb')
    print 'open'
    f.write(msg)
    print 'close\n'
    f.close()

time.sleep(5)

推荐答案

您应该能够使用 zmq 和 PUB/SUB 模式完成将文件分发给多个订阅者.

You shall be able to accomplish to distribute files to many subscribers using zmq and PUB/SUB pattern.

你的代码就快完成了，或者换句话说，它可以在大多数情况下工作，可以稍微改进一下.

Your code is almost there, or in other words, it might work in most situations, could be improved a bit.

消息在发布时必须适合内存(位于 PUB 套接字中)并保持在那里，直到最后一个当前订阅的消费者没有读取它或断开连接.

The message must fit into memory when getting published (living in PUB socket) and stays there until last currently subscribed consumer does not read it out or disconnects.

消息在收到时也必须适合内存.但是对于合理的大文件(比如你的 313 kB)，它应该可以工作，除非你的 RAM 真的很短.

The message must also fit into memory when being received. But with reasonable large files (like your 313 kB) it shall work unless you are really short with RAM.

如果您有多个消费者，并且其中一个的读取速度比其他的慢得多，它将开始减慢所有消费者的速度.Zmq 正在解释这个问题，并提出了一些如何避免它的方法(例如慢速订阅者的自杀).

In case you have multiple consumers, and one of them is reading much slower then the others, it will start slowing down all of them. Zmq is explaining this problem and also proposes some methods how to avoid it (e.g. suicide of slow subscriber).

但是，在大多数情况下，您不会遇到此问题.

However, in most situations, you will not encounter this problem.

zmq 消息传递非常快.没问题，如果你早点启动你的消费者，那么发布者，zmq 使这个场景变得容易，消费者会自动连接.

zmq messaging is extremely fast. There is no problem, if you start your consumer sooner, then the publisher, zmq makes this scenario easy and consumer will connect automatically.

但是，您的发布者应允许消费者在开始发布之前进行连接，您的代码在发送消息前会休眠 1 秒，这就足够了.

However, your publisher shall allow consumers to connect before it start publishing, your code does 1 second sleep before sending the message, this shall be sufficient.

• 你真的需要在 os.remove 之后睡觉吗?可能不会
• subscriber.recv - 不需要提前知道消息大小，zmq 包知道文件大小，所以如果你调用它没有接收字节数，你会正确地得到它.

zmq 提供了一个称为多部分消息的功能，但根据 doc，它必须在发送之前完全适合(所有消息部分)在内存中，所以这不是使用技巧.

zmq provides a feature called multipart messages, but according to doc, it has to fit completely (all message parts) in memory, before being sent out, so this is not the trick to use.

另一方面，您可以通过这种方式创建应用程序级多部分协议"，您可以决定发送具有类似 (hasNextPart, chunkData) 结构的消息.这样你就可以发送大小控制良好的消息，只有最后一个会告诉 "hasNextPart" == False.

On the other hand, you can create "application level multipart protocol" in such a way, that you decide sending messages with structure like (hasNextPart, chunkData). This way you would be sending in well controlled sized messages and only the last one would tell "hasNextPart" == False.

然后，消费者将读取所有部分并将其写入磁盘，直到最后一条消息，声称没有进一步的部分到达.

Consumer would then read and write to disk all the parts until last message, claiming that there is no further part arrives.

这篇关于ZMQ PUB 发送文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！