是否有BSD或MIT许可下发布的ZeroMQ的Python 2.7替代产品? [英] Are there any Python 2.7 alternatives to ZeroMQ that are released under the BSD or MIT license?
问题描述
我正在寻找在BSD或MIT许可下发布的Python 2.7替代ZeroMQ的方法.我正在寻找一种支持请求-答复和发布-订阅消息传递模式的东西.如有必要，我可以自己序列化数据.我从Twisted Matrix Labs找到了Twisted，但它似乎需要一个阻塞事件循环，即reactor.run().我需要一个在后台运行的库，并让我的应用程序在发生某些事件时检查消息.还有其他选择吗?
给nanomsg，一个ZeroMQ妹妹，尝试-同一个父亲，同一个美女
- 是的，它已获得 MIT /X11许可.
- 是，
REQ/REP-允许构建无状态服务集群以处理用户请求 - 是的，
PUB/SUB-将消息分发给大量感兴趣的订户 - 具有多个可用的Python绑定
https://github.com/tonysimpson/nanomsg-python (推荐)
https://github.com/sdiehl/pynanomsg
nanomsg和ZeroMQ之间的区别
(截至2014/11 v0.5-beta的状态-由 nanomsg.org提供>> >单击原始的HyperDoc )
许可
nanomsg库是MIT许可的.这意味着，与ZeroMQ不同，您可以修改源代码，并在其他许可下(作为专有产品等)重新发布源代码.有关许可的更多原因，请参见此处找到更多讨论.
发送和接收功能(nn_send，nn_sendmsg，nn_recv和nn_recvmsg)完全匹配POSIX语法和语义.
实施语言
该库是用C而不是C ++实现的.
从用户的角度来看，这意味着不依赖C ++运行时(libstdc ++或类似程序)，这在受限和嵌入式环境中可能会很方便.
从nanomsg开发人员的角度来看，它使生活更轻松.
由于使用了侵入式容器而不是C ++ STL容器，因此内存分配的数量大大减少了.
上面的内容还意味着更少的内存碎片，更少的缓存未命中等.
可以在此处和 transport.h 和 protocol.h )旨在缓解该问题并作为基础用于创建和尝试新的传输方式和协议.
请注意，这两个API仍然是新的，将来可能会进行一些调整，以使其可在多种情况下使用.
nanomsg实现了新的SURVEY协议.这个想法是向多个对等方发送一条消息(调查")，并等待所有对等方的响应.有关更多详细信息，请参见文章此处.另外，请在此处中查找.
在金融服务中，通常使用将消息从任何人传递给其他人"的消息传递.为了解决这个用例，在nanomsg中实现了一个新的BUS协议.在此处查看详细信息.
线程模型
我在ZeroMQ中所做的重大架构错误之一就是其线程模型.每个单独的对象仅由单个线程管理.这对于由工作线程处理的异步对象非常有效，但是，对于由用户线程管理的对象则成为麻烦.该线程可用于在任意时间段内执行不相关的工作，例如一个小时后，在此期间，它所管理的对象将完全卡住.不幸的结果是:无法实现以REQ/REP协议重发的请求，在应用程序执行其他工作时未应用PUB/SUB订阅等.在nanomsg中，对象没有紧密地绑定到特定线程，因此这些问题不存在.
REQ套接字不能在实际环境中真正使用，因为如果由于服务故障或类似原因导致消息丢失，它们将卡住.用户必须改用XREQ并实现自己重试的请求.使用nanomsg，重试功能内置在REQ套接字中.
在nanomsg中，REQ和REP均支持取消正在进行的处理.只需发送一个新请求而无需等待答复(对于REQ套接字)，或者获取一个新请求而无需答复上一个请求(对于REP套接字).
在ZeroMQ中，由于其线程模型的原因，首先绑定然后连接第二的场景不适用于Inproc传输.它以nanomsg固定.
由于类似的原因，自动重新连接不适用于ZeroMQ中的过程内传输.这个问题在nanomsg中也得到解决.
最后，nanomsg尝试使nanomsg套接字具有线程安全性.虽然仍然不鼓励使用来自多个线程的并行并行的单个套接字，但是在这种情况下ZeroMQ套接字随机失败的方式被证明是痛苦且难以调试的.
状态机
nanomsg库内部的内部交互被建模为一组状态机.目的是避免出现在ZeroMQ中无法理解的关闭机制，从而使库的开发更加容易.
TCP传输增强功能
TCP传输有一个较小的增强.连接时，可以选择指定用于连接的本地接口，如下所示:
nn_connect (s, "tcp://eth0;192.168.0.111:5555").
异步DNS
DNS查询(例如，将主机名转换为IP地址)以异步方式完成.在ZeroMQ中，此类查询是同步完成的，这意味着当DNS不可用时，整个库(包括未使用DNS的套接字)都将挂起.
零复制
虽然ZeroMQ提供了零复制" API，但它不是真正的零复制.而是零复制直到消息到达内核边界".从那时起，将像使用标准TCP一样复制数据.另一方面，nanomsg旨在支持真正的零复制机制，例如 RDMA (CPU旁路，直接内存到内存复制)和 shmem (使用共享内存在同一盒子上的进程之间进行数据传输).零复制消息传递的API入口点是nn_allocmsg和nn_freemsg函数，以及传递给send/recv函数的NN_MSG选项.
有效的订阅匹配
在ZeroMQ中，简单尝试用于存储和匹配PUB/SUB订阅.订阅机制旨在用于最多10,000个订阅，其中简单的Trie效果很好.但是，有些用户使用多达150,000,000个订阅.在这种情况下，需要更有效的数据结构.因此，nanomsg使用的是Patricia trie的内存高效版本，而不是简单的trie.
有关更多详细信息，请参见文章.
统一缓冲区模型
ZeroMQ具有奇怪的双重缓冲行为.传出数据和传入数据都存储在TCP的tx/rx缓冲区中的消息队列和中.例如，这意味着如果要限制传出数据量，则必须同时设置ZMQ_SNDBUF和ZMQ_SNDHWM套接字选项.鉴于两者之间没有语义上的差异，nanomsg仅使用TCP(或等效的)缓冲区来存储数据.
可扩展性协议
最后，在哲学层面上，nanomsg旨在实现不同的可扩展性协议"，而不是成为通用的网络库.具体来说:
不同的协议完全分开，您不能将REQ套接字连接到SUB套接字或类似的套接字.
每个协议都体现了具有明确定义的先决条件的分布式算法(例如，在REQ/REP情况下服务必须是无状态的")和保证(如果REQ套接字保持活动状态，则将最终处理请求).
部分失败由协议处理，而不是由用户处理.实际上，它对用户是透明的.
协议的规范位于/rfc子目录中.
目标是通过IETF标准化协议.
没有通用的类似UDP的套接字(ZMQ_ROUTER)，您应该使用L4协议来实现这种功能.
I am seeking Python 2.7 alternatives to ZeroMQ that are released under the BSD or MIT license. I am looking for something that supports request-reply and pub-sub messaging patterns. I can serialize the data myself if necessary. I found Twisted from Twisted Matrix Labs but it appears to require a blocking event loop, i.e. reactor.run(). I need a library that will run in the background and let my application check messages upon certain events. Are there any other alternatives?
Give nanomsg, a ZeroMQ younger sister, a try - same father, same beauty
- Yes, it is licensed under MIT/X11 license.
- Yes,
REQ/REP- allows to build clusters of stateless services to process user requests - Yes,
PUB/SUB- distributes messages to large sets of interested subscribers - Has several Python bindings available
https://github.com/tonysimpson/nanomsg-python (recommended)
https://github.com/sdiehl/pynanomsg
Differences between nanomsg and ZeroMQ
( state as of 2014/11 v0.5-beta - courtesy nanomsg.org >>> a-click-thru to the original HyperDoc )
Licensing
nanomsg library is MIT-licensed. What it means is that, unlike with ZeroMQ, you can modify the source code and re-release it under a different license, as a proprietary product, etc. More reasoning about the licensing can be found here.
POSIX Compliance
ZeroMQ API, while modeled on BSD socket API, doesn't match the API fully. nanomsg aims for full POSIX compliance.
Sockets are represented as ints, not void pointers.
Contexts, as known in ZeroMQ, don't exist in nanomsg. This means simpler API (sockets can be created in a single step) as well as the possibility of using the library for communication between different modules in a single process (think of plugins implemented in different languages speaking each to another). More discussion can be found here.
Sending and receiving functions ( nn_send, nn_sendmsg, nn_recv and nn_recvmsg ) fully match POSIX syntax and semantics.
Implementation Language
The library is implemented in C instead of C++.
From user's point of view it means that there's no dependency on C++ runtime (libstdc++ or similar) which may be handy in constrained and embedded environments. From nanomsg developer's point of view it makes life easier. Number of memory allocations is drastically reduced as intrusive containers are used instead of C++ STL containers. The above also means less memory fragmentation, less cache misses, etc. More discussion on the C vs. C++ topic can be found here and here.
Pluggable Transports and Protocols
In ZeroMQ there was no formal API for plugging in new transports (think WebSockets, DCCP, SCTP) and new protocols (counterparts to REQ/REP, PUB/SUB, etc.) As a consequence there were no new transports added since 2008. No new protocols were implemented either. The formal internal transport API (see transport.h and protocol.h) are meant to mitigate the problem and serve as a base for creating and experimenting with new transports and protocols.
Please, be aware that the two APIs are still new and may experience some tweaking in the future to make them usable in wide variety of scenarios.
nanomsg implements a new SURVEY protocol. The idea is to send a message ("survey") to multiple peers and wait for responses from all of them. For more details check the article here. Also look here.
In financial services it is quite common to use "deliver messages from anyone to everyone else" kind of messaging. To address this use case, there's a new BUS protocol implemented in nanomsg. Check the details here.
Threading Model
One of the big architectural blunders I've done in ZeroMQ is its threading model. Each individual object is managed exclusively by a single thread. That works well for async objects handled by worker threads, however, it becomes a trouble for objects managed by user threads. The thread may be used to do unrelated work for arbitrary time span, e.g. an hour, and during that time the object being managed by it is completely stuck. Some unfortunate consequences are: inability to implement request resending in REQ/REP protocol, PUB/SUB subscriptions not being applied while application is doing other work, and similar. In nanomsg the objects are not tightly bound to particular threads and thus these problems don't exist.
REQ socket in ZeroMQ cannot be really used in real-world environments, as they get stuck if message is lost due to service failure or similar. Users have to use XREQ instead and implement the request re-trying themselves. With nanomsg, the re-try functionality is built into REQ socket.
In nanomsg, both REQ and REP support cancelling the ongoing processing. Simply send a new request without waiting for a reply (in the case of REQ socket) or grab a new request without replying to the previous one (in the case of REP socket).
In ZeroMQ, due to its threading model, bind-first-then-connect-second scenario doesn't work for inproc transport. It is fixed in nanomsg.
For similar reasons auto-reconnect doesn't work for inproc transport in ZeroMQ. This problem is fixed in nanomsg as well.
Finally, nanomsg attempts to make nanomsg sockets thread-safe. While using a single socket from multiple threads in parallel is still discouraged, the way in which ZeroMQ sockets failed randomly in such circumstances proved to be painful and hard to debug.
State Machines
Internal interactions inside the nanomsg library are modeled as a set of state machines. The goal is to avoid the incomprehensible shutdown mechanism as seen in ZeroMQ and thus make the development of the library easier.
For more discussion see here and here.
IOCP Support
One of the long-standing problems in ZeroMQ was that internally it uses BSD socket API even on Windows platform where it is a second class citizen. Using IOCP instead, as appropriate, would require major rewrite of the codebase and thus, in spite of multiple attempts, was never implemented. IOCP is supposed to have better performance characteristics and, even more importantly, it allows to use additional transport mechanisms such as NamedPipes which are not accessible via BSD socket API. For these reasons nanomsg uses IOCP internally on Windows platforms.
Level-triggered Polling
One of the aspects of ZeroMQ that proved really confusing for users was the ability to integrate ZeroMQ sockets into an external event loops by using ZMQ_FD file descriptor. The main source of confusion was that the descriptor is edge-triggered, i.e. it signals only when there were no messages before and a new one arrived. nanomsg uses level-triggered file descriptors instead that simply signal when there's a message available irrespective of whether it was available in the past.
Routing Priorities
nanomsg implements priorities for outbound traffic. You may decide that messages are to be routed to a particular destination in preference, and fall back to an alternative destination only if the primary one is not available.
For more discussion see here.
TCP Transport Enhancements
There's a minor enhancement to TCP transport. When connecting, you can optionally specify the local interface to use for the connection, like this:
nn_connect (s, "tcp://eth0;192.168.0.111:5555").
Asynchronous DNS
DNS queries (e.g. converting hostnames to IP addresses) are done in asynchronous manner. In ZeroMQ such queries were done synchronously, which meant that when DNS was unavailable, the whole library, including the sockets that haven't used DNS, just hung.
Zero-Copy
While ZeroMQ offers a "zero-copy" API, it's not true zero-copy. Rather it's "zero-copy till the message gets to the kernel boundary". From that point on data is copied as with standard TCP. nanomsg, on the other hand, aims at supporting true zero-copy mechanisms such as RDMA (CPU bypass, direct memory-to-memory copying) and shmem (transfer of data between processes on the same box by using shared memory). The API entry points for zero-copy messaging are nn_allocmsg and nn_freemsg functions in combination with NN_MSG option passed to send/recv functions.
Efficient Subscription Matching
In ZeroMQ, simple tries are used to store and match PUB/SUB subscriptions. The subscription mechanism was intended for up to 10,000 subscriptions where simple trie works well. However, there are users who use as much as 150,000,000 subscriptions. In such cases there's a need for a more efficient data structure. Thus, nanomsg uses memory-efficient version of Patricia trie instead of simple trie.
For more details check this article.
Unified Buffer Model
ZeroMQ has a strange double-buffering behaviour. Both the outgoing and incoming data is stored in a message queue and in TCP's tx/rx buffers. What it means, for example, is that if you want to limit the amount of outgoing data, you have to set both ZMQ_SNDBUF and ZMQ_SNDHWM socket options. Given that there's no semantic difference between the two, nanomsg uses only TCP's (or equivalent's) buffers to store the data.
Scalability Protocols
Finally, on philosophical level, nanomsg aims at implementing different "scalability protocols" rather than being a generic networking library. Specifically:
Different protocols are fully separated, you cannot connect REQ socket to SUB socket or similar.
Each protocol embodies a distributed algorithm with well-defined prerequisites (e.g. "the service has to be stateless" in case of REQ/REP) and guarantees (if REQ socket stays alive request will be ultimately processed).
Partial failure is handled by the protocol, not by the user. In fact, it is transparent to the user.
The specifications of the protocols are in /rfc subdirectory.
The goal is to standardise the protocols via IETF.
There's no generic UDP-like socket (ZMQ_ROUTER), you should use L4 protocols for that kind of functionality.
这篇关于是否有BSD或MIT许可下发布的ZeroMQ的Python 2.7替代产品?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
