ZeroMQ - 模拟多个客户端到一台服务器的标准套接字 [英] ZeroMQ - Emulating standard socket for multiple clients to one server

问题描述

我希望利用 ZeroMQ 来处理同时进入认证服务器的大量请求的排队.

我在 HPC 环境中，因此有大量计算节点都运行相同的启动程序，这些启动程序通过服务器进行验证.此服务器代码运行在前端节点上，在成功验证客户端后，将释放密钥以便客户端解密作业数据.

当前使用标准套接字.当客户端最初向服务器发送某些内容时，会使用 sys/socket.h 中的 accept() 生成一个新套接字.这允许客户端和服务器在认证过程中(校验和等)在彼此之间发送多条消息，如果成功则最终返回密钥.

ZeroMQ 的问题在于不需要附加命令，因此不会为该特定证明创建辅助套接字.来自所有客户端的所有消息都按照它们进来的顺序进行处理，导致多部分证明过程不起作用.我花了很长时间阅读指南并在谷歌上搜索以尝试找到类似的解决方案，但到目前为止还没有任何运气.

有没有一种方法可以利用 ZeroMQ 在此应用程序中提供与标准套接字相同的行为?

解决方案

Q :有没有方法我可以利用 ZeroMQ 在这个应用程序中提供相同的行为作为标准插座?

ZeroMQ 是一个 非常智能，而是面向行为分布式系统，不是套接字.

鉴于您的意图是对 HPC 生态系统有价值，假设/定向使用某种工具的解决方案，并且必须尽可能地弯曲它，以便它变得接近类似于原生行为，但是对于其他工具，似乎不是典型的 HPC 级方法.

HPC 代码通常经过精心设计，因此具有计算成本效益(并祝福所有那些今天被允许不设计 HPC 代码的人的老板、首席财务官和政府/军用资金)为了最终的性能和硬件资源的使用效率 :o) ) - 在这里，如果一个人在 ZeroMQ 实例化上支付费用，那么这些非零的实例化成本和获得只是"一个套接字似乎没有任何好处-类似的行为，以成本为代价，具有负面的性能收益，没有对智能、集群范围的 ZeroMQ 服务(无论是 N+1 或 N+M 冗余、低延迟智能节点间集群信令、密码学)的一些未来收益进行任何调整、廉价的安全驱动的白名单，或任何可能代表任何额外 HPC 级项目的好处的东西，这可能证明初始 ZeroMQ 实例化的成本是合理的).

<小时>

ZMQ_STREAM 的定义原型可能会提供一些工具，但是，参考.以上

<块引用>

ZMQ_STREAM 类型的套接字用于在使用 tcp:// 运输.ZMQ_STREAM 套接字可以充当客户端和/或服务器，异步发送和/或接收 TCP 数据.

在接收 TCP 数据时，ZMQ_STREAM 套接字应在将消息传递给应用程序之前，在消息之前添加一个包含原始对等体身份的消息部分.收到的消息在所有连接的对等点之间进行公平排队.

当发送 TCP 数据时，ZMQ_STREAM 套接字应删除消息的第一部分并使用它来确定消息应路由到的对等方的身份，不可路由的消息将导致 EHOSTUNREACH 或 EAGAIN 错误.

要打开到服务器的连接，请使用 zmq_connect 调用，然后使用 ZMQ_IDENTITY 获取套接字标识zmq_getsockopt 调用.

要关闭特定连接，请发送身份帧，后跟零长度消息(参见示例部分).

建立连接后，应用程序将收到一条零长度消息.类似地，当对端断开连接(或连接丢失)时，应用程序将收到一条零长度的消息.

您必须先发送一个身份帧，然后再发送一个数据帧.标识帧需要 ZMQ_SNDMORE 标志，但在数据帧上被忽略.

ZMQ_STREAM 示例:

void *ctx = zmq_ctx_new();assert (ctx && "上下文实例化失败..." );void *socket = zmq_socket (ctx, ZMQ_STREAM);assert (socket && "socket 实例化失败..." );int rc = zmq_bind (socket, "tcp://*:8080");assert (rc == 0 && "socket.bind() 失败...");uint8_t id [256];/* 保存 ZMQ_STREAM ID 的数据结构 */size_t id_size = 256;uint8_t 原始 [256];/* 保存 ZMQ_STREAM 接收数据的数据结构 */size_t raw_size = 256;而 (1) {id_size = zmq_recv (socket, id, 256, 0);assert (id_size > 0 && "Get HTTP request; ID frame and then request; Failed..." )做 {raw_size = zmq_recv (socket, raw, 256, 0);断言 (raw_size >= 0 && "socket.recv() 失败...");} while (raw_size == 256);char http_response [] =/* 准备响应 */"HTTP/1.0 200 正常\r\n""内容类型:文本/纯文本\r\n""\r\n"你好，世界！";zmq_send (socket, id, id_size, ZMQ_SNDMORE);/* 发送 ID 帧，然后是响应 */zmq_send (socket, http_response, strlen (http_response), 0);zmq_send (socket, id, id_size, ZMQ_SNDMORE);/* 通过发送 ID 帧后跟零响应来关闭连接 */zmq_send (socket, 0, 0, 0);}zmq_close(套接字)；zmq_ctx_destroy (ctx);

<小时>

ZeroMQ zmq_getsockopt() 可以提供 POSIX/SOCKET 描述符，用于低级技巧

<块引用>

ZMQ_FD 选项应检索与指定套接字关联的文件描述符.返回的文件描述符可用于将套接字集成到现有的事件循环中；ØMQ 库应通过使文件描述符准备好读取来以边缘触发的方式向套接字上的任何未决事件发出信号.

从返回的文件描述符中读取的能力并不一定表示可以从底层套接字读取或写入消息；应用程序必须通过随后检索 ZMQ_EVENTS 选项来检索实际事件状态.

zmq_send 和 zmq_recv 函数也在内部使用返回的文件描述符.由于描述符是边缘触发的，应用程序必须在每次调用zmq_send或zmq_recv后更新ZMQ_EVENTS的状态.

更明确地说:在调用 zmq_send 之后，套接字可能变得可读(反之亦然)，而不会触发文件描述符上的读取事件.

返回的文件描述符旨在用于poll 或类似的系统调用仅.应用程序绝不能尝试直接读取或写入数据，也不应该尝试关闭它.

选项值类型:POSIX 系统上的 int，Windows 上的 SOCKET

<小时>

有关更多ZeroMQ 技巧的详细信息，可以阅读解决方案、性能基准、减少延迟的细节以及此处已经讨论过的其他问题解决技巧.

I am hoping to utilise ZeroMQ in order to handle queuing of a large number of requests coming in at the same time to an attestation server.

I am in an HPC environment and so have a large number of compute nodes all running the same launcher program that attests with a server. This server code runs on a front-end node and after successfully attesting the client, will then release a key in order for the client to decrypt job data.

Currently standard sockets are used. When a client initially sends something to the server a new socket is spawned using accept() from sys/socket.h. This allows the client and server to send multiple messages between each other in the attestation process (checksums etc) before finally returning the key if successful.

The issue with ZeroMQ is that the attach command is not necessary and so a secondary socket for that specific attestation is not created. All the messages are dealt with from all the clients in whatever order they come in, leading to the multi-part attestation process not working. I have spent ages going through the guide and googling to try and find a similar solution but have not had any luck so far.

Is there a way I can utilise ZeroMQ to give the same behaviour in this application as a standard socket?

解决方案

Q : Is there a way I can utilise ZeroMQ to give the same behaviour in this application as a standard socket?

ZeroMQ is a very smart and rather behaviour-oriented signaling / messaging-platform for distributed-systems, not a socket.

Given your intentions are to be worth for HPC-ecosystem, a solution postulated / directed to use some tool and a must to bend it as much as possible, so as it will become close to resemble a behaviour that is native, but for other tool, does not seem to be a typical HPC-grade approach.

HPC-code is typically very well-crafted so as to become computing-costs-efficient ( and bless the Boss, CFO & gov/mil-funding for all those, who are today permitted not to design the HPC-code for the ultimate performance and hardware-resources' use efficiency :o) ) - here, if one pays expenses on ZeroMQ instantiations, there seems no benefit to come from these non-zero costs of instantiations and getting "just"-a-socket-alike behaviour, at cost, has negative performance yield, without any adjustments in some future benefits from smart, cluster-wide ZeroMQ services ( be it an N+1 or N+M redundancy, low-latency smart inter-node cluster signaling, cryptography, cheap security-motivated white-listing, or anything that may represent any additional HPC-grade Project's benefit, that may justify the costs of the initial ZeroMQ instantiation ).

---

Defined archetype of ZMQ_STREAM may provide some tools, yet, ref. above

A socket of type ZMQ_STREAM is used to send and receive TCP data from a non-ØMQ peer, when using the tcp:// transport. A ZMQ_STREAM socket can act as client and/or server, sending and/or receiving TCP data asynchronously.

When receiving TCP data, a ZMQ_STREAM socket shall prepend a message part containing the identity of the originating peer to the message before passing it to the application. Messages received are fair-queued from among all connected peers.

When sending TCP data, a ZMQ_STREAM socket shall remove the first part of the message and use it to determine the identity of the peer the message shall be routed to, and unroutable messages shall cause an EHOSTUNREACH or EAGAIN error.

To open a connection to a server, use the zmq_connect call, and then fetch the socket identity using the ZMQ_IDENTITY zmq_getsockopt call.

To close a specific connection, send the identity frame followed by a zero-length message (see EXAMPLE section).

When a connection is made, a zero-length message will be received by the application. Similarly, when the peer disconnects (or the connection is lost), a zero-length message will be received by the application.

You must send one identity frame followed by one data frame. The ZMQ_SNDMORE flag is required for identity frames but is ignored on data frames.

ZMQ_STREAM Example:

void *ctx = zmq_ctx_new ();                          assert (ctx     && "Context Instantiation Failed..." );
void *socket = zmq_socket (ctx, ZMQ_STREAM);         assert (socket  && "socket Instantiation Failed..." );
int rc = zmq_bind (socket, "tcp://*:8080");          assert (rc == 0 && "socket.bind() Failed..." );
uint8_t id [256];                                 /* Data structure to hold the ZMQ_STREAM ID */
size_t id_size = 256;
uint8_t raw [256];                                /* Data structure to hold the ZMQ_STREAM received data */
size_t raw_size = 256;
while (1) {
   id_size = zmq_recv (socket, id, 256, 0);          assert (id_size > 0 && "Get HTTP request; ID frame and then request; Failed..." )
   do {
        raw_size = zmq_recv (socket, raw, 256, 0);   assert (raw_size >= 0 && "socket.recv() Failed..." );
   } while (raw_size == 256);

   char http_response [] =                        /* Prepares the response */
            "HTTP/1.0 200 OK\r\n"
            "Content-Type: text/plain\r\n"
            "\r\n"
            "Hello, World!";
   zmq_send (socket, id, id_size, ZMQ_SNDMORE);   /* Sends the ID frame followed by the response */
   zmq_send (socket, http_response, strlen (http_response), 0);
   zmq_send (socket, id, id_size, ZMQ_SNDMORE);   /* Closes the connection by sending the ID frame followed by a zero response */
   zmq_send (socket, 0, 0, 0);
}
zmq_close (socket);
zmq_ctx_destroy (ctx);

---

ZeroMQ zmq_getsockopt() can deliver a POSIX/SOCKET descriptor, for low-level tricks

The ZMQ_FD option shall retrieve the file descriptor associated with the specified socket. The returned file descriptor can be used to integrate the socket into an existing event loop; the ØMQ library shall signal any pending events on the socket in an edge-triggered fashion by making the file descriptor become ready for reading.

The ability to read from the returned file descriptor does not necessarily indicate that messages are available to be read from, or can be written to, the underlying socket; applications must retrieve the actual event state with a subsequent retrieval of the ZMQ_EVENTS option.

The returned file descriptor is also used internally by the zmq_send and zmq_recv functions. As the descriptor is edge triggered, applications must update the state of ZMQ_EVENTS after each invocation of zmq_send or zmq_recv.

To be more explicit: after calling zmq_send the socket may become readable (and vice versa) without triggering a read event on the file descriptor.

The returned file descriptor is intended for use with a poll or similar system call only. Applications must never attempt to read or write data to it directly, neither should they try to close it.

Option value type: int on POSIX systems, SOCKET on Windows

---

For more details on ZeroMQ tricks one may enjoy to read solutions, performance benchmarks, latency-shaving details and other problem-solving tricks that have already been discussed here.

这篇关于ZeroMQ - 模拟多个客户端到一台服务器的标准套接字的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！