我如何使用 ZeroMQ 为协议缓冲区编写自己的 RPC 实现 [英] How can i write my own RPC Implementation for Protocol Buffers utilizing ZeroMQ

问题描述

根据定义服务"下的 Google Protocol Buffers 文档他们说，

<块引用>

也可以在您自己的 RPC 实现中使用协议缓冲区.

据我了解，Protocol Buffers 本身并没有实现 RPC.相反，它们提供了一系列必须由用户实现的抽象接口(就是我！).所以我想利用 ZeroMQ 实现这些抽象接口进行网络通信.

我正在尝试使用 ZeroMQ 创建一个 RPC 实现，因为我正在处理的项目已经实现了用于基本消息传递的 ZeroMQ(因此我不使用 gRPC，正如文档所建议的那样).

在通读 proto 文档后，我发现我必须实现抽象接口 RpcChannel 和 RpcController 用于我自己的实现.

我已经构建了一个最小化的示例，说明我目前的 RPC 实现

.proto 文件:为简洁起见省略了 SearchRequest 和 SearchResponse 架构

service SearchService {rpc 搜索(SearchRequest)返回(SearchResponse)；}

SearchServiceImpl.h:

class SearchServiceImpl : public SearchService {上市:无效搜索(谷歌::protobuf::RpcController *控制器，const SearchRequest *请求，搜索响应 *响应，google::protobuf::Closure *done) 覆盖 {//处理请求并获取结果的静态函数SearchResponse res = GetSearchResult(request);//调用回调函数如果(完成！= NULL){完成->运行()；}}}};

MyRPCController.h:

class MyRPCController : public google::protobuf::RpcController {上市:MyRPCController();void Reset() 覆盖；bool Failed() 常量覆盖；std::string ErrorText() 常量覆盖；void StartCancel() 覆盖；void SetFailed(const std::string &reason) 覆盖；bool IsCanceled() 常量覆盖；void NotifyOnCancel(google::protobuf::Closure *callback) 覆盖；私人的:布尔失败_；std::string message_;};

MyRPCController.cpp - 基于 这个

void MyRPCController::Reset() { failed_ = false;}bool MyRPCController::Failed() const { return failed_;}std::string MyRPCController::ErrorText() const { return message_;}void MyRPCController::StartCancel() { }void MyRPCController::SetFailed(const std::string &reason) {失败_ = 真；消息_ = 原因；}bool MyRPCController::IsCanceled() const { return false;}void MyRPCController::NotifyOnCancel(google::protobuf::Closure *callback) { }MyRPCController::ChiRpcController() : RpcController() { Reset();}

MyRpcChannel.h:

class MyRPCChannel: public google::protobuf::RpcChannel {上市:void CallMethod(const google::protobuf::MethodDescriptor *method, google::protobuf::RpcController *controller,const google::protobuf::Message *request, google::protobuf::Message *response,google::protobuf::Closure *done) 覆盖；};

到目前为止，我对示例的疑问:

• 我在哪里可以将 ZeroMQ 放入其中?
  • 它似乎应该进入 RPCChannel，因为在我看到的示例中(参见第三个代码块 此处)，它们传递一个字符串，该字符串具有要绑定的端口(即 MyRpcChannel channel("rpc:hostname:1234/myservice");)
• 我关心的是我的 RPCController 实现，它看起来太简单了.应该更多的去这里吗?
• 我如何实现 RPCChannel，它看起来与 SearchServiceImpl 非常相似.这些类中的 1 虚函数具有非常相似的方法签名，只是它是通用的.

以下是我遇到的其他一些 Stack Overflow 问题，其中包含有关该主题的一些有用信息:

1. Protobuf-Net:实现服务器、rpc 控制器和 rpcchannel - 这是我找到 RPCController 实现示例的地方.
2. 使用协议缓冲区在 ZeroMQ 中实现 RPC -这个答案很有趣，因为在最佳答案中，似乎他们建议不要为 .proto 文件使用内置 RPC 格式的 Protobufs.
   • 我也在这个文件中注意到了同样的概念，在名为 libpbrpc 的存储库中，这似乎是示例代码的好来源
3. 我可以/应该使用诸如 RPCZ 之类的现有实现吗?

感谢您的帮助.我希望我提供了足够的信息并且清楚我在寻找什么.如果有什么不清楚或缺乏信息，请告诉我.我很乐意相应地编辑问题.

解决方案

• ZeroMQ 为基于可以包含任何数据的消息的网络通信提供低级 API.
• ProtoBuffers 是一个库，可将结构化数据编码为压缩的二进制数据并解码此类数据.
• gRPC 是一个 RPC 框架，它为基于网络通信的 RPC 服务生成代码，并具有将数据作为 ProtoBuffers 数据交换的功能.

ZeroMQ 和 gRPC 都支持网络通信，但方式不同.您必须选择 ZeroMQ 或 gRPC 进行网络通信.如果您选择 ZeroMQ，则可以使用 ProtoBuffers 交换二进制结构化数据对消息进行编码.

重点是 ProtoBuffers 库允许对变体记录(类似于 C/C++ 联合)进行编码和解码，可以完全模拟 RPC 服务提供的功能，这些服务具有交换 ProtoBuffers 消息的功能.

所以选项是:

1. 将 ZeroMQ 与发送和接收原语以及 ProtoBuffers 编码的可变消息一起使用，这些消息可以包含各种子消息，例如

<块引用>

联合请求{字节 msgType;消息类型 1 msg1;MessageType2 msg2;MessageType3 msg3;}工会反应{字节 msgType;MessageType3 msg1;MessageType4 msg2;MessageType5 msg3;}发送(请求请求)；接收(响应响应)；

2. 使用 gRPC 生成具有函数的服务，例如

<块引用>

service MyService{rpc function1(MessageType1) 返回 (Response);rpc function2(MessageType2) 返回 (Response);rpc function3(MessageType3) 返回 (Response);rpc 函数N(MessageType3) 返回(MessageType5)；}

(这里可以使用多种组合)

3. 仅使用单一功能的 gRPC 服务，例如

<块引用>

service MyService{rpc 函数(请求)返回(响应)；}

该选项可能取决于

• 客户端的首选目标:基于 ZeroMQ 或 gRPC 的客户端
• 比较 ZeroMQ 与基于 gRPC 的服务的性能原因
• 特定功能，例如在 ZeroMQ 与基于 gRPC 的服务和客户端中如何使用/处理订阅(参见 如何在grpc中正确设计发布订阅模式?)

对于第一个选项，与第二个选项相比，您必须做很多事情.您必须将发送的消息类型与要接收的预期消息类型相匹配.

如果其他人将开发客户端，则第二个选项将允许更容易/更快地了解所提供服务的功能.

为了在 ZeroMQ 上开发 RPC 服务，我会定义这样的 .proto 文件，指定函数、参数(所有可能的输入和输出参数)和如下错误:

enum 函数{F1 = 0;F2 = 1;F3 = 2;}枚举错误{E1 = 0;E2 = 1;E3 = 2;}留言请求{所需的功能函数= 1;重复输入数据=2；}消息 响应{所需的功能函数= 1;必需的错误错误= 2;重复输出数据=3；}消息输入{可选 Input1 data1 = 1;可选的 Input2 data2 = 2;...可选 InputN dataN = n;}消息输出{可选输出 1 数据 1 = 1；可选输出 2 数据 2 = 2；...可选输出N dataN = n;}留言 留言{重复的请求请求；重复响应响应；}

并且根据函数 id，在运行时必须检查参数的数量和类型.

According to the Google Protocol Buffers documentation under 'Defining Services' they say,

it's also possible to use protocol buffers with your own RPC implementation.

To my understanding, Protocol Buffers does not implement RPC natively. Instead, they provide a series of abstract interfaces that must be implemented by the user (Thats me!). So I want to implement these abstract interfaces utilizing ZeroMQ for network communication.

I'm trying to create an RPC implementation using ZeroMQ because the project i'm working on already implements ZeroMQ for basic messaging (Hence why I'm not using gRPC, as the documentation recommends).

After reading through the proto documentation thoroughly, i found that I have to implement the abstract interfaces RpcChannel and RpcController for my own implementation.

I've constructed a minimalized example of where I'm currently at with my RPC Implementation

.proto file: Omitted SearchRequest and SearchResponse schema for brevity

service SearchService {
    rpc Search (SearchRequest) returns (SearchResponse);
}

SearchServiceImpl.h:

class SearchServiceImpl : public SearchService {
 public:
  void Search(google::protobuf::RpcController *controller,
                    const SearchRequest *request,
                    SearchResponse *response,
                    google::protobuf::Closure *done) override {
    // Static function that processes the request and gets the result
    SearchResponse res = GetSearchResult(request);

    // Call the callback function
    if (done != NULL) {
    done->Run();
    }
    }
  }
};

MyRPCController.h:

class MyRPCController : public google::protobuf::RpcController {
 public:
    MyRPCController();

    void Reset() override;

    bool Failed() const override;

    std::string ErrorText() const override;

    void StartCancel() override;

    void SetFailed(const std::string &reason) override;

    bool IsCanceled() const override;

    void NotifyOnCancel(google::protobuf::Closure *callback) override;
 private:
  bool failed_;
  std::string message_;
};

MyRPCController.cpp - Based off of this

void MyRPCController::Reset() {  failed_ = false; }

bool MyRPCController::Failed() const { return failed_; }

std::string MyRPCController::ErrorText() const { return message_; }

void MyRPCController::StartCancel() { }

void MyRPCController::SetFailed(const std::string &reason) {
  failed_ = true;
  message_ = reason;
}

bool MyRPCController::IsCanceled() const { return false; }

void MyRPCController::NotifyOnCancel(google::protobuf::Closure *callback) { }

MyRPCController::ChiRpcController() : RpcController() { Reset(); }

MyRpcChannel.h:

class MyRPCChannel: public google::protobuf::RpcChannel {
 public:
    void CallMethod(const google::protobuf::MethodDescriptor *method, google::protobuf::RpcController *controller,
                    const google::protobuf::Message *request, google::protobuf::Message *response,
                    google::protobuf::Closure *done) override;
};

Questions I have with my example thus far:

• Where do I fit ZeroMQ into this?
  • It seems like it should be going into RPCChannel, because in the examples i see (See 3rd code block here), they pass a string that has the ports to bind to (i.e. MyRpcChannel channel("rpc:hostname:1234/myservice");)
• I'm concerned with my RPCController implementation, it seems too simple. Should more be going here?
• How do i implement RPCChannel, it seems very similar to the SearchServiceImpl. The 1 virtual function in these classes has a very similar method signature, except it's generic.

Here's some other Stack Overflow questions I came across that had some helpful information on the topic:

1. Protobuf-Net: implementing server, rpc controller and rpc channel - This is where i found the example for the RPCController implementation.
2. Using Protocol Buffers for implementing RPC in ZeroMQ - This answer is interesting because in the top answer, is seems that they're recommending against using Protobufs built in RPC formatting for the .proto file.
   • I also noticed this same notion in this file, in a repository called libpbrpc which seemed like a good source for example code
3. Can I/Should I be using an existing implementation such as RPCZ?

Thank you for your help. I hope I gave enough information and was clear in what I'm looking for. Please let me know if something is unclear or lacking in information. I'd be happy to edit the question accordingly.

解决方案

• ZeroMQ provides a low-level API for network communication based on messages that can contain any data.
• ProtoBuffers is a library that encodes structured data as compressed binary data and decodes such data.
• gRPC is a RPC framework that generates code for network communication based RPC services with functions that exchange data as ProtoBuffers data.

Both ZeroMQ and gRPC provides support for network communication but in different ways. You have to chose either ZeroMQ, either gRPC for network communication. If you choose ZeroMQ, messages can be encoded using ProtoBuffers exchanging binary structured data.

The main point is ProtoBuffers library allows variant records (similar to C/C++ unions) to be encoded and decoded that can fully emulate the functionality provided by RPC services having functions exchanging ProtoBuffers messages.

So the options are:

1. Use ZeroMQ with send and receive primitives and ProtoBuffers encoded variant messages that can contain various sub-messages, like

union Request
{
  byte msgType;
  MessageType1 msg1;
  MessageType2 msg2;
  MessageType3 msg3;
}

union Response
{
  byte msgType;
  MessageType3 msg1;
  MessageType4 msg2;
  MessageType5 msg3;
}

send(Request request);
receive(Response response);

2. Use gRPC generating a service with functions, like

service MyService 
{
  rpc function1(MessageType1) returns (Response);
  rpc function2(MessageType2) returns (Response);
  rpc function3(MessageType3) returns (Response);

  rpc functionN(MessageType3) returns (MessageType5);
}

(here it's possible to use many many combinations)

3. Use just a single-function gRPC service, like

service MyService 
{
    rpc function(Request) returns (Response);
}

The option could depend on

• preferred target for client: ZeroMQ or gRPC based client
• performance reasons comparing ZeroMQ vs gRPC based service
• specific features like how subscription is used/handled in ZeroMQ vs gRPC based service and client (see How to design publish-subscribe pattern properly in grpc?)

For the 1st option, you have to do a lot of stuff comparing to 2nd option. You have to match the type of message sent with the types of expected messages to be received.

The 2nd option would allow an easier/faster understanding of functionality of the service provided if the somebody else will develop the client.

For developing a RPC service on top on ZeroMQ I would define such .proto file specifying the functions, parameters (all possible input and output parameters) and errors like this:

enum Function 
{
    F1 = 0;
    F2 = 1;
    F3 = 2;
}

enum Error 
{
    E1 = 0;
    E2 = 1;
    E3 = 2;
}

message Request
{ 
    required Function function = 1;
    repeated Input data = 2;
}

message Response
{ 
    required Function function = 1;
    required Error error = 2;
    repeated Output data = 3;
}

message Input
{ 
    optional Input1 data1 = 1;
    optional Input2 data2 = 2;
    ...
    optional InputN dataN = n;
}

message Output
{ 
    optional Output1 data1 = 1;
    optional Output2 data2 = 2;
    ...
    optional OutputN dataN = n;
}

message Message
{
   repeated Request requests;
   repeated Response responses;
}

and depending on function id, at run-time the number and the types of parameters have to be checked.

这篇关于我如何使用 ZeroMQ 为协议缓冲区编写自己的 RPC 实现的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！