是否可以使用 AVRO 为 Kafka 中的 Google Pub/Sub 主题定义模式? [英] Is it possible to define a schema for Google Pub/Sub topics like in Kafka with AVRO?

问题描述

据我所知，我们可以在 Kafka 上定义 AVRO 模式，使用该模式定义的主题将只接受与该模式匹配的数据.在接受进入队列之前验证数据结构非常有用.

As far as I know, we can define AVRO schemas on Kafka and the topic defined with this schema will only accept the data matching with that schema. It's really useful to validate data structure before accepting into the queue.

Google Pub/Sub 中有没有类似的东西?

Is there anything similar in Google Pub/Sub?

推荐答案

Kafka 本身不验证模式，因此主题本身没有模式，除了一对字节数组和一些元数据.序列化程序是生产客户端的一部分，在数据到达主题之前执行验证.同样，在 PubSub 中，归根结底，它只是存储/发送 byte[] 数据.

Kafka itself is not validating a schema, and topics therefore do not inherently have schemas other than a pair of byte arrays plus some metadata. It's the serializer that's part of the producing client that performs the validation before the data reaches the topic. Similarly in PubSub, at the end of the day, it is only storing/sending byte[] data.

因此，理论上，在通过 PubSub 移动的数据的任一端使用类似于 Confluent Avro Schema Registry 的东西是完全可行的.Google 不提供这样的功能 AFAIK，因此您需要重新创建可以执行 Avro 兼容性检查的服务，并在该服务的客户端周围绑定一个 PubSub 序列化+生产者客户端.例如，您可以首先将 Registry 本身作为 GKE 中的容器运行.

Therefore, in theory, it's completely feasible to use something similar to the Confluent Avro Schema Registry on either end of the data that moves through PubSub. Google does not offer such a feature, AFAIK, so you would need to recreate said service that can perform your Avro compatibility checks, plus tie a PubSub serialization+producer client around that service's client. For instance, you can run the Registry itself as a container in GKE to begin with.

可能想查看 Google Cloud Pub-Sub 的 Avro 消息?

这篇关于是否可以使用 AVRO 为 Kafka 中的 Google Pub/Sub 主题定义模式?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！