我正在评估 Google Pub/Sub 与 Kafka.有什么区别? [英] I am evaluating Google Pub/Sub vs Kafka. What are the differences?

问题描述

我对 kafka 的工作不多，但想在 GCE 中构建数据管道.所以我们想了解 Kafka 与 PUB/Sub.基本上我想知道在Kafka和Pub/sub中如何维护消息一致性、消息可用性、消息可靠性

I have not worked on kafka much but wanted to build data pipeline in GCE. So we wanted to know Kafka vs PUB/Sub. Basically I want to know how message consistency, message availability, message reliability is maintained in both Kafka and Pub/sub

谢谢

推荐答案

除了 Google Pub/Sub 由 Google 管理和 Kafka 是开源的，另外一个区别是 Google Pub/Sub 是一个消息队列(例如 RabbitMQ)，而 Kafka 更像是一个流日志.您无法使用 Pubsub重新阅读"或重播"消息.(编辑 - 截至 2019 年 2 月，您可以根据下面的评论重播消息并及时向后寻找某个时间戳)

In addition to Google Pub/Sub being managed by Google and Kafka being open source, the other difference is that Google Pub/Sub is a message queue (e.g. Rabbit MQ) where as Kafka is more of a streaming log. You can't "re-read" or "replay" messages with Pubsub. (EDIT - as of 2019 Feb, you CAN replay messages and seek backwards in time to a certain timestamp, per comment below)

使用 Google Pub/Sub，一旦从订阅中读出消息并确认消息，它就会消失.为了让不同的读者阅读更多的消息副本，您可以通过为该主题创建订阅"来扇出"该主题，其中每个订阅都将拥有进入该主题的所有内容的完整副本.但这也会增加成本，因为 Google 根据读取的数据量收取 Pub/Sub 使用费.

With Google Pub/Sub, once a message is read out of a subscription and ACKed, it's gone. In order to have more copies of a message to be read by different readers, you "fan-out" the topic by creating "subscriptions" for that topic, where each subscription will have an entire copy of everything that goes into the topic. But this also increases cost because Google charges Pub/Sub usage by the amount of data read out of it.

使用 Kafka，您可以设置保留期(我认为默认情况下是 7 天)并且无论有多少消费者阅读消息，消息都会保留在 Kafka 中.您可以添加一个新的消费者(又名订阅者)，并随时从主题的前面开始消费.您还可以将保留期设置为无限，然后基本上可以将 Kafka 用作不可变数据存储，如下所述:http:///stackoverflow.com/a/22597637/304262

With Kafka, you set a retention period (I think it's 7 days by default) and the messages stay in Kafka regardless of how many consumers read it. You can add a new consumer (aka subscriber), and have it start consuming from the front of the topic any time you want. You can also set the retention period to be infinite, and then you can basically use Kafka as an immutable datastore, as described here: http://stackoverflow.com/a/22597637/304262

Amazon AWS Kinesis 是 Kafka 的托管版本，而我认为 Google Pubsub 是 Rabbit MQ 的托管版本.带有 SQS 的 Amazon SNS 也类似于 Google Pubsub(SNS 提供扇出，SQS 提供排队).

Amazon AWS Kinesis is a managed version of Kafka whereas I think of Google Pubsub as a managed version of Rabbit MQ. Amazon SNS with SQS is also similar to Google Pubsub (SNS provides the fanout and SQS provides the queueing).

这篇关于我正在评估 Google Pub/Sub 与 Kafka.有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！