我正在评估Google Pub/Sub与Kafka.有什么区别? [英] I am evaluating Google Pub/Sub vs Kafka. What are the differences?

问题描述

我在kafka上工作不多，但是想在GCE中建立数据管道.所以我们想知道Kafka vs PUB/Sub.基本上我想知道在Kafka和Pub/sub

I have not worked on kafka much but wanted to build data pipeline in GCE. So we wanted to know Kafka vs PUB/Sub. Basically I want to know how message consistency, message availability, message reliability is maintained in both Kafka and Pub/sub

谢谢

推荐答案

除了由Google管理的Google Pub/Sub和开放源代码的Kafka之外，另一个区别是Google Pub/Sub是消息队列(例如Rabbit)MQ)，其中Kafka更像是流日志.您不能使用Pubsub来重读"或重播"消息.(编辑-从2019年2月开始，您可以重播消息并向后追溯到某个时间戳，请参见以下评论)

In addition to Google Pub/Sub being managed by Google and Kafka being open source, the other difference is that Google Pub/Sub is a message queue (e.g. Rabbit MQ) where as Kafka is more of a streaming log. You can't "re-read" or "replay" messages with Pubsub. (EDIT - as of 2019 Feb, you CAN replay messages and seek backwards in time to a certain timestamp, per comment below)

使用Google Pub/Sub，一旦从订阅中读取并确认了一条消息，该消息就消失了.为了使消息的更多副本可供不同的读者阅读，您可以通过为该主题创建订阅"来散开"该主题，其中每个订阅都将包含该主题中所有内容的完整副本.但这也会增加成本，因为Google会根据从其中读取的数据量来收取发布/订阅使用费用.

With Google Pub/Sub, once a message is read out of a subscription and ACKed, it's gone. In order to have more copies of a message to be read by different readers, you "fan-out" the topic by creating "subscriptions" for that topic, where each subscription will have an entire copy of everything that goes into the topic. But this also increases cost because Google charges Pub/Sub usage by the amount of data read out of it.

使用Kafka，您可以设置保留期限(我认为默认为7天)，无论有多少消费者阅读该邮件，邮件都会保留在Kafka中.您可以添加新的使用者(又名订户)，并在需要时从主题的开头开始使用.您还可以将保留期限设置为无限，然后基本上可以将Kafka用作不可变的数据存储，如下所述: http://stackoverflow.com/a/22597637/304262

With Kafka, you set a retention period (I think it's 7 days by default) and the messages stay in Kafka regardless of how many consumers read it. You can add a new consumer (aka subscriber), and have it start consuming from the front of the topic any time you want. You can also set the retention period to be infinite, and then you can basically use Kafka as an immutable datastore, as described here: http://stackoverflow.com/a/22597637/304262

Amazon AWS Kinesis是Kafka的托管版本，而我认为Google Pubsub是Rabbit MQ的托管版本.具有SQS的Amazon SNS也类似于Google Pubsub(SNS提供扇出，SQS提供排队).

Amazon AWS Kinesis is a managed version of Kafka whereas I think of Google Pubsub as a managed version of Rabbit MQ. Amazon SNS with SQS is also similar to Google Pubsub (SNS provides the fanout and SQS provides the queueing).

这篇关于我正在评估Google Pub/Sub与Kafka.有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！