在微服务中用作消息总线时,kafka是可靠的 [英] is kafka reliable when used as a message bus in micro services

查看:608
本文介绍了在微服务中用作消息总线时,kafka是可靠的的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我将kafka用作微服务体系结构的消息总线,因此,多个服务在主题上侦听消息.因此,服务高度依赖于要直播的主题.

I am using kafka as a message bus for Micro Service architecture, hence multiple services listen on a topic for a message. Therefore, the services are highly dependent on the topic to be live.

但是,在很多情况下,我对主题的了解分别为leader not availablebroker not availableleader= - 1.

But, there are many instances where I get leader not available, broker not available and leader= - 1 on the topics.

现在,我不确定我是否可以依赖kafka主题,因为当平台上的问题与主题相关时,服务会中断.

Now, I am not sure if I can rely on the kafka topics, as services get interrupted when there are issues on the topics with cause issues in the platform.

有人可以阐明这些主题的可靠性和可靠性吗?如果能够解决上述问题,我们是否可以康复?

Can someone throw some light on the reliability and dependability on the topics and can we recover if we can across the above issues.

推荐答案

我将通过解释Kafka的总体工作方式以及如何处理故障来回答您的问题.

I'll answer to your question by explaining how Kafka works in general and how it deals with failures.

每个 topic ,都是特定的数据流(类似于数据库中的表).主题分为 partitions (任意多个),分区中的每条消息都会获得一个增量ID,称为偏移量,如下所示.

Every topic, is a particular stream of data (similar to a table in a database). Topics, are split into partitions (as many as you like) where each message within a partition gets an incremental id, known as offset as shown below.

分区0:

+---+---+---+-----+
| 0 | 1 | 2 | ... |
+---+---+---+-----+

分区1:

+---+---+---+---+----+
| 0 | 1 | 2 | 3 | .. |
+---+---+---+---+----+

现在,Kafka集群由多个经纪人组成.每个代理都有一个ID标识,并且可以包含某些主题分区.

Now a Kafka cluster is composed of multiple brokers. Each broker is identified with an ID and can contain certain topic partitions.

2个主题的示例(每个主题分别具有3个分区和2个分区):

Example of 2 topics (each having 3 and 2 partitions respectively):

经纪人1:

+-------------------+
|      Topic 1      |
|    Partition 0    |
|                   |
|                   |
|     Topic 2       |
|   Partition 1     |
+-------------------+

经纪人2:

+-------------------+
|      Topic 1      |
|    Partition 2    |
|                   |
|                   |
|     Topic 2       |
|   Partition 0     |
+-------------------+

经纪人3:

+-------------------+
|      Topic 1      |
|    Partition 1    |
|                   |
|                   |
|                   |
|                   |
+-------------------+

请注意,数据是分布式的(并且经纪人3 不包含任何主题2 的数据).

Note that data is distributed (and Broker 3 doesn't hold any data of topic 2).

主题,应该具有replication-factor> 1(通常为2或3),以便在代理崩溃时,另一个代理可以提供主题数据.例如,假设我们有一个包含2个分区的主题,并且replication-factor设置为2,如下所示:

Topics, should have a replication-factor > 1 (usually 2 or 3) so that when a broker is down, another one can serve the data of a topic. For instance, assume that we have a topic with 2 partitions with a replication-factor set to 2 as shown below:

经纪人1:

+-------------------+
|      Topic 1      |
|    Partition 0    |
|                   |
|                   |
|                   |
|                   |
+-------------------+

经纪人2:

+-------------------+
|      Topic 1      |
|    Partition 0    |
|                   |
|                   |
|     Topic 1       |
|   Partition 0     |
+-------------------+

经纪人3:

+-------------------+
|      Topic 1      |
|    Partition 1    |
|                   |
|                   |
|                   |
|                   |
+-------------------+

现在假定经纪人2 失败了. 经纪人1 和3仍然可以为主题1提供数据.因此,replication-factor始终为3是一个好主意,因为它允许出于维护目的而删除一个经纪人,也可以删除另一个经纪人.被意外拆除. 因此,Apache-Kafka提供了强大的耐用性和容错保证.

Now assume that Broker 2 has failed. Broker 1 and 3 can still serve the data for topic 1. So a replication-factor of 3 is always a good idea since it allows for one broker to be taken down for maintenance purposes and also for another one to be taken down unexpectedly. Therefore, Apache-Kafka offers strong durability and fault tolerance guarantees.

有关领导者的说明: 在任何时候,只有一个代理可以成为该分区的领导者,并且只有该领导者可以接收和提供该分区的数据.其余的代理将仅同步数据(同步副本).另请注意,当replication-factor设置为1时,如果代理失败,则 leader 不能移到其他位置.通常,当分区的所有副本失败或脱机时,leader将自动设置为-1.

Note about Leaders: At any time, only one broker can be a leader of a partition and only that leader can receive and serve data for that partition. The remaining brokers will just synchronize the data (in-sync replicas). Also note that when the replication-factor is set to 1, the leader cannot be moved elsewhere when a broker fails. In general, when all replicas of a partition fail or go offline, the leader will automatically be set to -1.

这篇关于在微服务中用作消息总线时,kafka是可靠的的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆