在微服务中用作消息总线时，kafka 是否可靠 [英] is kafka reliable when used as a message bus in micro services

问题描述

我使用 kafka 作为微服务架构的消息总线，因此多个服务监听一个主题以获取消息.因此，服务高度依赖于要直播的主题.

I am using kafka as a message bus for Micro Service architecture, hence multiple services listen on a topic for a message. Therefore, the services are highly dependent on the topic to be live.

但是，在很多情况下，我会在主题上获得 leader not available、broker not available 和 leader= - 1.

But, there are many instances where I get leader not available, broker not available and leader= - 1 on the topics.

现在，我不确定是否可以依赖 kafka 主题，因为当主题出现问题并导致平台出现问题时，服务会中断.

Now, I am not sure if I can rely on the kafka topics, as services get interrupted when there are issues on the topics with cause issues in the platform.

有人可以对这些主题的可靠性和可靠性有所了解吗?如果我们可以解决上述问题，我们是否可以恢复.

Can someone throw some light on the reliability and dependability on the topics and can we recover if we can across the above issues.

推荐答案

我将通过解释 Kafka 的一般工作原理以及它如何处理故障来回答您的问题.

I'll answer to your question by explaining how Kafka works in general and how it deals with failures.

每个主题，都是一个特定的数据流(类似于数据库中的表).主题被分成分区(任意多)，其中分区内的每条消息都有一个增量 ID，称为偏移量，如下所示.

Every topic, is a particular stream of data (similar to a table in a database). Topics, are split into partitions (as many as you like) where each message within a partition gets an incremental id, known as offset as shown below.

分区 0:

+---+---+---+-----+
| 0 | 1 | 2 | ... |
+---+---+---+-----+

第 1 部分:

+---+---+---+---+----+
| 0 | 1 | 2 | 3 | .. |
+---+---+---+---+----+

现在一个Kafka集群由多个broker组成.每个代理都用一个 ID 标识，并且可以包含某些主题分区.

Now a Kafka cluster is composed of multiple brokers. Each broker is identified with an ID and can contain certain topic partitions.

2 个主题的示例(每个主题分别有 3 个和 2 个分区):

Example of 2 topics (each having 3 and 2 partitions respectively):

经纪人 1:

+-------------------+
|      Topic 1      |
|    Partition 0    |
|                   |
|                   |
|     Topic 2       |
|   Partition 1     |
+-------------------+

经纪人 2:

+-------------------+
|      Topic 1      |
|    Partition 2    |
|                   |
|                   |
|     Topic 2       |
|   Partition 0     |
+-------------------+

经纪人 3:

+-------------------+
|      Topic 1      |
|    Partition 1    |
|                   |
|                   |
|                   |
|                   |
+-------------------+

请注意，数据是分布式的(Broker 3 不保存 topic 2 的任何数据).

Note that data is distributed (and Broker 3 doesn't hold any data of topic 2).

Topics，应该有一个 replication-factor > 1(通常是 2 或 3)，这样当一个 broker 宕机时，另一个 broker 可以提供一个 topic 的数据.例如，假设我们有一个包含 2 个分区的主题，其中 replication-factor 设置为 2，如下所示:

Topics, should have a replication-factor > 1 (usually 2 or 3) so that when a broker is down, another one can serve the data of a topic. For instance, assume that we have a topic with 2 partitions with a replication-factor set to 2 as shown below:

经纪人 1:

+-------------------+
|      Topic 1      |
|    Partition 0    |
|                   |
|                   |
|                   |
|                   |
+-------------------+

经纪人 2:

+-------------------+
|      Topic 1      |
|    Partition 0    |
|                   |
|                   |
|     Topic 1       |
|   Partition 0     |
+-------------------+

经纪人 3:

+-------------------+
|      Topic 1      |
|    Partition 1    |
|                   |
|                   |
|                   |
|                   |
+-------------------+

现在假设 Broker 2 失败了.Broker 1 和 3 仍然可以为主题 1 提供数据.所以 replication-factor 为 3 总是一个好主意，因为它允许一个 broker 被删除维护目的，也是为了另一个意外被删除.因此，Apache-Kafka 提供了强大的持久性和容错保证.

Now assume that Broker 2 has failed. Broker 1 and 3 can still serve the data for topic 1. So a replication-factor of 3 is always a good idea since it allows for one broker to be taken down for maintenance purposes and also for another one to be taken down unexpectedly. Therefore, Apache-Kafka offers strong durability and fault tolerance guarantees.

关于领导者的注意事项:在任何时候，只有一个代理可以是分区的领导者，并且只有该领导者可以为该分区接收和提供数据.其余的代理将只同步数据(同步副本).另请注意，当 replication-factor 设置为 1 时，当代理失败时，leader 无法移动到其他地方.一般来说，当一个partition的所有副本都失效或者下线时，leader会自动设置为-1.

Note about Leaders: At any time, only one broker can be a leader of a partition and only that leader can receive and serve data for that partition. The remaining brokers will just synchronize the data (in-sync replicas). Also note that when the replication-factor is set to 1, the leader cannot be moved elsewhere when a broker fails. In general, when all replicas of a partition fail or go offline, the leader will automatically be set to -1.

这篇关于在微服务中用作消息总线时，kafka 是否可靠的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！