Kafka分区和Kafka副本之间有什么区别? [英] What is the difference between Kafka partitions and Kafka replicas?
问题描述
我创建了3个Kafka经纪人设置，其经纪人ID为20、21、22.然后，我创建了这个主题:
I created 3 Kafka brokers setup with broker id's 20,21,22. Then I created this topic:
bin/kafka-topics.sh --zookeeper localhost:2181 \
--create --topic zeta --partitions 4 --replication-factor 3
导致:
当生产者发送消息"hello world"时，到主题zeta，Kafka首先将消息写入哪个分区?
When a producer sends message "hello world" to topic zeta, to which partition the message first gets written to by Kafka?
你好世界"消息是否在所有四个分区中复制?
The "hello world" message gets replicated in all 4 partitions?
3个代理中的每个代理都包含所有4个分区?在上述情况下，它与3的复制因子有什么关系?
Each broker among the 3 brokers contain all the 4 partitions? How is that related to replica factor of 3 in above context?
如果我有8位使用者在他们自己的进程或线程中并行运行并订阅了zeta主题，那么Kafka如何分配分区或代理来并行服务这些人?
If I have 8 consumers running in their own processes or threads in parallel subscribed to zeta topic, how partitions or brokers are assigned by Kafka to serve these in parallel?
推荐答案
复制和分区是两个不同的东西.
复制将在整个集群中复制相同的数据，以提高可用性/耐久性. 分区是Kafka在整个群集中分发非冗余数据的方式，并且会随着分区数量的增加而扩展.
Replication will copy identical data across the cluster for higher availability/durability. Partitions are Kafka's way to distribute non-redundant data across the cluster and it scales with the number of partitions.
当生产者发送消息"hello world"时，到主题zeta，Kafka首先将消息写入哪个分区?
When a producer sends message "hello world" to topic zeta, to which partition the message first gets written to by Kafka?
当您发送"hello world"消息时，消息到主题，默认情况下，生产者根据该消息的密钥(例如hash(key) % number_of_partitions)应用哈希算法.如果您没有提供密钥，则生产者将进行轮询，因此无法预测消息将发送到哪个分区.我想如果这是第一则消息，它将最终出现在分区0中.
When you send a "hello world" message to a topic, by default, your producer applies a hashing algorithm based on the key of that message (like hash(key) % number_of_partitions). In case you did not provide a key the producer will do round-robin and it is therefore not predictable to which partitions the message will be sent. I am guessing if it is the first message, it will end up in partition 0.
你好世界"消息是否在所有四个分区中复制?
The "hello world" message gets replicated in all 4 partitions?
这条消息将在所有副本中复制，但不会复制到4个分区.
This one message will get replicated across all your Replicas but not to the 4 partitions.
您将在代理20、21、22上找到该消息.但是，每个分区都有一个负责人，负责处理该分区的所有读写操作.在屏幕快照中，您还可以发现每个分区的负责人的经纪人ID.从分区0的Leader: 21中，您可以知道该分区的负责人位于代理21上.
You will find the message on the broker 20, 21, 22. However, each partition has a leader which is responsible for all reads and writes from and to that partition. In your screenshot you can also spot the broker id of the leader of each partition. From Leader: 21 for partition 0 you can tell that the leader of that partition sits on broker 21.
3个代理中的每个代理都包含所有4个分区?在上述情况下，它与3的复制因子有什么关系?
Each broker among the 3 brokers contain all the 4 partitions? How is that related to replica factor of 3 in above context?
当您将复制因子设置为3时，集群中总共有3个代理，所以所有三个代理都包含所有四个分区.同样，分区和副本之间是有区别的.您可以使用Kafka的集群"拥有一个经纪人，但在该主题中仍然有20个分区.
As you have set the replication factor to 3 while having in total 3 brokers in your cluster all three brokers contain all four partitions. Again, there is a difference between partitions and replicas. You could have a Kafka "cluster" with a single broker and still have, say, 20 partitions in the topic.
如果我有8位使用者在他们自己的进程或线程中并行运行并订阅了zeta主题，那么Kafka如何分配分区或代理来并行服务这些人?
If I have 8 consumers running in their own processes or threads in parallel subscribed to zeta topic, how partitions or brokers are assigned by Kafka to serve these in parallel?
在此取决于这8个消费者是否属于同一个消费者组.重要的是要知道，一个分区最多只能由来自特定使用者组的一个使用者线程读取.
Here it depends if those 8 consumers belong to the same Consumer Group or not. It is important to know that one partition can be read at most by one consumer thread from a particular consumer group.
如果所有8个使用者都属于同一组，则其中4个使用者将从一个分区读取(仅从分区领导者读取)，而其他四个则处于空闲状态.
If all 8 consumers belong to the same group, 4 of them will read from one partition (only from the partition leader) and the other four will be idle.
这篇关于Kafka分区和Kafka副本之间有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
