Kafka 主题分区 [英] Kafka topic partitions
问题描述
一个关于 Kafka 主题和分区的快速问题.假设以下场景:
A quick question concerning Kafka's topic and partitioning. Suppose to following scenario:
Producer1 将数据写入 Topic1.
Producer1 writes data into Topic1.
Producer2 将数据写入 Topic2
Producer2 writes data into Topic2
Consumer1 从 Topic1 和 Topic2 读取数据.
Consumer1 reads data from Topic1 and Topic2.
Consumer2 仅从 Topic2 读取数据.
Consumer2 reads data only from Topic2.
问题是:每个Topic里面有多少个分区?真的是靠消费者的数量来推动并行性吗?或者它只是设置到文件 server.config 中的参数?在后一种情况下，有没有办法让不同的主题在里面有不同数量的分区?
The question is: how many partitions are there inside each Topic? Is it true that it depends on the number of consumers to promote parallelism? Or it's just a parameter set into the file server.config? In the latter case, is there a way to have different topics with different number of partitions inside?
推荐答案
首先要理解的是，主题分区是 Kafka 中的并行单元.在生产者和代理端，对不同分区的写入可以完全并行完成.在消费者方面，Kafka 总是将单个分区的数据提供给一个消费者线程.因此，消费者(在消费者组内)的并行度受正在消费的分区数量的限制.因此，一般情况下，Kafka 集群中的分区越多，可以达到的吞吐量就越高.
The first thing to understand is that a topic partition is the unit of parallelism in Kafka. On both the producer and the broker side, writes to different partitions can be done fully in parallel. On the consumer side, Kafka always gives a single partition’s data to one consumer thread. Thus, the degree of parallelism in the consumer (within a consumer group) is bounded by the number of partitions being consumed. Therefore, in general, the more partitions there are in a Kafka cluster, the higher the throughput one can achieve.
每个Topic里面有多少个分区?那是可配置的.您可以增加分区，但一旦增加，您就不能减少它.Apache Kafka 为我们提供了alter 命令来更改主题行为和添加/修改配置.我们将使用 alter 命令向现有主题添加更多分区.
How many partitions are there inside each Topic? That's configurable. You can increase partition but once increased, you can not decrease it. Apache Kafka provides us with alter command to change Topic behavior and add/modify configurations. We will be using alter command to add more partitions to an existing Topic.
这是将主题my-topic"的分区数增加到 20 的命令 -
Here is the command to increase the partitions count for topic 'my-topic' to 20 -
./bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic --partitions 20
可以使用describe命令验证分区是否增加，如下 -
You can verify whether partitions have been increased by using describe command as follows -
./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic
您需要为一个主题设置多少个分区?请在此处阅读这篇写得很好的文档:https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
How many partitions you need to set for a topic? Please read this well written document here: https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
这篇关于Kafka 主题分区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
