Kafka主题分区 [英] Kafka topic partitions

问题描述

有关Kafka主题和分区的快速问题.假设以下情况:

A quick question concerning Kafka's topic and partitioning. Suppose to following scenario:

• Producer1将数据写入Topic1.

• Producer1 writes data into Topic1.

Producer2将数据写入Topic2

Producer2 writes data into Topic2

Consumer1从Topic1和Topic2读取数据.

Consumer1 reads data from Topic1 and Topic2.

Consumer2仅从Topic2读取数据.

Consumer2 reads data only from Topic2.

问题是:每个主题内有多少个分区?促进并行性取决于消费者的数量，这是真的吗?还是仅仅是文件server.config中设置的参数?在后一种情况下，是否可以使用内部具有不同数量的分区的不同主题?

The question is: how many partitions are there inside each Topic? Is it true that it depends on the number of consumers to promote parallelism? Or it's just a parameter set into the file server.config? In the latter case, is there a way to have different topics with different number of partitions inside?

推荐答案

首先要了解的是，主题分区是Kafka中并行性的单位.在生产者端和代理端，可以完全并行地完成对不同分区的写入.在使用者方面，Kafka始终将单个分区的数据提供给一个使用者线程.因此，使用者(在使用者组内)的并行度受要使用的分区数的限制.因此，通常，Kafka群集中的分区越多，可以实现的吞吐量就越高.

The first thing to understand is that a topic partition is the unit of parallelism in Kafka. On both the producer and the broker side, writes to different partitions can be done fully in parallel. On the consumer side, Kafka always gives a single partition’s data to one consumer thread. Thus, the degree of parallelism in the consumer (within a consumer group) is bounded by the number of partitions being consumed. Therefore, in general, the more partitions there are in a Kafka cluster, the higher the throughput one can achieve.

每个主题内有多少个分区?这是可配置的.您可以增加分区，但是一旦增加，就不能减少分区.Apache Kafka为我们提供了alter命令，以更改Topic行为并添加/修改配置.我们将使用alter命令向现有主题添加更多分区.

How many partitions are there inside each Topic? That's configurable. You can increase partition but once increased, you can not decrease it. Apache Kafka provides us with alter command to change Topic behavior and add/modify configurations. We will be using alter command to add more partitions to an existing Topic.

这是将主题"my-topic"的分区数增加到20的命令-

Here is the command to increase the partitions count for topic 'my-topic' to 20 -

./bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic --partitions 20

您可以使用以下describe命令来验证分区是否已增加-

You can verify whether partitions have been increased by using describe command as follows -

./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic

您需要为一个主题设置多少个分区?请在此处阅读该书面文件: https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/

How many partitions you need to set for a topic? Please read this well written document here: https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/

这篇关于Kafka主题分区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！