Kafka Streams线程号 [英] Kafka Streams thread number

问题描述

我是Kafka Streams的新手，目前对Kafka Streams应用程序的最大并行性感到困惑.我经过以下链接，但没有得到我要查找的答案. https://docs.confluent.io/current/streams/faq.html#streams-faq-scalability-maximum-parallelism

I am new to Kafka Streams, I am currently confused with the maximum parallelism of Kafka Streams application. I went through following link and did not get the answer what I am trying to find. https://docs.confluent.io/current/streams/faq.html#streams-faq-scalability-maximum-parallelism

如果我有2个输入主题，一个有10个分区，另一个有5个分区，并且只有一个Kafka Streams应用程序实例正在运行以处理这两个输入主题，在这种情况下，我可以拥有的最大线程数是多少? 10还是15?

If I have 2 input topics, one have 10 partitions and the other have 5 partitions, and only one Kafka Streams application instance is running to process these two input topics, what is the maximum thread number I can have in this case? 10 or 15?

推荐答案

如果我有2个输入主题，一个主题有10个分区，另一个主题有5个分区

If I have 2 input topics, one have 10 partitions and the other have 5 partitions

听起来不错.因此，您总共有15个分区.假设您有一个简单的处理器拓扑，没有联接和聚合，因此所有15个分区都只是无状态转换.

Sounds good. So you have 15 total partitions. Let's assume you have a simple processor topology, without joins and aggregations, so that all 15 partitions are just being statelessly transformed.

然后，这15个输入分区中的每个分区都将映射到单个Kafka Streams任务".如果您有1个线程，则这15个任务的输入将由该1个线程处理.如果您有15个线程，则每个任务将有一个专用线程来处理其输入.因此，您可以运行具有15个线程的1个应用程序或具有1个线程的15个应用程序，这在逻辑上是相似的:您可以在15个线程中处理15个任务.唯一的区别是15个具有1个线程的应用程序使您可以将负载分散到整个JVM中.

Then, each of the 15 input partitions will map to a single a Kafka Streams "task". If you have 1 thread, input from these 15 tasks will be processed by that 1 thread. If you have 15 threads, each task will have a dedicated thread to handle its input. So you can run 1 application with 15 threads or 15 applications with 1 thread and it's logically similar: you process 15 tasks in 15 threads. The only difference is that 15 applications with 1 thread allows you to spread your load over across JVMs.

同样，如果启动应用程序的15个实例，每个实例具有1个线程，则将为每个应用程序分配1个任务，每个应用程序中的每个1个线程将处理其给定的1个任务.

Likewise, if you start 15 instances of the application, each instance with 1 thread, then each application will be assigned 1 task, and each 1 thread in each application will handle its given 1 task.

在这种情况下，我可以拥有的最大线程数是多少? 10还是15?

what is the maximum thread number I can have in this case? 10 or 15?

您可以将最大线程数设置为任何值.如果您所有任务的线程数都超过任务总数，那么某些线程将保持空闲状态.

You can set your maximum thread count to anything. If your thread count across all tasks exceeds the total number of tasks, then some of the threads will remain idle.

我建议阅读 https://docs.confluent.io/current/streams/architecture.html#parallelism-model (如果尚未安装).另外，研究应用程序启动时生成的日志.每个线程都会记录它分配的任务，如下所示:

I recommend reading https://docs.confluent.io/current/streams/architecture.html#parallelism-model, if you haven't yet. Also, study the logs your application produces when it starts up. Each thread logs the tasks it gets assigned, like this:

[2018-01-04 16:45:26,859] INFO (org.apache.kafka.streams.processor.internals.StreamThread:351) stream-thread [entities-eb9c0a9b-ecad-48c1-b4e8-715dcf2afef3-StreamThread-3] partition assignment took 110 ms.
current active tasks: [0_0, 0_2, 1_2, 2_2, 3_2, 4_2, 5_2, 6_2, 7_2, 8_2, 9_2, 10_2, 11_2, 12_2, 13_2, 14_2]
current standby tasks: []
previous active tasks: []

这篇关于Kafka Streams线程号的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！