kafka-python中的多处理 [英] multiprocessing in kafka-python

问题描述

我一直在使用python-kaka模块从kafka经纪人那里消费.我想从同一个主题中并行使用'x'个分区.该文档具有以下内容:

I have been using the python-kaka module to consume from a kafka broker. I want to consume from the same topic with 'x' number of partitions in parallel. The documentation has this :

# Use multiple consumers in parallel w/ 0.9 kafka brokers
# typically you would run each on a different server / process / CPU
 consumer1 = KafkaConsumer('my-topic',
                      group_id='my-group',
                      bootstrap_servers='my.server.com')
  consumer2 = KafkaConsumer('my-topic',
                      group_id='my-group',
                      bootstrap_servers='my.server.com')

这是否意味着我可以为产生的每个进程创建一个单独的使用者?另外，消费者1和消费者2使用的消息是否会有重叠?

Does this mean I can create a separate consumer for each process that I spawn? Also, will there be an overlap on the messages being consumed by consumer1 and consumer2 ?

谢谢

推荐答案

是的，您可以在多个线程/进程中创建多个使用者(甚至可以在不同的机器上并行运行它们).只要所有使用者都使用相同的 groupID ，就不会有重叠. Kafka将每个主题分区分配给使用者组中的单个使用者.请注意，使用的使用者多于可用的主题分区，将导致使用者闲置.

Yes, you can create multiple consumers in multiple threads/processes (and even run them in parallel on different machines). As long as all consumers use the same groupID, there will be no overlap. Kafka assigned each topic partition to a single consumer within a consumer group. Be aware, that using more consumers than available topic partitions will result in idle consumers.

这篇关于kafka-python中的多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！