如何强迫消费者读取kafka中的特定分区 [英] How to force a consumer to read a specific partition in kafka

问题描述

我有一个应用程序，可以从1位Kafka生产者生成的URL流中下载特定的Web内容.我创建了一个具有5个分区的主题，并且有5个kafka消费者.但是，网页下载的超时为60秒. 在下载其中一个URL时，服务器假定该消息已丢失，然后将数据重新发送给其他使用者.

I have an application for downloading specific web-content, from a stream of URL's generated from 1 Kafka-producer. I've created a topic with 5 partitions and there are 5 kafka-consumers. However the timeout for the webpage download is 60 seconds. While one of the url is getting downloaded, the server assumes that the message is lost and resends the data to different consumers.

我已经尝试了

Kafka使用者配置/性能问题

和

https://github.com/spring-projects/spring-kafka/issues/202

但是我每次都会得到不同的错误.

But I keep getting different errors everytime.

是否可以将特定用户与kafka中的分区绑定在一起? 我正在为我的应用程序使用kafka-python

Is it possible to tie a specific consumer with a partition in kafka? I am using kafka-python for my application

推荐答案

我错过了Kafka-python的文档.我们可以使用TopicPartition类为特定使用者分配一个分区.

I missed on the documentation of Kafka-python. We can use TopicPartition class to assign a specific consumer with one partition.

http://kafka-python.readthedocs.io/en/master/

>>> # manually assign the partition list for the consumer
>>> from kafka import TopicPartition
>>> consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
>>> consumer.assign([TopicPartition('foobar', 2)])
>>> msg = next(consumer)

这篇关于如何强迫消费者读取kafka中的特定分区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！