kafka主题的每个分区中的提交数和偏移量 [英] Number of commits and offset in each partition of a kafka topic

问题描述

如何在已知 kafka主题的每个分区中查找提交次数和当前偏移量.我正在使用kafka v0.8.1.1

How to find the number of commits and current offset in each partition of a known kafka topic. I am using kafka v0.8.1.1

推荐答案

您的问题尚不清楚，您对哪种补偿感兴趣.偏移量实际上有三种:

It is not clear from your question, what kind of offset you're interested in. There are actually three types of offsets:

1. 第一个可用消息在主题分区中的偏移量.使用-2(最早)作为GetOffsetShell工具的--time参数
2. 主题分区中最后一条可用消息的偏移量.使用-1(latest)作为--time范围.
3. 上一次读取/处理的消息偏移量由卡夫卡消费者.高级消费者针对每个消费者组将这些信息存储在内部的Kafka主题(以前是Zookeeper)，负责在调用commit()或自动提交时保持最新设置为true.对于简单的使用者，您的代码必须关心管理偏移量.

除命令行实用程序外，还可通过SimpleConsumer.earliestOrLatestOffset()获得#1和#2的偏移信息.

In addition to command line utility, the offset information for #1 and #2 is also available via SimpleConsumer.earliestOrLatestOffset().

如果消息的数量不是太大，则可以为GetOffsetShell指定一个较大的--offsets参数，然后计算该工具返回的行数.否则，您可以在scala/java中编写一个简单的循环，该循环将从最早开始迭代所有可用的偏移量.

If the number of messages is not too large, you can specify a large --offsets parameter to GetOffsetShell and then count number of lines returned by the tool. Otherwise, you can write a simple loop in scala/java that would iterate all available offsets starting from the earliest.

来自Kafka文档:

Get Offset Shell
get offsets for a topic
bin/kafka-run-class.sh kafka.tools.GetOffsetShell

required argument [broker-list], [topic]
Option Description 
------ ----------- 
--broker-list <hostname:port,..., REQUIRED: The list of hostname and hostname:port> port of the server to connect to. 
--max-wait-ms <Integer: ms> The max amount of time each fetch request waits. (default: 1000) 
--offsets <Integer: count> number of offsets returned (default: 1)
--partitions <partition ids> comma separated list of partition ids. If not specified, will find offsets for all partitions (default) 
--time <Long: timestamp in milliseconds / -1(latest) / -2 (earliest) timestamp; offsets will come before this timestamp, as in getOffsetsBefore  > 
--topic <topic> REQUIRED: The topic to get offsets from.

这篇关于kafka主题的每个分区中的提交数和偏移量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！