kafka 主题的每个分区中的提交和偏移量 [英] Number of commits and offset in each partition of a kafka topic

问题描述

如何在已知 kafka 主题的每个分区中查找提交次数和当前偏移量.我正在使用 kafka v0.8.1.1

How to find the number of commits and current offset in each partition of a known kafka topic. I am using kafka v0.8.1.1

推荐答案

从您的问题中不清楚您对哪种偏移感兴趣.实际上有三种类型的偏移量:

It is not clear from your question, what kind of offset you're interested in. There are actually three types of offsets:

1. 主题分区中第一条可用消息的偏移量.使用 -2(最早)作为 GetOffsetShell 工具的 --time 参数
2. 主题分区中最后一条可用消息的偏移量.使用 -1(latest) 作为 --time范围.
3. 最后读取/处理的消息偏移量由卡夫卡消费者.高级消费者将每个消费者组的这些信息存储在一个内部 Kafka 主题(曾经是 Zookeeper)并负责当您调用 commit() 或自动提交时保持最新设置设置为true.对于简单的消费者，您的代码必须关心管理抵消.

除了命令行实用程序，#1 和 #2 的偏移量信息也可以通过 SimpleConsumer.earliestOrLatestOffset() 获得.

In addition to command line utility, the offset information for #1 and #2 is also available via SimpleConsumer.earliestOrLatestOffset().

如果消息数量不是太多，可以给GetOffsetShell指定一个大的--offsets参数，然后统计工具返回的行数.否则，您可以在 scala/java 中编写一个简单的循环，该循环将从最早的开始迭代所有可用的偏移量.

If the number of messages is not too large, you can specify a large --offsets parameter to GetOffsetShell and then count number of lines returned by the tool. Otherwise, you can write a simple loop in scala/java that would iterate all available offsets starting from the earliest.

来自 Kafka 文档:

Get Offset Shell
get offsets for a topic
bin/kafka-run-class.sh kafka.tools.GetOffsetShell

required argument [broker-list], [topic]
Option Description 
------ ----------- 
--broker-list <hostname:port,..., REQUIRED: The list of hostname and hostname:port> port of the server to connect to. 
--max-wait-ms <Integer: ms> The max amount of time each fetch request waits. (default: 1000) 
--offsets <Integer: count> number of offsets returned (default: 1)
--partitions <partition ids> comma separated list of partition ids. If not specified, will find offsets for all partitions (default) 
--time <Long: timestamp in milliseconds / -1(latest) / -2 (earliest) timestamp; offsets will come before this timestamp, as in getOffsetsBefore  > 
--topic <topic> REQUIRED: The topic to get offsets from.

这篇关于kafka 主题的每个分区中的提交和偏移量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！