如何在Java中获得消费者Kafka滞后 [英] How to get consumer Kafka lag in java

问题描述

我在 java 中有一个生产者，在 nodeJS 中有一个消费者.我想在 Java 中知道什么是消费者滞后，所以我知道我是否可以为该主题生成更多数据.

java 中获取消费者延迟的 API 是什么?

解决方案

您可以从 Java 调用的实际类是 kafka.admin.ConsumerGroupCommand.它是 Scala 代码，但很容易直接从 Java 调用.

这里有一个 Github 代码链接供细读:https://github.com/apache/kafka/blob/bf237fa7c576bd141d78fdea9f17f65ea269c290/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala

如果您不传入任何参数，这里是帮助输出.这应该对您需要发送的命令有所帮助.

<前>列出所有消费者组，描述一个消费者组，删除消费者组信息，或重置消费者组偏移量.选项说明————————————--all-topics 考虑分配给一个的所有主题组在`reset-offsets`过程中.--bootstrap-server--by-duration 重置偏移量以按持续时间偏移从当前时间戳.格式:'PnDTnHnMnS'--command-config 传递给管理客户端和消费者.--delete 分组传递删除主题分区偏移量和所有权整个消费者的信息团体.例如 --group g1 --g2组--describe 描述消费者组和列表偏移滞后(消息数不尚未处理)与给定的团体.--dry-run 只显示结果不执行消费者组的变化.支持的操作:重置偏移.--execute 执行操作.支持的操作:重置偏移量.--export 将操作执行导出为 CSV文件.支持的操作:reset-抵消.--from-file 将偏移量重置为 CSV 中定义的值文件.--group 我们希望对其采取行动的消费者群体.--list 列出所有消费者组.--members 描述组的成员.这个选项可以与--describe"一起使用和--bootstrap-server"选项只要.示例:--bootstrap-server 本地主机:9092 --describe --group group1 --成员--offsets 描述组并列出所有主题组中的分区以及他们的抵消滞后.这是和的默认子操作可能是与 '--describe' 和 '-- 一起使用仅引导服务器的选项.示例:--bootstrap-server 本地主机:9092 --describe --group group1 --抵消--reset-offsets 重置消费者组的偏移量.支持一个消费者群体时间和实例应该是不活跃有 2 个执行选项:--dry-run(默认)计划哪些偏移重置，并 --execute 更新偏移量.此外，——导出选项用于导出结果转换为 CSV 格式.您必须选择以下之一重置规格:--to-datetime，--by-period, --to-earliest, --to-最新, --shift-by, --from-file, --到电流.要定义范围，请使用 --all-topics或--主题.一个范围必须是除非您使用 '--from-文件'.--shift-by 重置偏移量移动当前偏移量by 'n'，其中 'n' 可以是正数或消极的.--state 描述组状态.这个选项可以与--describe"和--仅引导服务器的选项.示例:--bootstrap-server 本地主机:9092 --describe --group group1 --状态--timeout 可以为某些设置的超时时间用例.例如，它可以是在描述组时使用指定最长时间在毫秒之前等待组稳定(当组刚刚创建或正在经历一些变化).(默认值:5000)--to-current 将偏移重置为当前偏移.--to-datetime 重置偏移量以偏移日期时间.格式:'YYYY-MM-DDTHH:mm:SS.sss'--to-earliest 将偏移量重置为最早的偏移量.--to-latest 将偏移量重置为最新的偏移量.--to-offset 将偏移重置为特定的偏移.--topic 消费组的主题信息应被删除或应包含在其中的主题复位偏移过程.在`重置-offsets` 的情况下，分区可以是使用以下格式指定:`topic1:0,1,2`，其中 0,1,2 是分区要包含在过程.重置偏移也支持多个主题输入.--verbose 提供附加信息，如果任何，在描述组时.这个选项可以与'--偏移量'/'--members'/'--state' 和仅限--bootstrap-server"选项.示例:--bootstrap-server 本地主机:9092 --describe --group group1 --成员——详细

I have a producer in java and consumer in nodeJS. I want to know in java what is the consumer lag, so i know if i can produce more data to the topic.

What is the API in java to get the consumer lag?

解决方案

The actual class you can call from Java is the kafka.admin.ConsumerGroupCommand. It's Scala code, but it's easy to call directly from Java.

Here's a link to the Github code for perusal: https://github.com/apache/kafka/blob/bf237fa7c576bd141d78fdea9f17f65ea269c290/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala

And here's the help output if you don't pass in any parameters. That should help out with the commands you need to send.

 
List all consumer groups, describe a consumer group, delete consumer group info, or reset consumer group offsets.
Option                                  Description                            
------                                  -----------                            
--all-topics                            Consider all topics assigned to a      
                                          group in the `reset-offsets` process.
--bootstrap-server                                                                   
--by-duration         Reset offsets to offset by duration    
                                          from current timestamp. Format:      
                                          'PnDTnHnMnS'                         
--command-config                    passed to Admin Client and Consumer. 
--delete                                Pass in groups to delete topic         
                                          partition offsets and ownership      
                                          information over the entire consumer 
                                          group. For instance --group g1 --    
                                          group g2                             
--describe                              Describe consumer group and list       
                                          offset lag (number of messages not   
                                          yet processed) related to given      
                                          group.                               
--dry-run                               Only show results without executing    
                                          changes on Consumer Groups.          
                                          Supported operations: reset-offsets. 
--execute                               Execute operation. Supported           
                                          operations: reset-offsets.           
--export                                Export operation execution to a CSV    
                                          file. Supported operations: reset-   
                                          offsets.                             
--from-file   Reset offsets to values defined in CSV 
                                          file.                                
--group         The consumer group we wish to act on.  
--list                                  List all consumer groups.              
--members                               Describe members of the group. This    
                                          option may be used with '--describe' 
                                          and '--bootstrap-server' options     
                                          only.                                
                                        Example: --bootstrap-server localhost: 
                                          9092 --describe --group group1 --    
                                          members                              
--offsets                               Describe the group and list all topic  
                                          partitions in the group along with   
                                          their offset lag. This is the        
                                          default sub-action of and may be     
                                          used with '--describe' and '--       
                                          bootstrap-server' options only.      
                                        Example: --bootstrap-server localhost: 
                                          9092 --describe --group group1 --    
                                          offsets                              
--reset-offsets                         Reset offsets of consumer group.       
                                          Supports one consumer group at the   
                                          time, and instances should be        
                                          inactive                             
                                        Has 2 execution options: --dry-run     
                                          (the default) to plan which offsets  
                                          to reset, and --execute to update    
                                          the offsets. Additionally, the --    
                                          export option is used to export the  
                                          results to a CSV format.             
                                        You must choose one of the following   
                                          reset specifications: --to-datetime, 
                                          --by-period, --to-earliest, --to-    
                                          latest, --shift-by, --from-file, --  
                                          to-current.                          
                                        To define the scope use --all-topics   
                                          or --topic. One scope must be        
                                          specified unless you use '--from-    
                                          file'.                               
--shift-by     Reset offsets shifting current offset  
                                          by 'n', where 'n' can be positive or 
                                          negative.                            
--state                                 Describe the group state. This option  
                                          may be used with '--describe' and '--
                                          bootstrap-server' options only.      
                                        Example: --bootstrap-server localhost: 
                                          9092 --describe --group group1 --    
                                          state                                
--timeout           The timeout that can be set for some   
                                          use cases. For example, it can be    
                                          used when describing the group to    
                                          specify the maximum amount of time   
                                          in milliseconds to wait before the   
                                          group stabilizes (when the group is  
                                          just created, or is going through    
                                          some changes). (default: 5000)       
--to-current                            Reset offsets to current offset.       
--to-datetime         Reset offsets to offset from datetime. 
                                          Format: 'YYYY-MM-DDTHH:mm:SS.sss'    
--to-earliest                           Reset offsets to earliest offset.      
--to-latest                             Reset offsets to latest offset.        
--to-offset               Reset offsets to a specific offset.    
--topic                  The topic whose consumer group         
                                          information should be deleted or     
                                          topic whose should be included in    
                                          the reset offset process. In `reset- 
                                          offsets` case, partitions can be     
                                          specified using this format: `topic1:
                                          0,1,2`, where 0,1,2 are the          
                                          partition to be included in the      
                                          process. Reset-offsets also supports 
                                          multiple topic inputs.               
--verbose                               Provide additional information, if     
                                          any, when describing the group. This 
                                          option may be used with '--          
                                          offsets'/'--members'/'--state' and   
                                          '--bootstrap-server' options only.   
                                        Example: --bootstrap-server localhost: 
                                          9092 --describe --group group1 --    
                                          members --verbose  

这篇关于如何在Java中获得消费者Kafka滞后的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！