如果我在主题级别和生产者级别设置“compression.type"，则优先 [英] If i set 'compression.type' at topic level and producer level, which takes precedence

问题描述

我正在尝试了解compression.type"配置，我的问题是，如果我在主题级别和生产者级别设置compression.type"，哪个优先?

I'm trying to understand the 'compression.type' configuration and my question is, If i set 'compression.type' at topic level and producer level, which takes precedence?

推荐答案

我尝试了一些实验来回答这个问题:

I tried out some experiments to answer this:

**Note:** server.properties has the config compression.type=producer 

./kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1--config compression.type=producer --topic t

./kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1--config compression.type=producer --topic t

./kafka-console-producer.sh --broker-list node:6667  --topic t
./kafka-console-producer.sh --broker-list node:6667  --topic t --compression-codec gzip
./kafka-console-producer.sh --broker-list node:6667  --topic t

sh kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration --files /kafka-logs/t-0/00000000000000000000.log

Dumping /kafka-logs/t-0/00000000000000000000.log
Starting offset: 0
offset: 0 position: 0 compresscodec: NONE 
offset: 1 position: 69 compresscodec: GZIP 
offset: 2 position: 158 compresscodec: NONE 

./kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1--config compression.type=gzip --topic t1

./kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1--config compression.type=gzip --topic t1

./kafka-console-producer.sh --broker-list node:6667  --topic t1
./kafka-console-producer.sh --broker-list node:6667  --topic t1 --compression-codec gzip
./kafka-console-producer.sh --broker-list node:6667  --topic t1 --compression-codec snappy

 sh kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration --files /kafka-logs/t1-0/00000000000000000000.log
Dumping /kafka-logs/t1-0/00000000000000000000.log
Starting offset: 0
offset: 0 position: 0 compresscodec: GZIP 
offset: 1 position: 89 compresscodec: GZIP 
offset: 2 position: 178 compresscodec: GZIP 

显然该主题占据了主导地位.

w.r.t 压缩和解压来自 Kafka 的文字——权威指南

w.r.t compression and decompression text from Kafka - the definitive guide

然而，Kafka 代理必须解压缩所有消息批次，以验证各个消息的校验和并分配偏移量.然后它需要重新压缩消息批处理以将其存储在磁盘上.

The Kafka broker must decompress all message batches, however, in order to validate the checksum of the individual messages and assign offsets. It then needs to recompress the message batch in order to store it on disk.

从 0.10 版开始，有一种新的消息格式，允许在消息批处理中使用相对偏移量.这意味着新的生产者将在发送消息批次之前设置相对偏移量，这允许代理跳过消息批次的重新压缩.

As of version 0.10, there is a new message format that allows for relative offsets in a message batch. This means that newer producers will set relative offsets prior to sending the message batch, which allows the broker to skip recompression of the message batch.

所以，当压缩类型不同时，主题压缩被尊重.如果相同，则保留生产者设置的原始压缩编解码器.

So, when the compression type is different, the topic compression is honoured. if it is same, it will retain the original compression codec set by the producer.

参考 - https://kafka.apache.org/documentation/

这篇关于如果我在主题级别和生产者级别设置“compression.type"，则优先的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！