Kafka消息编解码器 - 压缩和解压缩 [英] Kafka message codec - compress and decompress

问题描述

使用 kafka 时，可以通过设置kafka.compression.codec属性来设置编码解码器

When using kafka, I can set a codec by setting the kafka.compression.codec property of my kafka producer.

假设我在我的生产者中使用snappy压缩，当使用kafka消费者消费来自kafka的消息时，我应该做一些解码数据snappy或是kafka消费者的某些内置功能？

Suppose I use snappy compression in my producer, when consuming the messages from kafka using some kafka-consumer, should I do something to decode the data from snappy or is it some built-in feature of kafka consumer?

在相关文档我找不到任何与kafka消费者编码相关的属性（它只与生产者有关）。

In the relevant documentation I could not find any property that relates to encoding in kafka consumer (it only relates to the producer).

有人可以清除这个？

推荐答案

根据我的理解，解压缩由消费者自己处理。如他们的官方wiki页面中提到的
消费者迭代器透明地解压缩压缩数据并且只返回一个未压缩的消息

As per my understanding goes the de-compression is taken care by the Consumer it self. As mentioned in their official wiki page The consumer iterator transparently decompresses compressed data and only returns an uncompressed message

如这篇文章中所述，消费者的工作方式如下

As found in this article the way consumer works is as follows

消费者具有背景提取器线程，可以从代理中连续获取1MB的数据并将其添加到内部阻塞队列。消费者线程从这个阻塞队列中取出数据，通过消息解压缩和迭代

http://kafka.apache.org/documentation.html\">结束批量压缩，其中写道：

And also in the doc page under End-to-end Batch Compression its written that

一批消息可以被压缩在一起，并以此形式发送到服务器。此批消息将以压缩形式写入，并将在日志中保持压缩状态，并且只能由消费者解压缩。

A batch of messages can be clumped together compressed and sent to the server in this form. This batch of messages will be written in compressed form and will remain compressed in the log and will only be decompressed by the consumer.

看起来解压缩部分在消费者中处理它自己所有你需要做的是提供有效/支持的压缩类型使用 compression.codec ProducerConfig属性，同时创建生产者。我找不到任何例子或解释，它说，在消费者解压缩的任何方法。如果我错了，请更正我。

So it appears that the decompression part is handled in the consumer it self all you need to do is to provide the valid / supported compression type using the compression.codec ProducerConfig attribute while creating the producer. I couldn't find any example or explanation where it says any approach for decompression in the consumer end. Please correct me if I am wrong.

这篇关于Kafka消息编解码器 - 压缩和解压缩的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！