Kafka - 代理:消息大小太大 [英] Kafka - Broker: Message size too large
问题描述
当我尝试发送超过 1 Mb 的消息时，我收到 Message size too large 异常.当我尝试生成消息时，错误出现在我的客户端应用程序中.经过一番谷歌搜索后，我发现应该更改设置以增加最大消息大小.好吧，我在 /kafka/config/server.properties 文件中做到了.我添加了接下来的 2 个设置:
I get Message size too large exception, when I try to send a message which is over 1 Mb size. The error appears in my client application, when I try to produce a message. After a little googling I found out that the settings should be changed in order to increase max message size. Well, I did that in /kafka/config/server.properties file. I added next 2 settings:
message.max.bytes=15728640
replica.fetch.max.bytes=15728640
此外，我将 fetch.message.max.bytes=15728640 添加到 /kafka/config/consumer.properties 文件.所有其他设置保持默认.
Also, I added fetch.message.max.bytes=15728640 to the /kafka/config/consumer.properties file. All other settings remain default.
我重启了 kafka 服务器，但我仍然遇到同样的错误.
I did kafka server restart, but I'm still getting the same error.
P.S Kafka 版本是 1.1.0.
P.S Kafka version is 1.1.0.
推荐答案
您有正确的配置，但是您还需要在生产者端设置 max.request.size.
You have the right configuration however you need to also set max.request.size on the producer side.
props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 15728640);
max.request.size 请求的最大大小(以字节为单位).此设置将限制生产者将发送的记录批次数在单个请求中避免发送大量请求.这实际上也是最大记录批量大小的上限.
max.request.size The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size.
在 Broker 端，您已经配置了以下应该可以工作的参数
On the Broker side, you have already configured the below parameter that should work
message.max.bytes Kafka 允许的最大记录批大小.
message.max.bytes The largest record batch size allowed by Kafka.
replica.fetch.max.bytes 尝试为每个分区获取的消息字节数.这不是绝对最大值，如果提取的第一个非空分区中的第一批记录是大于此值，记录批次仍将返回确保能够取得进展.最大记录批量大小经纪人接受是通过 message.max.bytes 定义的(经纪人config) 或 max.message.bytes(主题配置).
replica.fetch.max.bytes The number of bytes of messages to attempt to fetch for each partition. This is not an absolute maximum if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config).
在主题端 max.message.bytes，如果您已经在代理端设置了 message.max.bytes，则不需要该值
On the topic side max.message.bytes which is not required in case you have already set message.max.bytes in the broker side
max.message.bytes - 这是代理消息的最大大小将允许附加到主题.这个尺寸经过验证预压缩.(默认为经纪人的 message.max.bytes.)
max.message.bytes - this is the largest size of the message the broker will allow being appended to the topic. This size is validated pre-compression. (Defaults to broker's message.max.bytes.)
参考https://kafka.apache.org/documentation/
这篇关于Kafka - 代理:消息大小太大的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
