Kafka-经纪人:邮件大小过大 [英] Kafka - Broker: Message size too large

问题描述

当我尝试发送超过1 Mb大小的消息时，出现消息大小太大异常.当我尝试生成一条消息时，该错误出现在我的客户端应用程序中.经过一番谷歌搜索后，我发现应该更改设置以增加最大邮件大小.好吧，我是在/kafka/config/server.properties 文件中完成的.我添加了下2个设置:

  message.max.bytes = 15728640copy.fetch.max.bytes = 15728640 

此外，我在/kafka/config/consumer.properties 文件中添加了 fetch.message.max.bytes = 15728640 .所有其他设置仍为默认设置.

我没有重启kafka服务器，但是仍然出现相同的错误.

PS Kafka版本为1.1.0.

解决方案

您的配置正确，但是还需要在生产者端设置 max.request.size .

props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG，15728640);

max.request.size (请求的最大大小，以字节为单位).此设置将限制生产者将发送的记录批数在单个请求中避免发送大量请求.这实际上也是最大记录批次大小的上限.

在Broker端，您已经配置了以下应该起作用的参数

message.max.bytes (最大邮件字节数)

replica.fetch.max.bytes 尝试为每个分区获取的消息的字节数.这不是绝对最大值，如果提取的第一个非空分区中的第一个记录批处理是大于此值，记录批仍将返回到确保可以取得进展.最大记录批量大小代理接受的消息是通过message.max.bytes(经纪人)定义的配置)或max.message.bytes(主题配置).

在主题端，如果您已经在代理端设置了message.max.bytes，则不需要max.message.bytes

最大消息字节数-这是代理的最大消息大小将允许附加到该主题.此大小已验证预压缩.(默认为代理的message.max.bytes.)

参考 https://kafka.apache.org/documentation/

I get Message size too large exception, when I try to send a message which is over 1 Mb size. The error appears in my client application, when I try to produce a message. After a little googling I found out that the settings should be changed in order to increase max message size. Well, I did that in /kafka/config/server.properties file. I added next 2 settings:

message.max.bytes=15728640
replica.fetch.max.bytes=15728640

Also, I added fetch.message.max.bytes=15728640 to the /kafka/config/consumer.properties file. All other settings remain default.

I did kafka server restart, but I'm still getting the same error.

P.S Kafka version is 1.1.0.

解决方案

You have the right configuration however you need to also set max.request.size on the producer side.

props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 15728640);

max.request.size The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size.

On the Broker side, you have already configured the below parameter that should work

message.max.bytes The largest record batch size allowed by Kafka.

replica.fetch.max.bytes The number of bytes of messages to attempt to fetch for each partition. This is not an absolute maximum if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config).

On the topic side max.message.bytes which is not required in case you have already set message.max.bytes in the broker side

max.message.bytes - this is the largest size of the message the broker will allow being appended to the topic. This size is validated pre-compression. (Defaults to broker's message.max.bytes.)

Refrence https://kafka.apache.org/documentation/

这篇关于Kafka-经纪人:邮件大小过大的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！