是否需要密钥作为向 Kafka 发送消息的一部分? [英] Is key required as part of sending messages to Kafka?

问题描述

KeyedMessage<String, byte[]> keyedMessage = new KeyedMessage<String, byte[]>(request.getRequestTopicName(), SerializationUtils.serialize(message)); 
producer.send(keyedMessage);

目前，我正在发送没有任何密钥的消息作为密钥消息的一部分，它是否仍然适用于 delete.retention.ms?我是否需要将密钥作为消息的一部分发送?将密钥作为消息的一部分这样好吗?

Currently, I am sending messages without any key as part of keyed messages, will it still work with delete.retention.ms? Do I need to send a key as part of the message? Is this good to make key as part of the message?

推荐答案

如果您需要密钥的强顺序并且正在开发诸如状态机之类的东西，则密钥最有用/必不可少.如果您要求始终以正确的顺序查看具有相同键(例如，唯一 id)的消息，则将键附加到消息将确保具有相同键的消息始终进入主题中的同一分区.Kafka 保证分区内的顺序，但不保证主题中的分区之间的顺序，因此不提供键(这将导致跨分区的循环分配)将不会保持这种顺序.

Keys are mostly useful/necessary if you require strong order for a key and are developing something like a state machine. If you require that messages with the same key (for instance, a unique id) are always seen in the correct order, attaching a key to messages will ensure messages with the same key always go to the same partition in a topic. Kafka guarantees order within a partition, but not across partitions in a topic, so alternatively not providing a key - which will result in round-robin distribution across partitions - will not maintain such order.

在状态机的情况下，密钥可以与 log.cleaner.enable 一起使用，以删除具有相同密钥的条目的重复数据.在这种情况下，Kafka 假定您的应用程序只关心给定键的最新实例，并且只有当键不为空时，日志清理器才会删除给定键的旧副本.这种形式的日志压缩由 log.cleaner.delete.retention 属性控制并需要密钥.

In the case of a state machine, keys can be used with log.cleaner.enable to deduplicate entries with the same key. In that case, Kafka assumes that your application only cares about the most recent instance of a given key and the log cleaner deletes older duplicates of a given key only if the key is not null. This form of log compaction is controlled by the log.cleaner.delete.retention property and requires keys.

或者，默认情况下启用的更常见的属性 log.retention.hours 通过删除过期日志的完整段来工作.在这种情况下，不必提供密钥.Kafka 只会删除早于给定保留期的日志块.

Alternatively, the more common property log.retention.hours, which is enabled by default, works by deleting complete segments of the log that are out of date. In this case keys do not have to be provided. Kafka will simply delete chunks of the log that are older than the given retention period.

就是这样，如果您启用了日志压缩或者要求对具有相同密钥的消息进行严格排序，那么您绝对应该使用密钥.否则，在某些键可能比其他键出现更多的情况下，空键可以提供更好的分布并防止潜在的热点问题.

That's all to say, if you've enabled log compaction or require strict order for messages with the same key then you should definitely be using keys. Otherwise, null keys may provide better distribution and prevent potential hot spotting issues in cases where some keys may appear more than others.

这篇关于是否需要密钥作为向 Kafka 发送消息的一部分?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！