向Kafka发送消息时是否需要密钥? [英] Is key required as part of sending messages to Kafka?

问题描述

KeyedMessage<String, byte[]> keyedMessage = new KeyedMessage<String, byte[]>(request.getRequestTopicName(), SerializationUtils.serialize(message)); 
producer.send(keyedMessage);

当前，我正在发送不带任何键的消息作为键控消息的一部分，它是否仍可以与delete.retention.ms一起使用?我是否需要发送密钥作为消息的一部分?将密钥作为消息的一部分吗?

Currently, I am sending messages without any key as part of keyed messages, will it still work with delete.retention.ms? Do I need to send a key as part of the message? Is this good to make key as part of the message?

推荐答案

如果您需要严格的键顺序并正在开发状态机之类的键，则键最有用/很有必要.如果您要求始终以正确的顺序查看具有相同密钥(例如，唯一ID)的消息，则将密钥附加到消息将确保具有相同密钥的消息始终进入主题的同一分区. Kafka保证分区中的顺序，但不能保证主题中跨分区的顺序，因此，不提供密钥(将导致跨分区的循环分布)将不会保持这种顺序.

Keys are mostly useful/necessary if you require strong order for a key and are developing something like a state machine. If you require that messages with the same key (for instance, a unique id) are always seen in the correct order, attaching a key to messages will ensure messages with the same key always go to the same partition in a topic. Kafka guarantees order within a partition, but not across partitions in a topic, so alternatively not providing a key - which will result in round-robin distribution across partitions - will not maintain such order.

对于状态机，可以将键与 log.cleaner.enable 一起使用，以重复删除具有相同键的条目.在这种情况下，Kafka假定您的应用程序仅关心给定键的最新实例，并且日志清理器仅在键不为null的情况下才删除给定键的较旧副本.这种日志压缩形式由 log.cleaner.delete.retention 属性控制，并且需要密钥.

In the case of a state machine, keys can be used with log.cleaner.enable to deduplicate entries with the same key. In that case, Kafka assumes that your application only cares about the most recent instance of a given key and the log cleaner deletes older duplicates of a given key only if the key is not null. This form of log compaction is controlled by the log.cleaner.delete.retention property and requires keys.

或者，默认情况下启用的更常见的属性 log.retention.hours 通过删除日志中所有过期的完整段来工作.在这种情况下，不必提供密钥. Kafka只会删除早于给定保留期限的日志块.

Alternatively, the more common property log.retention.hours, which is enabled by default, works by deleting complete segments of the log that are out of date. In this case keys do not have to be provided. Kafka will simply delete chunks of the log that are older than the given retention period.

这就是说，如果您启用了日志压缩或要求使用相同密钥的邮件严格排序，那么您肯定应该使用密钥.否则，在某些键可能比其他键出现更多的情况下，空键可能会提供更好的分配并防止潜在的热点问题.

That's all to say, if you've enabled log compaction or require strict order for messages with the same key then you should definitely be using keys. Otherwise, null keys may provide better distribution and prevent potential hot spotting issues in cases where some keys may appear more than others.

这篇关于向Kafka发送消息时是否需要密钥?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！