Kafka Streams 可以配置为等待 KTable 加载吗? [英] Can Kafka Streams be configured to wait for KTable to load?
问题描述
我使用物化的 KTable 与我的 KStream 进行左连接(而流在左侧).
I'm using materialized KTable to use for left join with my KStream(while the stream is the left side).
然而，它似乎立即处理，无需等待当前版本的 KTable 加载..
However, it seem to process immediately, without waiting for the current version of the KTable to load..
我的 KTable 源主题中有很多值，当我启动应用程序时，很多连接都失败了(好吧，不是真的，因为它是左连接).
I have a lot of values in my source topic for the KTable and when I start the application, a lot of joins fail(well, not really since it is a left join).
我可以让它延迟启动以便等待初始主题加载吗?
Can I make it start in delay so it would wait for the initial topic load?
推荐答案
处理在 Kafka Streams 中是时间同步的.因此，表输入主题和流输入主题是根据记录时间戳顺序处理的.这在语义上是合理的，因为在流表连接中，您不想将流记录与旧版本或新版本的 KTable 连接，而是基于基于的正确版本流记录时间戳.
Processing is time synchronized in Kafka Streams. Hence, the table input topic and stream input topic are processed based on record timestamp order. This is semantically sound, because on a stream-table join, you don't want to join a stream record with an older version nor with a newer version of the KTable, but with the right version based on the stream record timestamp.
如果您的数据没有正确的时间戳，您可以尝试为 via builder.table(..., Consumed.with(...)) 指定自定义时间戳提取器以返回时间戳确保正确的行为(即，可能小于第一个流记录的时间戳?)
If your data is not properly timestamped, you can try to specify a custom timestamp extractor for via builder.table(..., Consumed.with(...)) to return timestamps that ensure proper behavior (ie, maybe smaller than timestamp of the first stream record?)
请注意，正确的时间戳同步需要 Kafka Streams 2.1.旧版本仅以尽力而为的方式同步时间，可能无法提供您想要的行为.有关更多详细信息，请参阅 KIP-353.
Note, that a proper timestamp synchronization requires Kafka Streams 2.1. Older version synchronize time in best effort manner only and may not provide the behavior you want. For more details, see KIP-353.
Kafka 3.0 提供了更多时间戳同步改进:https://cwiki.apache.org/confluence/display/KAFKA/KIP-695%3A+Further+Improve+Kafka+Streams+Timestamp+Synchronization
Kafka 3.0 ships with more timestamp synchronization improvements: https://cwiki.apache.org/confluence/display/KAFKA/KIP-695%3A+Further+Improve+Kafka+Streams+Timestamp+Synchronization
这篇关于Kafka Streams 可以配置为等待 KTable 加载吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
