通过Kafka Streams DSL多次使用同一主题作为来源 [英] Use the same topic as a source more than once with Kafka Streams DSL

问题描述

在使用Kafka Streams DSL时，是否可以将相同的主题用作两个不同处理例程的来源?

Is there a way to use the same topic as the source for two different processing routines, when using Kafka Streams DSL?

StreamsBuilder streamsBuilder = new StreamsBuilder();

// use the topic as a stream
streamsBuilder.stream("topic")...

// use the same topic as a source for KTable
streamsBuilder.table("topic")...

return streamsBuilder.build();

上面的

天真的实现会在运行时抛出 TopologyException :无效的拓扑:主题已经被另一个来源注册.如果我们深入研究底层，则完全有效.处理器API.使用它是唯一的出路吗?

Naive implementation from above throws a TopologyException at runtime: Invalid topology: Topic topic has already been registered by another source. Which is totally valid, if we dive into underlying Processor API. Is using it the only way out?

更新:到目前为止，我找到的最接近的替代方法:

UPDATE: The closest alternative I've found so far:

StreamsBuilder streamsBuilder = new StreamsBuilder();

final KStream<Object, Object> stream = streamsBuilder.stream("topic");

// use the topic as a stream
stream...

// create a KTable from the KStream
stream.groupByKey().reduce((oldValue, newValue) -> newValue)...

return streamsBuilder.build();

推荐答案

读取与流和表相同的主题在语义上是有问题的恕我直言.流模型不可变的事实，而您将用于读取KTable模型更新的changelog主题.

Reading the same topic as stream and as table is semantically questionable IMHO. Streams model immutable facts, while changelog topic that you would use to read into a KTable model updates.

如果要在多个流中使用单个主题，则可以多次重用同一个 KStream 对象(从语义上讲，类似于广播):

If you want to use a single topic in multiple streams, you can reuse the same KStream object multiple times (it's semantically like a broadcast):

KStream stream = ...
stream.filter();
stream.map();

还要比较: https://issues.apache.org/jira/browse/KAFKA-6687 (我们计划取消此限制.我怀疑，尽管我们会允许同时使用一个主题作为 KStream 和 KTable 比较上面的评论).

Also compare: https://issues.apache.org/jira/browse/KAFKA-6687 (there are plans to remove this restriction. I doubt, we will allow to use one topic as KStream and KTable at the same time though—compare my comment from above).

这篇关于通过Kafka Streams DSL多次使用同一主题作为来源的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！