使用 Kafka Streams DSL 多次使用相同的主题作为源 [英] Use the same topic as a source more than once with Kafka Streams DSL

问题描述

在使用 Kafka Streams DSL 时，是否可以使用相同的主题作为两个不同处理例程的源?

Is there a way to use the same topic as the source for two different processing routines, when using Kafka Streams DSL?

StreamsBuilder streamsBuilder = new StreamsBuilder();

// use the topic as a stream
streamsBuilder.stream("topic")...

// use the same topic as a source for KTable
streamsBuilder.table("topic")...

return streamsBuilder.build();

上面的简单实现在运行时抛出 TopologyException:无效的拓扑:主题主题已经被另一个源注册.如果我们深入研究底层，这是完全有效的处理器 API.使用它是唯一的出路吗?

Naive implementation from above throws a TopologyException at runtime: Invalid topology: Topic topic has already been registered by another source. Which is totally valid, if we dive into underlying Processor API. Is using it the only way out?

更新:迄今为止我找到的最接近的替代方案:

UPDATE: The closest alternative I've found so far:

StreamsBuilder streamsBuilder = new StreamsBuilder();

final KStream<Object, Object> stream = streamsBuilder.stream("topic");

// use the topic as a stream
stream...

// create a KTable from the KStream
stream.groupByKey().reduce((oldValue, newValue) -> newValue)...

return streamsBuilder.build();

推荐答案

阅读与流和表相同的主题在语义上是有问题的恕我直言.Streams 模型不可变的事实，而您将用来读入 KTable 模型更新的变更日志主题.

Reading the same topic as stream and as table is semantically questionable IMHO. Streams model immutable facts, while changelog topic that you would use to read into a KTable model updates.

如果你想在多个流中使用一个主题，你可以多次重用同一个 KStream 对象(它在语义上就像一个广播):

If you want to use a single topic in multiple streams, you can reuse the same KStream object multiple times (it's semantically like a broadcast):

KStream stream = ...
stream.filter();
stream.map();

还比较:https://issues.apache.org/jira/browse/KAFKA-6687(有计划取消此限制.我怀疑，我们是否会允许同时使用一个主题作为 KStream 和 KTable —比较我上面的评论).

Also compare: https://issues.apache.org/jira/browse/KAFKA-6687 (there are plans to remove this restriction. I doubt, we will allow to use one topic as KStream and KTable at the same time though—compare my comment from above).

这篇关于使用 Kafka Streams DSL 多次使用相同的主题作为源的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！