Kafka Streams输出主题可以位于单独的群集上吗? [英] Can Kafka Streams output topic be on a separate cluster?

问题描述

我有一个主题，其中所有日志都被推送到集中主题，但我希望将其中一些记录过滤到一个单独的主题中，并在可能的情况下进行聚类.

I have a topic where all logs are pushed to centralized topic but I would like to filter out some of those records to a separate topic and cluster if possible.

谢谢

推荐答案

Kafka流不允许创建具有来自不同Kafka群集的源和输出主题的流.因此以下代码对您不起作用

Kafka streams not allow to create stream with source and output topics from different Kafka clusters. So the following code will not work for you

streamsBuilder.stream(sourceTopicName).filter(..).to(outputTopicName)

在这种情况下，它期望outputTopicName与主题sourceTopicName来自同一群集.

in this case it expects that outputTopicName is from the same cluster as topic sourceTopicName.

一种解决方法，为了将消息从另一个集群发送到输出主题，您可以使用另外创建的具有属性bootstrap.servers的KafkaProducer，该属性将指向外部集群和KStream.foreach()方法.

As a workaround, in order to send messages into output topic from another cluster, you could use additionally created KafkaProducer with property bootstrap.servers that will point to external cluster and KStream.foreach() method.

streamsBuilder.stream(sourceTopicName)
    .filter((key, value) -> ..)
    .foreach((key, value) -> 
        sendMessage(kafkaProducerFromAnotherCluster, destinationTopicName, key, value);


public static void sendMessage(KafkaProducer<String, String> kafkaProducer, 
                               String destinationTopicName, String key, String value) {
    try {
        kafkaProducer.send(new ProducerRecord(destinationTopicName, key, value));
    } catch (RuntimeException ex) {
        log.error(errorMessage, ex);
    }
}

另一个选项是在您的Kafka集群中创建输出主题，该主题将过滤邮件并设置

Another option is to create output topic in your Kafka cluster that will have filtered messages and setup Kafka Mirroring between two clusters (so messages will be copied from one topic to second from another cluster).

这篇关于Kafka Streams输出主题可以位于单独的群集上吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！