有没有办法从 .NET 使用 Kafka Ksql Push 查询 [英] Is there a way to consume a Kafka Ksql Push query from .NET

问题描述

我目前正在 .NET 中与 Kafka 消费者一起处理大量 Kafka 消息.

I'm processing a large volume of Kafka messages in .NET with a Kafka consumer at the moment.

我处理的第 1 步是解析 JSON 并根据 JSON 中特定字段的值丢弃许多消息.

Step 1 in my processing is to Parse the JSON and to discard many of the messages based on the value of a specific field in the JSON.

我不想首先处理(特别是不下载)那些不需要的消息.

I'd like to not process (and specifically, not download) those unwanted messages in the first place.

它看起来像一个 kSql 查询——写成推送查询——可以有效地过滤掉我需要处理的消息.

It looks like a kSql query - written as a push query - can effectively filter out the messages I need to process.

我如何通过 .NET 使用这些?我看到一些文档提到了 REST API，但我怀疑这是一个好主意，我需要在一天的高峰时间每分钟处理超过 100 000 条记录.(如果我可以有选择地下载和处理消息，我只会正在处理当前卷的大约三分之一.)

How can I consume these via .NET though? I saw some documents mentioning a REST API, but I doubt that this is a good idea, I need to process in excess of 100 000 records per minute at peak times of day.( If I can selectively download and process messages, I will only be processing about one third of the current volume.)

不幸的是，我无法控制发布者，因此我无法更改消息的发布内容/方式.

Unfortunately I don't have control over the publisher, so I can't change what/how the messages are published.

推荐答案

是的，你可以使用 ksqlDB 来做到这一点

Yes, you can use ksqlDB to do this

-- Declare a stream on the source topic
-- Because it's JSON you'll need to specify the schema
CREATE STREAM my_source (COL1 VARCHAR, COL2 INT) 
  WITH (KAFKA_TOPIC='my_source_topic', VALUE_FORMAT='JSON');

-- Apply the filter to the stream, with the results written
-- to a new stream (backed by a new topic)
CREATE STREAM target WITH (KAFKA_TOPIC='my_target_topic') AS
  SELECT * FROM my_source WHERE COL1='FOO';

然后使用REST API 在您的应用程序中运行一个推送查询，它将只使用过滤后的消息:

Then using the REST API from within your application run a push query which will consume just the filtered messages:

SELECT * FROM target EMIT CHANGES;

<小时>

除了 ksqlDB，您可能还想看看社区最近发布的这个项目:https://github.com/LGouellec/kafka-streams-dotnet

这篇关于有没有办法从 .NET 使用 Kafka Ksql Push 查询的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！