有没有办法从 .NET 使用 Kafka Ksql Push 查询 [英] Is there a way to consume a Kafka Ksql Push query from .NET
问题描述
我目前正在 .NET 中与 Kafka 消费者一起处理大量 Kafka 消息.
I'm processing a large volume of Kafka messages in .NET with a Kafka consumer at the moment.
我处理的第 1 步是解析 JSON 并根据 JSON 中特定字段的值丢弃许多消息.
Step 1 in my processing is to Parse the JSON and to discard many of the messages based on the value of a specific field in the JSON.
我不想首先处理(特别是不下载)那些不需要的消息.
I'd like to not process (and specifically, not download) those unwanted messages in the first place.
它看起来像一个 kSql 查询——写成推送查询——可以有效地过滤掉我需要处理的消息.
It looks like a kSql query - written as a push query - can effectively filter out the messages I need to process.
我如何通过 .NET 使用这些?我看到一些文档提到了 REST API,但我怀疑这是一个好主意,我需要在一天的高峰时间每分钟处理超过 100 000 条记录.(如果我可以有选择地下载和处理消息,我只会正在处理当前卷的大约三分之一.)
How can I consume these via .NET though? I saw some documents mentioning a REST API, but I doubt that this is a good idea, I need to process in excess of 100 000 records per minute at peak times of day.( If I can selectively download and process messages, I will only be processing about one third of the current volume.)
不幸的是,我无法控制发布者,因此我无法更改消息的发布内容/方式.
Unfortunately I don't have control over the publisher, so I can't change what/how the messages are published.
推荐答案
是的,你可以使用 ksqlDB 来做到这一点
Yes, you can use ksqlDB to do this
-- Declare a stream on the source topic
-- Because it's JSON you'll need to specify the schema
CREATE STREAM my_source (COL1 VARCHAR, COL2 INT)
WITH (KAFKA_TOPIC='my_source_topic', VALUE_FORMAT='JSON');
-- Apply the filter to the stream, with the results written
-- to a new stream (backed by a new topic)
CREATE STREAM target WITH (KAFKA_TOPIC='my_target_topic') AS
SELECT * FROM my_source WHERE COL1='FOO';
然后使用REST API 在您的应用程序中运行一个推送查询,它将只使用过滤后的消息:
Then using the REST API from within your application run a push query which will consume just the filtered messages:
SELECT * FROM target EMIT CHANGES;
<小时>
除了 ksqlDB,您可能还想看看社区最近发布的这个项目:https://github.com/LGouellec/kafka-streams-dotnet
这篇关于有没有办法从 .NET 使用 Kafka Ksql Push 查询的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!