在Kafka Streams作业中进行数据库同步查询或静态调用是否是一种好习惯? [英] Is it a good practice to do sync database query or restful call in Kafka streams jobs?

问题描述

我使用Kafka流来处理实时数据，在Kafka流任务中，我需要访问MySQL来查询数据，并且需要调用其他静态服务.

I use Kafka streams to process real-time data, in the Kafka streams tasks, I need to access MySQL to query data, and need to call another restful service.

所有操作都是同步的.

恐怕同步调用会降低流任务的处理能力.

I'm afraid the sync call will reduce the process capability of the streams tasks.

这是一个好习惯吗?或有什么好主意吗?

Is this a good practice? or Is there any good idea to do this?

推荐答案

一种更好的方法是将您的MySQL表流式传输到Kafka中，然后在其中访问数据.这具有将您的流应用程序与MySQL数据库解耦的优势.如果您将来不再使用MySQL，只要数据仍然从其随后的住处写入Kafka主题，您的流应用就不会受到影响.如果只是在MySQL中存储的配置，您甚至可以采用某些人使用Kafka作为数据的 primary 存储的模式(使用日志压缩，以永久保存它).

A better way to do it would be to stream your MySQL table(s) into Kafka, and access the data there. This has the advantage of decoupling your streams app from the MySQL database. If you moved away from MySQL in the future, so long as the data were still written to the Kafka topic from wherever it subsequently lived, your streams app would be unaffected. If it's just configurations you're storing in MySQL, you could even adopt the pattern that some people use of using Kafka as the primary store for data (using log compaction, to retain it forever).

这篇关于在Kafka Streams作业中进行数据库同步查询或静态调用是否是一种好习惯?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！