在 Kafka 流作业中进行同步数据库查询或宁静调用是一个好习惯吗? [英] Is it a good practice to do sync database query or restful call in Kafka streams jobs?

问题描述

我使用Kafka流处理实时数据，在Kafka流任务中，我需要访问MySQL查询数据，需要调用另一个restful服务.

I use Kafka streams to process real-time data, in the Kafka streams tasks, I need to access MySQL to query data, and need to call another restful service.

所有操作都是同步的.

恐怕同步调用会降低流任务的处理能力.

I'm afraid the sync call will reduce the process capability of the streams tasks.

这是一个好习惯吗?或者这样做有什么好主意吗?

Is this a good practice? or Is there any good idea to do this?

推荐答案

更好的方法是将 MySQL 表流式传输到 Kafka，并访问那里的数据.这具有将流应用程序与 MySQL 数据库分离的优势.如果您将来不再使用 MySQL，只要数据仍从其随后所在的任何位置写入 Kafka 主题，您的流应用程序就不会受到影响.如果它只是你在 MySQL 中存储的配置，你甚至可以采用一些人使用的模式，使用 Kafka 作为数据的主要存储(使用日志压缩，永久保留它).

A better way to do it would be to stream your MySQL table(s) into Kafka, and access the data there. This has the advantage of decoupling your streams app from the MySQL database. If you moved away from MySQL in the future, so long as the data were still written to the Kafka topic from wherever it subsequently lived, your streams app would be unaffected. If it's just configurations you're storing in MySQL, you could even adopt the pattern that some people use of using Kafka as the primary store for data (using log compaction, to retain it forever).

这篇关于在 Kafka 流作业中进行同步数据库查询或宁静调用是一个好习惯吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！