如何将流查询的结果写入多个数据库表? [英] How to write result of streaming query to multiple database tables?

问题描述

我正在使用Spark结构化的流媒体，并从Kafka主题中进行阅读.目的是将消息写到PostgreSQL数据库的多个表中.

I am using spark structured streaming and reading from Kafka topic. The goal is to write the message to PostgreSQL database multiple tables.

消息模式为:

 root
  |-- id: string (nullable = true)
  |-- name: timestamp (nullable = true)
  |-- comment: string (nullable = true)
  |-- map_key_value: map (nullable = true)
    |-- key: string
    |-- value: string (valueContainsNull = true)

在删除map_key_value之后写入一个表时，可以使用以下代码:

While writing to one table after dropping map_key_value works with below code:

我的写代码是:

message.writeStream.foreachBatch { (batchDF: DataFrame, batchId: Long) =>
    batchDF.write.format("jdbc").option("url", "url")
      .option("user", "username")
      .option("password", "password")
      .option(JDBCOptions.JDBC_TABLE_NAME, "table_1')
      .mode(SaveMode.Append).save();
  }.outputMode(OutputMode.Append()).start().awaitTermination()

我想将消息写到两个数据库表中，表1(id，名称，注释)和表2需要具有map_key_value.

I want to write the message to two DB tables table 1(id, name, comment) and tables 2 need have the map_key_value.

推荐答案

对于N个接收器，您将需要N个流查询；t1和t2都算作一个单独的接收器.

You will need N streaming queries for N sinks; t1 and t2 both count as a separate sink.

writeStream当前不写入jdbc，因此您应该使用

writeStream does not currently write to jdbc so you should use foreachBatch operator.

这篇关于如何将流查询的结果写入多个数据库表?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！