如何在流查询中使用外部数据库(postgresql)作为输入? [英] How to use external database (postgresql) as input in streaming query?

问题描述

我正在尝试在 Postgresql 中实现流输入更新.具体来说，我想在 Spark 的流输入中使用 Postgresql 作为数据源.

看着文档，我不确定这是否可能.

I am trying to implement streaming input updates in Postgresql. Specifically , I would like to use Postgresql as datasource in stream input into spark.

Looking at the document, I was not sure if this is possible or not.

https://spark.apache.org/docs/latest/streaming-programming-guide.html

Would it be possible to stream input from PostgresQL, perhaps as a micro batch ?

解决方案

To stream your PSQL Data as a micro batch, Kafka is the best way. You can use Kafka connect (as a source) to established a connection between your Brokers and your Database.

See Kafka streaming integration with Spark Streaming: https://spark.apache.org/docs/2.2.0/streaming-kafka-0-10-integration.html

Kafka connect documentation: https://docs.confluent.io/current/connect/index.html

By this way, you can face to faults, concurrency and optimize your computation capabilities.

这篇关于如何在流查询中使用外部数据库(postgresql)作为输入?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！