从波束管道连接google cloud sql postgres实例 [英] connect google cloud sql postgres instance from beam pipeline
本文介绍了从波束管道连接google cloud sql postgres实例的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我想从运行在google dataflow上的apache Beam管道连接google cloud sql postgres实例.
我想使用Python SDK进行此操作.
我无法为此找到合适的文档.
在Cloud SQL中如何进行指导,我看不到任何有关数据流的文档.
https://cloud.google.com/sql/docs/postgres/
I want to connect google cloud sql postgres instance from apache beam pipeline running on google dataflow.
I want to do this using Python SDK.
I am not able to find proper documentation for this.
In cloud SQL how to guide I dont see any documentation for dataflow.
https://cloud.google.com/sql/docs/postgres/
有人可以提供文档链接/github示例吗?
Can someone provide documentation link/github example?
推荐答案
您可以使用 relational_db.Write 和 relational_db.Read 从首先安装梁形块:
pip install beam-nuggets
阅读:
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from beam_nuggets.io import relational_db
with beam.Pipeline(options=PipelineOptions()) as p:
source_config = relational_db.SourceConfiguration(
drivername='postgresql+pg8000',
host='localhost',
port=5432,
username='postgres',
password='password',
database='calendar',
)
records = p | "Reading records from db" >> relational_db.Read(
source_config=source_config,
table_name='months',
)
records | 'Writing to stdout' >> beam.Map(print)
写作:
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from beam_nuggets.io import relational_db
with beam.Pipeline(options=PipelineOptions()) as p:
months = p | "Reading month records" >> beam.Create([
{'name': 'Jan', 'num': 1},
{'name': 'Feb', 'num': 2},
])
source_config = relational_db.SourceConfiguration(
drivername='postgresql+pg8000',
host='localhost',
port=5432,
username='postgres',
password='password',
database='calendar',
create_if_missing=True,
)
table_config = relational_db.TableConfiguration(
name='months',
create_if_missing=True
)
months | 'Writing to DB' >> relational_db.Write(
source_config=source_config,
table_config=table_config
)
这篇关于从波束管道连接google cloud sql postgres实例的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文