我们如何从其他存储桶(而不是作曲家存储桶)访问任何配置文件 [英] How we can access any configuration file from other bucket (instead of composer bucket)
问题描述
我是python和google composer的新手. 我正在尝试从python脚本中从Google云存储中读取配置(.properties)文件.配置文件包含密钥&价值对. 我尝试使用configparser读取配置文件&与操作员也正常 如果我们在相同的作曲家环境中拥有相同的文件,那么我们可以给出路径 像'/home/airflow/gcs/dags/config.properties'
I am new in python and google composer. I am trying to read configuration(.properties) file from google cloud storage from my python script. configuration files contain key & value pair. I tried with configparser for reading the config file & normal with operator also If same file we have in same composer environment then we can give the path like '/home/airflow/gcs/dags/config.properties'
但是对于其他存储桶,我可以提供什么路径?
But for other bucket what path I can give ??
我正在尝试使用以下代码访问路径storage_client
I am trying to access path storage_client using below code
separator = "="
keys = {}
def iterate_bucket():
bucket_name = 'other-bucket'
storage_client = storage.Client.from_service_account_json(
'/home/airflow/gcs/data/*************.json')
bucket = storage_client.get_bucket(bucket_name)
blobs = bucket.list_blobs()
return blobs
def read_Prop():
blobs = iterate_bucket()
for blob in blobs:
if "config.properties" in blob.name:
print("hello : ", blob.name)
#file_name = "/home/airflow/gcs/" + blob.name
file_name = blob.name
with open(file_name, 'r') as f:
for line in f:
if separator in line:
name, value = line.split(separator, 1)
keys[name.strip()] = value.strip()
print(keys.get('any_key'))
我也使用了configparser
I used configparser also
config = configparser.ConfigParser()
config.read(blob.name)
在这两个条件下,我的python脚本都无法访问其他Google云存储(存储桶). 我没有得到这样的文件或目录错误
In both the condition for other google cloud storage(bucket) is not accesible from my python script. I am getting no such file or directory error
我们可以提供什么路径或任何其他方式来访问??
What path we can give or any other way to access ??
示例2-
def readYML():
bucket_name = 'external-bucket'
storage_client = storage.Client.from_service_account_json(
'/home/airflow/gcs/data/private-key.json')
bucket = storage_client.get_bucket(bucket_name)
file_name = ""
blobs = bucket.list_blobs()
print("blobs : ",blobs)
for blob in blobs:
if "sample_config.yml" in blob.name:
print("hello : ", blob.name)
file_name = blob.name
print("file_name : ",file_name)
with open(file_name, 'r') as ymlfile:
CFG = yaml.load(ymlfile)
print("inside readYML method : ", CFG['configs']['project_id'])
return CFG
请参阅日志-打印语句中的文件名即将到来,但是当我们读取错误时会发现没有此类文件或目录错误.
see the logs - Here in print statement file name is coming but when we are reading getting error for NO such file or directory error.
[2020-01-02 12:39:12,853] {base_task_runner.py:101} INFO - Job 11782:
Subtask Raw1 [2020-01-02 12:39:12,852] {logging_mixin.py:95} INFO - hello :
sample_config.yml
[2020-01-02 12:39:12,853] {logging_mixin.py:95} INFO - file_name :
sample_config.yml
[2020-01-02 12:39:12,853] {base_task_runner.py:101} INFO - Job 11782:
Subtask Raw1 [2020-01-02 12:39:12,853] {logging_mixin.py:95} INFO -
file_name : sample_config.yml
[2020-01-02 12:39:12,855] {models.py:1846} ERROR - [Errno 2] No such file or
directory: 'sample_config.yml'
预先感谢
推荐答案
我们无法从composer存储桶中访问任何配置文件.我们只能使用存储访问API来访问文件,但是直接我们不能从任何其他存储桶中读取任何属性文件.
We can't access any configuration file from composer bucket. we can only access the file using storage access API but directly we can't read any property file from any other bucket.
这篇关于我们如何从其他存储桶(而不是作曲家存储桶)访问任何配置文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!