如何使用Cloud Composer下载和访问文件? [英] How can I download and access files using Cloud Composer?
问题描述
我有一些与文件相关的用例，我不确定如何最好地使用Cloud Composer完成.我应该如何最好地完成这些任务?
I have a few file-related use cases that I'm not sure how to best accomplish using Cloud Composer. How should I best accomplish these?
1)我需要使用私钥(.pem)文件来访问SFTP服务器.该文件应该存储在哪里以及如何访问?在本地Airflow中，我只将文件放在与/dags/相同目录下的/keys/文件夹中.
1)I need to use a private key (.pem) file to access an SFTP server. Where should this file be stored and how should it be accessed? In on-prem Airflow, I would just have the file in a folder /keys/ in the same directory as /dags/.
2)我需要将文件从SFTP服务器移至Cloud Storage.借助Airflow，我将这些文件从SFTP服务器下载到Airflow工作程序实例上的特定位置，然后从那里上传.我是否可以使用Composer做类似的事情，还是因为我无法访问文件系统而有一种解决方法?
2)I need to move files from an SFTP server to Cloud Storage. With Airflow on prem, I download these from the SFTP server to a specific location on the Airflow worker instance and then upload from there. Am I able to do something similar with Composer, or is there a workaround as I am unable to access the file system?
推荐答案
1)假定仅需要在任务运行时访问.pem文件(与DAG定义解析时间相对)，您可以将其放在/data中环境的云存储分区的目录.它通过保险丝安装在/home/airflow/gcs/data路径上.您可以使用Cloud Composer gcloud组件.
1) Assuming the .pem file only needs to be accessed at task runtime (as opposed to DAG definition parse time), you can put it in the /data directory of the environment's Cloud Storage bucket. It is mounted with fuse on the path /home/airflow/gcs/data. You can upload files with the Cloud Composer gcloud component.
2)这里有2个选项.
2) There are 2 options here.
-
从SFTP服务器写入到/home/airflow/gcs/data，该文件通过保险丝安装到Cloud Storage存储桶中.您可以将其保留在此处或使用 GoogleCloudStorageToGoogleCloudStorageOperator 将其移动到您真正想要的位置.
Write from your SFTP server to /home/airflow/gcs/data, which is fuse mounted to your Cloud Storage bucket. You could leave it there or use the GoogleCloudStorageToGoogleCloudStorageOperator to move it to where you really want it.
如果要复制到本地磁盘以及从本地磁盘复制到Cloud Storage，则需要在同一任务中执行两个步骤(由于Cloud Composer环境使用CeleryExecutor，因此同一DAG中的任务不是"t保证在同一台机器上运行).您应该能够写到/home/airflow和/tmp.
If you want to copy to local disk and from local disk to Cloud Storage, you'll need to do both steps within the same task (since Cloud Composer environments use the CeleryExecutor, tasks within the same DAG aren't guaranteed to run on the same machine). You should be able to write to /home/airflow and /tmp.
这篇关于如何使用Cloud Composer下载和访问文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
