将数据从Google Cloud Storage导出到Amazon S3 [英] Exporting data from Google Cloud Storage to Amazon S3

问题描述

我想将数据从BigQuery中的表格传输到另一个Redshift中。
我的计划数据流如下：

BigQuery - > Google Cloud Storage - > Amazon S3 - > Redshift

我了解Google Cloud Storage Transfer Service，但我不确定它可以帮助我。来自Google Cloud文档：
$ b

云端存储转移服务

此页面描述了云存储传输服务，您可以使用
快速将在线数据导入Google云端存储。

<我知道这项服务可用于将数据导入到Google云端存储中，而不是从中导出数据。

有没有一种方法可以将数据从Google Cloud中导出存储到Amazon S3？

解决方案

您可以使用gsutil将数据从Google Cloud Storage存储桶复制到Amazon存储桶，命令如：

  gsutil -m rsync -rd gs：// your-gcs-bucket s3：// your-s3 -bucket 
  

请注意，上面的-d选项会导致gsutil rsync删除S3存储桶中的对象，不存在于您的GCS存储桶中（除了添加新对象外）。如果您只想从GCS添加新对象到您的S3存储桶，您可以忽略该选项。

I would like to transfer data from a table in BigQuery, into another one in Redshift. My planned data flow is as follows:

BigQuery -> Google Cloud Storage -> Amazon S3 -> Redshift

I know about Google Cloud Storage Transfer Service, but I'm not sure it can help me. From Google Cloud documentation:

Cloud Storage Transfer Service

This page describes Cloud Storage Transfer Service, which you can use to quickly import online data into Google Cloud Storage.

I understand that this service can be used to import data into Google Cloud Storage and not to export from it.

Is there a way I can export data from Google Cloud Storage to Amazon S3?

解决方案

You can use gsutil to copy data from a Google Cloud Storage bucket to an Amazon bucket, using a command such as:

gsutil -m rsync -rd gs://your-gcs-bucket s3://your-s3-bucket

Note that the -d option above will cause gsutil rsync to delete objects from your S3 bucket that aren't present in your GCS bucket (in addition to adding new objects). You can leave off that option if you just want to add new objects from your GCS to your S3 bucket.

这篇关于将数据从Google Cloud Storage导出到Amazon S3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！