以编程方式将文件从 Azure Blob Storage 传输到 Google Cloud Storage [英] Transfer file from Azure Blob Storage to Google Cloud Storage programmatically

问题描述

我有许多文件通过 Azure 数据工厂传输到 Azure Blob 存储.不幸的是，这个工具似乎没有为任何值设置 Content-MD5 值，所以当我从 Blob Storage API 中提取该值时，它是空的.

我的目标是将这些文件从 Azure Blob 存储转移到 Google 存储.我在

我们使用它将大约 1TB 的文件从 Azure Blob 存储传输到 Google Cloud Storage.我们还每天刷新一次，因此 Azure Blob 中的任何新文件都会自动复制到 Cloud Storage.

I have a number of files that I transferred into Azure Blob Storage via the Azure Data Factory. Unfortunately, this tool doesn't appear to set the Content-MD5 value for any of the values, so when I pull that value from the Blob Storage API, it's empty.

I'm aiming to transfer these files out of Azure Blob Storage and into Google Storage. The documentation I'm seeing for Google's Storagetransfer service at https://cloud.google.com/storage/transfer/reference/rest/v1/TransferSpec#HttpData indicates that I can easily initiate such a transfer if I supply a list of the files with their URL, length in bytes and an MD5 hash of each.

Well, I can easily pull the first two from Azure Storage, but the third doesn't appear to automatically get populated by Azure Storage, nor can I find any way to get it to do so.

Unfortunately, my other options look limited. In the possibilities so far:

1. Download file to local machine, determine the hash and update the Blob MD5 value
2. See if I can't write an Azure Functions app in the same region that can calculate the hash value and write it to the blob for each in the container
3. Use an Amazon S3 egress from Data Factory and then use Google's support for importing from S3 to pull it from there, per https://cloud.google.com/storage/transfer/reference/rest/v1/TransferSpec#AwsS3Data but this really seems like a waste of bandwidth (and I'd have to set up an Amazon account).

Ideally, I want to be able to write a script, hit go and leave it alone. I don't have the fastest download rate from Azure, so #1 would be less than desireable as it'd take a long time.

Have any other approaches?

解决方案

May 2020 update: Google Cloud Data Transfer now supports Azure Blob storage as a source. This is a no-code solution.

We used this to transfer ~ 1TB of files from Azure Blob storage to Google Cloud Storage. We also have a daily refresh so any new files in Azure Blob are automatically copied to Cloud Storage.

这篇关于以编程方式将文件从 Azure Blob Storage 传输到 Google Cloud Storage的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！