如何在Amazon-S3和Google Cloud Storage之间执行文件完整性检查 [英] How to perform files integrity check between Amazon-S3 and Google Cloud Storage

问题描述

我正在将数据从Amazon-S3迁移到Google-Cloud Storage. 我已经使用 gsutil :

I am migrating my data from Amazon-S3 to Google-Cloud Storage. I have copied my data using gsutil:

$ gsutil cp -R s3://my_bucket/* gs://my_bucket

我接下来要做的是检查S3中的所有文件在Google存储空间中是否正确存在.

What I want to do next is to check if all the files in S3 is properly exist in Google Storage.

目前，我要做的只是在文件中打印文件列表，然后执行简单的Unix diff，但这实际上并没有检查文件的完整性.

At the moment all I did is to do print file list in file and then do simple Unix diff but that doesn't really check the file integrity.

什么是检查的好方法?

推荐答案

gsutil验证在云提供程序之间复制的对象上的MD5校验和，因此，如果递归复制命令成功完成(shell返回码0)，则您应该已经成功复制了所有内容.请注意，gsutil无法比较大于5 GiB(具有gsutil不支持的非MD5校验和)的S3对象的校验和，并会在遇到情况时显示警告.

gsutil verifies MD5 checksums on objects copied between cloud providers, so if the recursive copy command completes successfully (shell return code 0), you should have copied everything successfully. Note that gsutil isn't able to compare checksums for S3 objects larger than 5 GiB (which have a non-MD5 checksum that gsutil doesn't support), and will print a warning for cases it encounters.

这篇关于如何在Amazon-S3和Google Cloud Storage之间执行文件完整性检查的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！