由于身份验证,gsutil无法复制到s3 [英] gsutil cannot copy to s3 due to authentication
问题描述
我需要将许多(1000+)个文件从GCS复制到s3,以利用AWS lambda函数.我已经编辑了~/.boto.cfg
并注释掉了2个aws身份验证参数,但是一个简单的gsutil ls s3://mybucket
在 GCE 或 EC2 VM中失败.
错误为The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
.
我使用的是gsutil版本:4.28
, GCS 和 S3 存储桶的位置分别是US-CENTRAL1
和US East (Ohio)
-如果需要的话.>
我一无所知,因为AWS密钥是有效的,并且启用了http/https.使用笔记本电脑的Cyberduck从GCS下载并上传到S3是不可行的(> 230Gb)
根据 https://issuetracker. google.com/issues/62161892 ,gsutil
v4.28 确实支持AWS
v4签名,方法是在~/.boto
中添加新的[s3]
部分,例如
[s3]
# Note that we specify region as part of the host, as mentioned in the AWS docs:
# http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
host = s3.eu-east-2.amazonaws.com
use-sigv4 = True
该部分的使用是从boto3
继承的,但是当前不是由gsutil config
创建的,因此需要为目标端点显式添加它.
对于s3-to-GCS,我将考虑使用更少服务器的Storage Transfer Service
API.
I need to copy many (1000+) files to s3 from GCS to leverage an AWS lambda function. I have edited ~/.boto.cfg
and commented out the 2 aws authentication parameters but a simple gsutil ls s3://mybucket
fails from either an GCE or EC2 VM.
Error is The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
.
I use gsutil version: 4.28
and locations of GCS and S3 bucket are respectively US-CENTRAL1
and US East (Ohio)
- in case this is relevant.
I am clueless as the AWS key is valid and I enabled http/https. Downloading from GCS and uploading to S3 using my laptop's Cyberduck is impracticable (>230Gb)
As per https://issuetracker.google.com/issues/62161892, gsutil
v4.28 does support AWS
v4 signatures by adding to ~/.boto
a new [s3]
section like
[s3]
# Note that we specify region as part of the host, as mentioned in the AWS docs:
# http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
host = s3.eu-east-2.amazonaws.com
use-sigv4 = True
The use of that section is inherited from boto3
but is currently not created by gsutil config
so it needs to be added explicitly for the target endpoint.
For s3-to-GCS, I will consider the more server-less Storage Transfer Service
API.
这篇关于由于身份验证,gsutil无法复制到s3的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!