无法使用awscli 1.5.4复制大文件(5 Gb) [英] Cannot copy large (5 Gb) files with awscli 1.5.4
问题描述
我在使用aws-cli时遇到问题，我做了一个yum更新，它更新了awscli(等等)，现在awscli在带有SignatureDoesNotMatch的大文件(例如5.1 Gb)上失败了.与较小文件完全相同的命令(到相同存储桶)有效.如果我使用python的boto，大文件仍然有效.
它会复制所有零件，但看起来像是两个零件(即最多计算745个零件中的743个零件)，然后出现错误消息.
看起来像awscli中的错误吗?当我在Google周围搜寻时，找不到任何有关它的信息.
这都是在ec2实例上执行的.
$/usr/bin/aws --versionaws-cli/1.5.4 Python/2.6.9 Linux/3.14.20-20.44.amzn1.x86_64$ ls -hl-rw-rw-r-- 1个用户user 4 Nov 4 21:14 small-rw-rw-r-- 1个用户5.1G用户11月4日20:02大$/usr/bin/aws s3 cp小s3://mybucket/uploaded.ext上传:./small到s3://mybucket/uploaded.ext$/usr/bin/aws s3 cp big s3://mybucket/uploaded.ext上传失败:./big到s3://mybucket/uploaded.ext调用UploadPart操作时发生客户端错误(SignatureDoesNotMatch):我们计算出的请求签名与您提供的签名不匹配.检查您的密钥和签名方法. 以为我找到了，但是仍然有这个问题.我宁愿坚持使用awscli，因为这是我们在整个代码中使用的，并且它比boto快得多(至少在工作时有效)
Edit2:它实际上变得越来越糟，我可以在控制台中重复相同的复制命令，有时会通过，有时却不会
aws-cli 可以当然可以处理5GB以上的上传.
我启动了一个EC2实例，并能够成功完成它:
$/usr/bin/aws --versionaws-cli/1.3.6 Python/2.6.9 Linux/3.10.35-43.137.amzn1.x86_64$ dd if =/dev/zero of = small bs = 1K count = 11 + 0条记录1 + 0条记录复制了1024个字节(1.0 kB)，0.000388469 s，2.6 MB/s$ dd if =/dev/zero of = big bs = 5M count = 1000中有1000 + 0条记录1000 + 0条记录复制了5242880000字节(5.2 GB)，73.733 s，71.1 MB/s$ aws s3 mb s3://target77make_bucket:s3://target77/$ aws s3 cp小s3://target77/small上传:./small到s3://target77/small$ aws s3 cp大s3://target77/big上传:./big到s3://target77/big$ aws s3 ls s3://target77/2014-11-17 09:44:42 5242880000大2014-11-17 09:44:29 1024小 到目前为止，太好了！
然后我更新到了更高版本的CLI，然后重试:
$ sudo yum更新...$/usr/bin/aws --versionaws-cli/1.5.5 Python/2.6.9 Linux/3.10.35-43.137.amzn1.x86_64$ aws s3 cp小s3://target77/small上传:./small到s3://target77/small$ aws s3 cp大s3://target77/big上传失败:./big至s3://target77/bigs)调用UploadPart操作时发生客户端错误(SignatureDoesNotMatch):我们计算出的请求签名与您提供的签名不匹配.检查您的密钥和签名方法. 我在相同区域和不同区域中使用EC2实例和S3存储桶重复了该实验.两种情况最终都会导致错误，但有时也可以解决.
因此，更新的aws-cli似乎确实存在问题.另外，当我尝试重新执行复制命令时，在随机数的份数"之后发生了错误./p>
SignatureDoesNotMatch 错误与安全凭证而不是Amazon S3服务有关.这听起来类似于人们在GitHub上报告的问题:重复 SignatureDoesNotMatch客户端错误#995
因此，我建议使用早期版本的aws-cli，直到问题解决.
I have problem with aws-cli, I did a yum update, it updated awscli (among other things) and now awscli fails on large files (e.g. 5.1 Gb) with SignatureDoesNotMatch. The exact same command (to same bucket) with smaller files works. The big file still works if I use boto from python.
It copies all parts but two it looks like (i.e. it was counted up to 743 of 745 parts), and then the error message comes.
Looks like a bug in awscli? I could not find anything about it when I google around though.
This is all executed on an ec2 instance.
$ /usr/bin/aws --version
aws-cli/1.5.4 Python/2.6.9 Linux/3.14.20-20.44.amzn1.x86_64
$ ls -hl
-rw-rw-r-- 1 user user 4 Nov 4 21:14 small
-rw-rw-r-- 1 user user 5.1G Nov 4 20:02 big
$ /usr/bin/aws s3 cp small s3://mybucket/uploaded.ext
upload: ./small to s3://mybucket/uploaded.ext
$ /usr/bin/aws s3 cp big s3://mybucket/uploaded.ext
upload failed: ./big to s3://mybucket/uploaded.ext
A client error (SignatureDoesNotMatch) occurred when calling the UploadPart operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
Edit: thought I found it, but still have this problem. I would prefer to stick with awscli since it is what we use all over the code, and it is considerable faster than boto (at least when it works)
Edit2: it actually gone from bad to worse, I can repeat the same copy command in the console, sometimes it goes through, sometimes it does not
The aws-cli can certainly handle uploads of 5GB and higher.
I launched an EC2 instance and was able to complete it successfully:
$ /usr/bin/aws --version
aws-cli/1.3.6 Python/2.6.9 Linux/3.10.35-43.137.amzn1.x86_64
$ dd if=/dev/zero of=small bs=1K count=1
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000388469 s, 2.6 MB/s
$ dd if=/dev/zero of=big bs=5M count=1000
1000+0 records in
1000+0 records out
5242880000 bytes (5.2 GB) copied, 73.733 s, 71.1 MB/s
$ aws s3 mb s3://target77
make_bucket: s3://target77/
$ aws s3 cp small s3://target77/small
upload: ./small to s3://target77/small
$ aws s3 cp big s3://target77/big
upload: ./big to s3://target77/big
$ aws s3 ls s3://target77/
2014-11-17 09:44:42 5242880000 big
2014-11-17 09:44:29 1024 small
So far, so good!
I then updated to a newer version of the CLI and tried again:
$ sudo yum update
...
$ /usr/bin/aws --version
aws-cli/1.5.5 Python/2.6.9 Linux/3.10.35-43.137.amzn1.x86_64
$ aws s3 cp small s3://target77/small
upload: ./small to s3://target77/small
$ aws s3 cp big s3://target77/big
upload failed: ./big to s3://target77/bigs) remaining
A client error (SignatureDoesNotMatch) occurred when calling the UploadPart operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I repeated the experiment with the EC2 instance and S3 bucket in the same region, and in different regions. Both situations eventually gave the error, but also sometimes worked.
So, it does seem to be an issue with the updated aws-cli. Plus, the error occurred after a random number of 'parts' when I tried re-executing the copy command.
The SignatureDoesNotMatch error is related to security credentials rather than the Amazon S3 service. It sounds similar to issues that people have reported on GitHub: Recurring SignatureDoesNotMatch client error #995
So, I'd recommend using an earlier version of aws-cli until the issue is resolved.
这篇关于无法使用awscli 1.5.4复制大文件(5 Gb)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
