我该怎么做才能加快S3的上载/更新? [英] What can I do to speed up S3 uploads/updates?

问题描述

我一直试图整天上传一些小到s3的内容. 500个目录中约有2万个文件，总计约3GB.对于称为简单存储服务"的服务而言，这绝对是合理的.我可以平均以大约500k/s-1mb/s(1.8和3.6 gb/h之间)的速度上传到不同的地方.我一直试图整天将这些文件上传到s3，但我必须以令人沮丧的速度上传(总计大约100 mb/h左右).

I have been trying to upload something small to s3 all day today. About 20k files in 500 directories that total about 3GB. Something absolutely reasonable for a service called Simple Storage Service. I can upload to different places on average at about 500k/s - 1mb/s (between 1.8 and 3.6 gb/h). I have been trying to upload these files to s3 all day, I must have uploaded at dismal rate on aggregate (think about 100 mb/h or something).

我尝试过:

• 具有各种操作系统上的各种浏览器的s3网络控制台
• boto使用了我编写的各种脚本，并在网上找到了(主要是在SO上找到的.)

我希望您能帮助我诊断以下问题:

My problems, which I was hoping you would be so kind to help me diagnose are the following:

• 将其拖放到s3控制台(仅用于计数20k文件，大约需要一个小时).为什么?除非我能解决这个问题，否则Web控制台对我几乎毫无用处.
• 上传本身非常慢，很少快于100 k/s.
• 整天上传后，我发现文件名有一个简单的问题，不想整夜上传，我使用了以下脚本:

• dragging and dropping to the s3 console (just for it to count the 20k files, takes like an hour). why? unless I can solve this the web console is mostly useless to me.
• the upload itself is extremely slow, seldomly faster than 100 k/s.
• after all day uploading I noticed a simple problem with the filenames, not wanting to spend all night uploading again, I used this script: Amazon S3 boto: How do you rename a file in a bucket? which everyone claims works really fast. It manages to rename about 1 200kb file every 2-3 seconds. why?
• after uploading, making the all the files public (using the web console) has taken like 4 hours and still it has not finished.

这真令人沮丧，一定是我做错了什么.我希望一切都能以大约10倍的速度运行，但事实并非如此.我已经读过，如果拆分文件s3的运行速度更快，并且我已经阅读到该区域(我在纽约市)非常重要.哪些更改将使我的上传速度获得最大的提高?

It is really frustrating, there must be something I am doing wrong. I expect everything to work about 10x faster and it doesn't. I've read that if split the file s3 runs faster and I've read that the zone (I'm in NYC) is really important. What change will give me the biggest increase in upload speed?

推荐答案

也许可以通过更改AWS服务器位置

我刚刚发现问题是什么:上传持续时间(大小为35MB)

I just figured out, what the problem was in my case: duration of upload (size 35MB)

• 美国西部2州俄勒冈:5-6分钟
• 德国法兰克福，欧洲中央:1分钟！ (大约是我的最大连接数)

我位于维也纳，而不是美国->检查您的AWS服务器位置

I'm based in Vienna, not in the US -> check your AWS server location

这篇关于我该怎么做才能加快S3的上载/更新?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！