从Internet下载文件到S3存储桶 [英] downloading a file from Internet into S3 bucket

问题描述

我想直接从Internet上获取文件并将其粘贴到S3存储桶中，然后将其复制到PIG群集中.由于文件的大小以及我的互联网连接不佳，可能无法选择先将文件下载到我的PC上，然后再将其上传到Amazon.

I would like to grab a file straight of the Internet and stick it into an S3 bucket to then copy it over to a PIG cluster. Due to the size of the file and my not so good internet connection downloading the file first onto my PC and then uploading it to Amazon might not be an option.

有什么办法可以获取互联网文件并将其直接粘贴到S3中?

Is there any way I could go about grabbing a file of the internet and sticking it directly into S3?

推荐答案

[2017编辑] 我在2013年给出了原始答案.今天，我建议使用

[2017 edit] I gave the original answer back at 2013. Today I'd recommend using AWS Lambda to download a file and put it on S3. It's the desired effect - to place an object on S3 with no server involved.

[原始答案] 无法直接执行.

[Original answer] It is not possible to do it directly.

为什么不使用EC2实例而不是本地PC来执行此操作?在同一区域中从EC2到S3的上传速度非常好.

Why not do this with EC2 instance instead of your local PC? Upload speed from EC2 to S3 in the same region is very good.

关于从s3读/写流，我使用python的 smart_open

regarding stream reading/writing from/to s3 I use python's smart_open

这篇关于从Internet下载文件到S3存储桶的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！