从 Internet 下载文件到 S3 存储桶 [英] downloading a file from Internet into S3 bucket

问题描述

我想直接从互联网上获取一个文件并将其粘贴到 S3 存储桶中，然后将其复制到 PIG 集群中.由于文件的大小和我不太好的互联网连接，首先将文件下载到我的 PC 上，然后将其上传到亚马逊可能不是一种选择.

I would like to grab a file straight of the Internet and stick it into an S3 bucket to then copy it over to a PIG cluster. Due to the size of the file and my not so good internet connection downloading the file first onto my PC and then uploading it to Amazon might not be an option.

有什么办法可以抓取互联网文件并将其直接粘贴到 S3 中吗?

Is there any way I could go about grabbing a file of the internet and sticking it directly into S3?

推荐答案

[2017 编辑]我在 2013 年给出了原始答案.今天我建议使用 AWS Lambda 下载文件并将其放在 S3 上.这是预期的效果 - 在不涉及服务器的情况下将对象放置在 S3 上.

[2017 edit] I gave the original answer back at 2013. Today I'd recommend using AWS Lambda to download a file and put it on S3. It's the desired effect - to place an object on S3 with no server involved.

【原答案】直接做是不可能的.

[Original answer] It is not possible to do it directly.

为什么不使用 EC2 实例而不是本地 PC 来执行此操作?同区域从EC2上传到S3的速度非常好.

Why not do this with EC2 instance instead of your local PC? Upload speed from EC2 to S3 in the same region is very good.

关于从/到 s3 的流读取/写入我使用 python 的 smart_open

regarding stream reading/writing from/to s3 I use python's smart_open

这篇关于从 Internet 下载文件到 S3 存储桶的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！