直接从网址将文件上传到S3存储桶 [英] Upload files to S3 Bucket directly from a url

问题描述

我们需要将视频文件存储移至AWS S3.旧位置是CDN，因此我只有每个文件的网址(1000多个文件，文件总大小> 1TB).不能直接在存储服务器上运行上传工具.

We need to move our video file storage to AWS S3. The old location is a cdn, so I only have url for each file (1000+ files, > 1TB total file size). Running an upload tool directly on the storage server is not an option.

我已经创建了一个工具，该工具可以下载文件，将文件上传到S3存储桶并使用新的HTTP URL更新数据库记录，并且除了可以永久使用之外，还可以完美地运行.

I already created a tool that downloads the file, uploads file to S3 bucket and updates the DB records with new HTTP url and works perfectly except it takes forever.

下载文件需要花费一些时间(考虑每个文件接近1 GB)，而上传则需要更长的时间.

Downloading the file takes some time (considering each file close to a gigabyte) and uploading it takes longer.

是否可以将视频文件直接从CDN上传到S3，这样我可以将处理时间减少一半?类似于读取文件块，然后在读取下一个块时将其放到S3中.

Is it possible to upload the video file directly from cdn to S3, so I could reduce processing time into half? Something like reading chunk of file and then putting it to S3 while reading next chunk.

当前，我使用System.Net.WebClient下载文件，并使用AWSSDK上传.

Currently I use System.Net.WebClient to download the file and AWSSDK to upload.

PS:我的互联网速度没有问题，我在具有1GBit网络连接的服务器上运行了该应用程序.

PS: I have no problem with internet speed, I run the app on a server with 1GBit network connection.

推荐答案

这已经由我在这个问题中得到了回答. ，这是要点:

This has been answered by me in this question, here's the gist:

obj = Aws::S3::Object.new(bucket_name: 'target-bucket', key: 'target-key')
obj.upload_stream do |write_stream|
  IO.copy_stream(URI.open('http://example.com/file.ext'), write_stream))
end

这篇关于直接从网址将文件上传到S3存储桶的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！