直接从 url 上传文件到 S3 Bucket [英] Upload files to S3 Bucket directly from a url

问题描述

我们需要将视频文件存储移动到 AWS S3.旧位置是一个 cdn，所以我只有每个文件的 url(1000 多个文件，> 1TB 总文件大小).直接在存储服务器上运行上传工具不是一种选择.

We need to move our video file storage to AWS S3. The old location is a cdn, so I only have url for each file (1000+ files, > 1TB total file size). Running an upload tool directly on the storage server is not an option.

我已经创建了一个工具，可以下载文件，将文件上传到 S3 存储桶，并使用新的 HTTP url 更新数据库记录，并且工作完美，但需要永远.

I already created a tool that downloads the file, uploads file to S3 bucket and updates the DB records with new HTTP url and works perfectly except it takes forever.

下载文件需要一些时间(考虑到每个文件接近 1 GB)并且上传需要更长的时间.

Downloading the file takes some time (considering each file close to a gigabyte) and uploading it takes longer.

是否可以将视频文件直接从 cdn 上传到 S3，这样我可以将处理时间减少一半?类似于读取文件块，然后在读取下一个块的同时将其放入 S3.

Is it possible to upload the video file directly from cdn to S3, so I could reduce processing time into half? Something like reading chunk of file and then putting it to S3 while reading next chunk.

目前我使用 System.Net.WebClient 下载文件和 AWSSDK 上传.

Currently I use System.Net.WebClient to download the file and AWSSDK to upload.

PS:我对互联网速度没有问题，我在具有 1GBit 网络连接的服务器上运行该应用程序.

PS: I have no problem with internet speed, I run the app on a server with 1GBit network connection.

推荐答案

这个问题我已经在这个问题中回答了，这里是要点:

This has been answered by me in this question, here's the gist:

object = Aws::S3::Object.new(bucket_name: 'target-bucket', key: 'target-key')
object.upload_stream do |write_stream|
  IO.copy_stream(URI.open('http://example.com/file.ext'), write_stream)
end

这篇关于直接从 url 上传文件到 S3 Bucket的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！