扶手：如何从S3文件发送到远程服务器 [英] Rails: How to send file from S3 to remote server

问题描述

我一直在狩猎周围，似乎无法找到一个很好的解决方案。我的Rails应用程序商店是文件在Amazon S3中。我现在需要把它们发送到远程（第三方）的服务。

I've been hunting around and can't seem to find a good solution for this. My Rails app stores it's files in Amazon S3. I now need to send them to a remote (3rd party) service.

我用RestClient张贴到第三方服务器是这样的：

I'm using RestClient to post to the 3rd party server like this:

send_file = RestClient::Request.execute(
    :method => :post,
    :url => "http://remote-server-url.com",
    :payload => File.new("some_local_file.avi", 'rb'),
    :multipart => true,
    etc.... )

它适用于本地文件，但我怎么能发送一个远程文件从S3直接向这个第三方服务？

It works for local files, but how can I send a remote file from S3 directly to this 3rd party service?

我找到了答案在这里使用开放式的URI，其中有人是：红宝石读取文件从S3开放-URI

I found an answer here where someone was using open-uri: ruby reading files from S3 with open-URI

我测试了一下我自己，和它的工作。

I tested that for myself, and it worked.

:payload => open(URI.parse("http://amazon-s3-example.com/some_file.avi"))

不过，我已经在这里阅读评论，指出开放-URI简单地加载远程文件到内存中。查看这个答案最后的评论： http://stackoverflow.com/a/264239/2785592

这不是很理想，因为我可能处理大型视频文件。我也看到某处RestClient负荷甚至本地文件到内存中;再次，这是不理想的。有谁知道，如果这是真的吗？

This wouldn't be ideal, as I'm handling potentially large video files. I've also read somewhere the RestClient loads even local files into memory; again, this isn't ideal. Does anyone know if that's true?

当然，我不可能是唯一一个有这个问题。我知道我可以在发送之前本地下载S3文件，但我希望，以节省时间和放大器;带宽。另外，如果RestClient真正的确实的负荷，即使本地文件存储，不是下载在本地不保存我什么。嘿嘿。

Surely I can't be the only one that has this problem. I know I could download the S3 file locally before sending it, but I was hoping to save on time & bandwidth. Also, if RestClient truly does load even local files to memory, than downloading it locally doesn't save me anything. Heh heh.

任何意见将是多少AP preciated。感谢：）

Any advice would be much appreciated. Thanks :)

更新： 远程服务器只是响应POST请求的API。我没有改变自己的东西到底的能力。

Update: The remote server is just an API that responds to post requests. I don't have the ability to change anything on their end.

推荐答案

看看： 的https://github.com/rest-client/rest-client/blob/master/lib/restclient/payload.rb

RestClient绝对支持流媒体上传。条件是，在有效载荷传递的东西是不是字符串或哈希，你传递的东西响应读取和大小。 （所以基本上流）。

RestClient definitely supports streamed uploads. The condition is that in payload you pass something that is not a string or a hash, and that something you pass in responds to read and size. (so basically a stream).

在S3的一面，你基本上需要抓住一个流，发送之前没有看过整个对象。您可以使用的http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#get_object-instance_method你说你想要得到的响应目标（而不是字符串）的IO对象。出于这个目的，你可以使用IO.pipe

On the S3 side, you basically need to grab a stream, not read the whole object before sending it. You use http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#get_object-instance_method and you say you want to get an IO object in the response target (not a string). For this purpose you may use an IO.pipe

reader, writer = IO.pipe

fork do 
    reader.close
    s3.get_object(bucket: 'bucket-name', key: 'object-key') do |chunk|
      writer.write(chunk)
    end
end

writer.close

您传递读者的RestClient :: Payload.generate并使用它作为你的有效载荷。如果读出部分比写入部较慢您可能仍然读了很多在内存中。你想，写只做接受的数额时，你愿意在内存中缓冲。可以读取与writer.stat.size流（叉内）的大小和旋转上它一旦获得过去一定大小

you pass in the reader to the RestClient::Payload.generate and use that as your payload. If the reading part is slower than the writing part you may still read a lot in memory. you want, when writing to only do accept the amount you are willing to buffer in memory. You can read the size of the stream with writer.stat.size (inside the fork) and spin on it once it gets past a certain size.

这篇关于扶手：如何从S3文件发送到远程服务器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！