限制 ruby​​ 文件流的速率 [英] Rate limiting a ruby file stream

问题描述

我正在从事一个项目，该项目涉及将 Flash 视频文件从多个地理分布的节点上传到 S3 存储桶.

I am working on a project which involves uploading flash video files to a S3 bucket from a number of geographically distributed nodes.

视频文件每个大约 2-3mb，我们每十分钟只发送一个文件(每个节点)，但是我们消耗的带宽需要限制在 ~20k/s，因为这些节点正在传输流媒体到 CDN，由于位置的原因，我们最多只能上传 512k.

The video files are about 2-3mb each, and we are only sending one file (per node) every ten minutes, however the bandwidth we consume needs to be rate limited to ~20k/s, as these nodes are delivering streaming media to a CDN, and due to the locations we are only able to get 512k max upload.

我一直在研究 ASW-S3 gem，虽然它不提供任何类型的速率限制，但我知道您可以传入 IO 流.鉴于此，我想知道是否有可能创建一个覆盖 read 方法的速率限制流，添加速率限制逻辑(例如，以最简单的形式调用 sleep 读取之间)，然后调用被覆盖方法的超级.

I have been looking into the ASW-S3 gem and while it doesn't offer any kind of rate limiting I am aware that you can pass in a IO Stream. Given this I am wondering if it might be possible to create a rate-limited stream which overrides the read method, adds in the rate limiting logic (e.g. in its simplest form a call to sleep between reads) and then call out to the super of the overridden method.

我考虑的另一个选择是破解 Net::HTTP 的代码并将速率限制放入使用 while 循环的 send_request_with_body_stream 方法中，但我不完全确定哪个是最好的选择.

Another option I considered is hacking the code for Net::HTTP and putting the rate limiting into the send_request_with_body_stream method which is using a while loop, but I'm not entirely sure which would be the best option.

我曾尝试扩展 IO 类，但这根本不起作用，只是从带有 class ThrottledIO < 的类继承而来.IO 什么也没做.

I have attempted at extending the IO class, however that didn't work at all, simply inheriting from the class with class ThrottledIO < IO didn't do anything.

任何建议将不胜感激.

推荐答案

如果你想扩充"一个 IO，你需要使用 Delegate.这会在 IO 对象周围放置一个外观"，该对象的所有外部"读取器都将使用该外观"，但不会影响对象本身的操作.

You need to use Delegate if you want to "augment" an IO. This puts a "facade" around your IO object that will be used by all "external" readers of the object but will have no effect on the operation of the object itself.

我把它提取到一个 gem 中，因为它被证明是普遍有用的

I've extracted that into a gem since it proved to be generally useful

这是一个从 IO 读取的示例

Here's an example for an IO that gets read from

http://rubygems.org/gems/progressive_io

这里有一个方面添加到所有阅读方法中.我认为你可以扩展它来做基本的节流.完成后，您将能够将您的文件(例如文件)包装到其中:

Here there is an aspect added to all reading methods. I think you might be able to extend that to do basic throttling. After you are done you will be able to wrap your, say, File, into it:

 throttled_file = ProgressiveIO.new(some_file) do | offset, size |
    # compute rate and if needed sleep()
 end

这篇关于限制 ruby​​ 文件流的速率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！