在红宝石并行的HTTP请求 [英] Parallel HTTP requests in ruby

问题描述

我有网址的数组，我wan't打开每个之一，获取一个特定的标签。结果
但我想这样做并行。

I have an array of URLs and I wan't to open each one and fetch a specific tag.
But I want to do this in parallel.

下面是伪code什么我想要做的：

Here is the pseudocode for what I want to do:

urls = [...]
tags = []
urls.each do |url|
  fetch_tag_asynchronously(url) do |tag|
    tags << tag
  end
end
wait_for_all_requests_to_finish()

如果这可以在一个不错的和安全的方式，这将是真棒完成。结果
我可以使用线程，但它并不像数组是线程安全的红宝石

If this could be done in a nice and safe way that would be awesome.
I could use thread but it doesn't look like arrays are thread safe in ruby.

推荐答案

您可以通过使用实现线程安全互斥 ：

You can achieve thread-safety by using a Mutex:

require 'thread'  # for Mutex

urls = %w(
  http://test1.example.org/
  http://test2.example.org/
  ...
)

threads = []
tags = []
tags_mutex = Mutex.new

urls.each do |url|
  threads << Thread.new(url, tags) do |url, tags|
    tag = fetch_tag(url)
    tags_mutex.synchronize { tags << tag }
  end
end

threads.each(&:join)

这可能不过是适得其反使用一个新的线程为每个URL，因此限制了这样的线程数可能是更好的性能：

It could however be counter-productive to use a new thread for every URL, so limiting the number of threads like this might be more performant:

THREAD_COUNT = 8  # tweak this number for maximum performance.

tags = []
mutex = Mutex.new

THREAD_COUNT.times.map {
  Thread.new(urls, tags) do |urls, tags|
    while url = mutex.synchronize { urls.pop }
      tag = fetch_tag(url)
      mutex.synchronize { tags << tag }
    end
  end
}.each(&:join)

这篇关于在红宝石并行的HTTP请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！