运行使用Rails多个后台并行作业 [英] Running multiple background parallel jobs with Rails

问题描述

在我的Ruby on Rails应用程序，我需要并行执行50后台作业。每个作业创建一个TCP连接到不同的服务器，fecths一些数据并更新活动记录的对象。

On my Ruby on Rails application I need to execute 50 background jobs in parallel. Each job creates a TCP connection to a different server, fecths some data and updates an active record object.

我知道不同的解决方案来并行执行此任务，但其中任何一个。例如，delayed_job的（DJ）可能是一个很好的解决方案，只要它能够并行执行的所有作业。

I know different solutions to perform this task but any of them in parallel. For example, delayed_job (DJ) could be a great solution if only it could execute all jobs in parallel.

任何想法？谢谢你。

推荐答案

的几点思考...

• 仅仅因为你需要阅读50个站点，自然需要一些并行工作做的不的意思是，你需要50的进程或线程。你需要平衡经济放缓和开销。怎么样含有10个或20个进程每次读了几个网站？

• Just because you need to read 50 sites and naturally want some parallel work does not mean that you need 50 processes or threads. You need to balance the slowdown and overhead. How about having 10 or 20 processes each read a few sites?

根据所Ruby中，你正在使用，请注意绿色线程，你可能不会得到你想要的结果并行

Depending on which Ruby you are using, be careful about the green threads, you may not get the parallel result you want

您可能要构建它像一个反向的，客户端的inetd，并使用 connect_nonblock 和 IO.select 来得到你想要通过使所有服务器并行响应并行连接。你真的不想要的结果的并行处理，你只需要在平行的所有服务器在网上得到的，因为这是延迟确实是。

You might want to structure it like a reverse, client-side inetd, and use connect_nonblock and IO.select to get the parallel connections you want by making all the servers respond in parallel. You don't really need parallel processing of the results, you just need to get in line at all the servers in parallel, because that is where the latency really is.

所以，像这样从套接字库...扩展它的多个未完成的连接...

So, something like this from the socket library...extend it for multiple outstanding connections...

require 'socket'
include Socket::Constants
socket = Socket.new(AF_INET, SOCK_STREAM, 0)
sockaddr = Socket.sockaddr_in(80, 'www.google.com')
begin
  socket.connect_nonblock(sockaddr)
  rescue Errno::EINPROGRESS
  IO.select(nil, [socket])
  begin
    socket.connect_nonblock(sockaddr)
    rescue Errno::EISCONN
  end
end
socket.write("GET / HTTP/1.0\r\n\r\n")
# here perhaps insert IO.select. You may not need multiple threads OR multiple
# processes with this technique, but if you do insert them here
results = socket.read

这篇关于运行使用Rails多个后台并行作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！