使用 Rails 运行多个后台并行作业 [英] Running multiple background parallel jobs with Rails

问题描述

在我的 Ruby on Rails 应用程序中，我需要并行执行 50 个后台作业.每个作业都会创建一个到不同服务器的 TCP 连接，传递一些数据并更新一个活动记录对象.

On my Ruby on Rails application I need to execute 50 background jobs in parallel. Each job creates a TCP connection to a different server, fecths some data and updates an active record object.

我知道执行此任务的不同解决方案，但其中任何一个都是并行的.例如，delayed_job (DJ) 可能是一个很好的解决方案，前提是它可以并行执行所有作业.

I know different solutions to perform this task but any of them in parallel. For example, delayed_job (DJ) could be a great solution if only it could execute all jobs in parallel.

有什么想法吗?谢谢.

推荐答案

一些想法...

• 仅仅因为您需要阅读 50 个站点并且自然需要一些并行工作，并不意味着您需要 50 个进程或线程.您需要平衡减速和开销.让 10 或 20 个进程每个读取几个站点怎么样?

• Just because you need to read 50 sites and naturally want some parallel work does not mean that you need 50 processes or threads. You need to balance the slowdown and overhead. How about having 10 or 20 processes each read a few sites?

根据您使用的 Ruby，请注意绿色线程，您可能无法获得所需的并行结果

Depending on which Ruby you are using, be careful about the green threads, you may not get the parallel result you want

您可能希望像反向的客户端 inetd 那样构建它，并使用 connect_nonblock 和 IO.select 来获得您想要的并行连接使所有服务器并行响应.您实际上并不需要对结果进行并行处理，您只需要在所有服务器上并行排队，因为这才是真正存在延迟的地方.

You might want to structure it like a reverse, client-side inetd, and use connect_nonblock and IO.select to get the parallel connections you want by making all the servers respond in parallel. You don't really need parallel processing of the results, you just need to get in line at all the servers in parallel, because that is where the latency really is.

因此，来自套接字库的类似内容...将其扩展为多个未完成的连接...

So, something like this from the socket library...extend it for multiple outstanding connections...

require 'socket'
include Socket::Constants
socket = Socket.new(AF_INET, SOCK_STREAM, 0)
sockaddr = Socket.sockaddr_in(80, 'www.google.com')
begin
  socket.connect_nonblock(sockaddr)
  rescue Errno::EINPROGRESS
  IO.select(nil, [socket])
  begin
    socket.connect_nonblock(sockaddr)
    rescue Errno::EISCONN
  end
end
socket.write("GET / HTTP/1.0

")
# here perhaps insert IO.select. You may not need multiple threads OR multiple
# processes with this technique, but if you do insert them here
results = socket.read

这篇关于使用 Rails 运行多个后台并行作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！