单人工作台上的并行后台任务 [英] Parallel background tasks on single worker dyno

问题描述

我们已经有了一个Rails应用程序，其中某些请求会在带有delayed_job的工人测功机上触发长时间运行的任务，而前端轮询则会收到结果。

我们的流量很小，但却在不断增长，任务通常只需几秒钟即可完成，但可能需要一分钟。现在，单一的网络和工人测功机应该足以应付我们的负载。

问题是，delayed_job队列不会并行处理作业，所以一个较长的任务最终会阻止它后面的任务。

我们正在寻找的是类似于后端Unicorn的产品，其中单个工人动态测试可以同时处理多个任务。因为它是Rails，我们正在寻找一些多进程，而不是多线程。

（我们尝试在proc文件中创建多个工作条目 - 本地开发盒，但不在Heroku上）

Procfile：
$ b

  web：bundle exec unicorn -p $ PORT -c ./config/unicorn.rb 
＃此处定义的多个工作程序不会转化为单个工人dyno上的多个进程：
 worker：bundle exec rake jobs：work 
 worker：bundle exec rake jobs：work 
 worker：bundle exec rake jobs：work 
  

这篇Heroku文章建议使用 resque-pool宝石作为解决方案。是唯一的解决方案，或可以delayed_job被用于并行后台作业吗？

解决方案

根据@ radiospiel的相关帖子，您可以使用工头启动多个进程。

1）将工头添加到您的Gemfile中

2）创建两个文件：

Procfile：

  web：bundle exec unicorn -p $ PORT -c ./config/unicorn.rb 
 worker：bundle exec foreman start -f Procfile.workers 
  

Procfile.workers：

  dj_worker：bundle exec rake作业：工作
 dj_worker：bundle exec rake jobs：work 
 dj_worker：bundle exec rake jobs：work 
  

I只是部署到Heroku，效果很好。

We've got a Rails app where certain requests trigger long-running tasks on a worker dyno with delayed_job, while the front-end polls until it receives a result.

Our traffic is small but growing, and the tasks generally take only a few seconds to complete, but can take up to a minute. Right now a single web and worker dyno each should be sufficient to handle our load.

The problem is, the delayed_job queue won't process jobs in parallel, so a longer task ends up holding up the tasks behind it.

What we're looking for is something like Unicorn for the backend, where a single worker dyno can process multiple tasks concurrently. And since it's Rails, we're looking for something multi-process, not multi-threaded.

(We tried creating multiple worker entries in our procfile- This worked on the local dev box, but not on Heroku)

Procfile:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
# Multiple workers defined here doesn't translate into multiple processes on single worker dyno:
worker: bundle exec rake jobs:work
worker: bundle exec rake jobs:work
worker: bundle exec rake jobs:work

This Heroku article proposes using the resque-pool gem as a solution. Is that the only solution, or can delayed_job be used for parallel background jobs as well?

解决方案

According to @radiospiel's related post, you can use foreman to start multiple processes.

1) Add foreman to your Gemfile

2) Create two files:

Procfile:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec foreman start -f Procfile.workers

Procfile.workers:

dj_worker: bundle exec rake jobs:work
dj_worker: bundle exec rake jobs:work
dj_worker: bundle exec rake jobs:work

I just deployed this to Heroku, works great.

这篇关于单人工作台上的并行后台任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！