在Heroku上设置Sidekiq：并发 [英] Setting Sidekiq :concurrency on Heroku

问题描述

我无法弄清楚Heroku上的并发性如何工作，以及如何设置Sidekiq的最佳：concurrency 值

以下是设置 -

  Puma Web服务器
 2工作者
 5线程
 
 Heroku Dynos 
 8 web dynos 
 2workerdynos（这些将运行Sidekiq，不要与Puma Workers混淆）
 
 DB连接
 120 Postgres允许的最大连接数
 5活动记录池大小（默认）
？ Sidekiq：并发值
  

每个Puma工作者都可以使用默认的5个ActiveRecord DB连接（这对每个有5个线程）。由于每个Web Dyno有2个Puma工作站，因此每个Web Dyno最多可使用10个连接。在所有8个Web dynos中，它们占用80个连接

这会使120 - 80 = 40个连接被2个dynos使用，运行Sidekiq。假设每个dyno上运行一个Sidekiq进程（这是真的吗？），每个Sidekiq进程可以自由使用 20个连接（即设置：concurrency 20


1. 这个逻辑是否正确，或者我误解了这些进程如何在dynos上运行？




2. Sidelog进程是否受ActiveRecord连接池限制5限制？如果是这样，听起来像设置为20是没用的，因为它一次最多只能使用5个连接。

你的逻辑听起来很正确，但有一些注释。

Puma不会在所有5个线程中不断推动Sidekiq，因此您可以使用较低的值，例如所以5个线程共享2个连接：

  Sidekiq.configure_client do | config | 
 config.redis = {size：2} 
 end 
  

Sidekiq并发性是工作线程的数量，而不是连接。您希望显式限制连接池大小：

  Sidekiq.configure_server do | config | 
 config.redis = {size：20} 
 end 
  

默认情况下， Sidekiq的连接池大小为（concurrency + 2），但20个连接应该可以正常工作，默认并发度为25。

I'm having trouble figuring out how exactly concurrency works on Heroku, and how to go about setting the optimal :concurrency value for Sidekiq

Here's the set up -

Puma Web Server
  2 workers
  5 threads

Heroku Dynos
  8 web dynos
  2 "worker" dynos (These will run Sidekiq, not to be confused with Puma Workers)

DB Connections
 120 Max Connections Allowed by Postgres
 5   Active Record Pool Size (default)
 ??  Sidekiq :concurrency value

Each Puma worker is allowed the default 5 ActiveRecord DB connections (which corresponds nicely to each having 5 threads). Since there are 2 of these Puma workers per Web Dyno, each Web Dyno consumes up to 10 connections. Across all 8 Web dynos, they take up 80 connections

That leaves 120 - 80 = 40 connections to be used by the 2 dynos that will run Sidekiq. Assuming there's one Sidekiq process running on each dyno (is this true?), each Sidekiq process is free to use up to 20 connections (i.e. set :concurrency 20)

1. Is that logic correct, or did I misunderstand how these processes run on dynos?

2. Is the Sidekiq process limited in any way by the ActiveRecord connection pool limit of 5? If it is, sounds like setting it to 20 is useless since it can really only use a maximum of 5 connections at a time.

Thanks for the help!

解决方案

Your logic sounds right but a couple of notes.

Puma won't be pushing to Sidekiq on all 5 threads constantly so you can use a lower value, e.g. so the 5 threads share 2 connections:

Sidekiq.configure_client do |config|
  config.redis = { size: 2 }
end

Sidekiq concurrency is the number of worker threads, not connections. You want to explicitly limit the connection pool size:

Sidekiq.configure_server do |config|
  config.redis = { size: 20 }
end

By default, Sidekiq's connection pool size is (concurrency + 2) but 20 connections should work ok for the default concurrency of 25.

这篇关于在Heroku上设置Sidekiq：并发的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！