使用 sidekiq 使用两个单独的 redis 实例? [英] Work with two separate redis instances with sidekiq?

问题描述

下午好，

我有两个独立但相关的应用.他们都应该有自己的后台队列(阅读:独立的 Sidekiq 和 Redis 进程).但是，我希望偶尔能够将作业从 app1 推送到 app2 的队列中.

从简单的队列/推送角度来看，如果 app1 没有现有的 Sidekiq/Redis 堆栈，则很容易做到这一点:

#在一个进程中，很远很远# 配置客户端Sidekiq.configure_client 做 |config|config.redis = {:url =>'redis://redis.example.com:7372/12', :namespace =>'我的命名空间' }结尾# 推送没有类定义的作业Sidekiq::Client.push('class' => 'Example::Workers::Trace', 'args' => ['hello!'])# 推送作业覆盖默认值Sidekiq::Client.push('queue' => 'example', 'retry' => 3, 'class' => 'Example::Workers::Trace', 'args' => ['hello!'])

然而，鉴于我已经从 app1 调用了 Sidekiq.configure_client 和 Sidekiq.configure_server，这里可能有一个步骤需要发生某些事情的地方.

显然，我可以直接从 Sidekiq 内部获取序列化和规范化代码，然后手动推送到 app2 的 redis 队列，但这似乎是一个脆弱的解决方案.我希望能够使用 Client.push 功能.

我想我的理想解决方案是这样的:

SidekiqTWO.configure_client { 远程连接..... }SidekiqTWO::Client.push(job....)

甚至:

$redis_remote = remote_connection.....

Sidekiq::Client.push(job, $redis_remote)

显然有点滑稽，但这是我的理想用例.

谢谢！

解决方案

所以有一件事是 根据常见问题解答，Sidekiq 消息格式非常简单且稳定:它只是 JSON 格式的散列."重点是我的——我不认为将 JSON 发送到 sidekiq 太脆弱了.尤其是当您想要对将作业发送到哪个 Redis 实例进行细粒度控制时(如在 OP 的情况下)，我可能只会编写一个小包装器，让我指示一个 Redis 实例以及正在排队的作业.

对于 Kevin Bedell 将作业循环到 Redis 实例的更一般情况，我想您不想想要控制使用哪个 Redis 实例——您只想入队并自动管理分发.看起来到目前为止只有一个人提出过这个请求，而且他们提出了一个使用Redis::Distributed:

datastore_config = YAML.load(ERB.new(File.read(File.join(Rails.root, "config", "redis.yml"))).result)datastore_config = datastore_config["defaults"].merge(datastore_config[::Rails.env])如果 datastore_config[:host].is_a?(Array)如果 datastore_config[:host].length == 1datastore_config[:host] = datastore_config[:host].first别的datastore_config = datastore_config[:host].map do |host|host_has_port = 主机 =~/:\d+\z/如果host_has_portredis://#{host}/#{datastore_config[:db] || 0}"别的redis://#{host}:#{datastore_config[:port] || 6379}/#{datastore_config[:db] || 0}"结尾结尾结尾结尾Sidekiq.configure_server 做 |config|config.redis = ::ConnectionPool.new(:size => Sidekiq.options[:concurrency] + 2, :timeout => 2) 做redis = 如果 datastore_config.is_a?大批Redis::Distributed.new(datastore_config)别的Redis.new(datastore_config)结尾Redis::Namespace.new('resque', :redis => redis)结尾结尾

在寻求获得高可用性和故障转移时需要考虑的另一件事是获得 Sidekiq Pro，其中包括可靠性功能:Sidekiq Pro 客户端可以承受 Redis 暂时中断.它会在出现错误时在本地排队作业，并在连接恢复后尝试交付这些作业."由于 sidekiq 无论如何都是用于后台进程的，因此如果 Redis 实例出现故障，短暂的延迟应该不会影响您的应用程序.如果您的两个 Redis 实例中的一个出现故障并且您正在使用轮询，除非您使用此功能，否则您仍然会失去一些工作.

Good afternoon,

I have two separate, but related apps. They should both have their own background queues (read: separate Sidekiq & Redis processes). However, I'd like to occasionally be able to push jobs onto app2's queue from app1.

From a simple queue/push perspective, it would be easy to do this if app1 did not have an existing Sidekiq/Redis stack:

# In a process, far far away

# Configure client 
Sidekiq.configure_client do |config|
  config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
end

# Push jobs without class definition 
Sidekiq::Client.push('class' => 'Example::Workers::Trace', 'args' => ['hello!'])

# Push jobs overriding default's 
Sidekiq::Client.push('queue' => 'example', 'retry' => 3, 'class' =>     'Example::Workers::Trace', 'args' => ['hello!'])

However given that I would already have called a Sidekiq.configure_client and Sidekiq.configure_server from app1, there's probably a step in between here where something needs to happen.

Obviously I could just take the serialization and normalization code straight from inside Sidekiq and manually push onto app2's redis queue, but that seems like a brittle solution. I'd like to be able to use the Client.push functionality.

I suppose my ideal solution would be someting like:

SidekiqTWO.configure_client { remote connection..... } SidekiqTWO::Client.push(job....)

Or even:

$redis_remote = remote_connection.....

Sidekiq::Client.push(job, $redis_remote)

Obviously a bit facetious, but that's my ideal use case.

Thanks!

解决方案

So one thing is that According to the FAQ, "The Sidekiq message format is quite simple and stable: it's just a Hash in JSON format." Emphasis mine-- I don't think sending JSON to sidekiq is too brittle to do. Especially when you want fine-grained control around which Redis instance you send the jobs to, as in the OP's situation, I'd probably just write a little wrapper that would let me indicate a Redis instance along with the job being enqueued.

For Kevin Bedell's more general situation to round-robin jobs into Redis instances, I'd imagine you don't want to have the control of which Redis instance is used-- you just want to enqueue and have the distribution be managed automatically. It looks like only one person has requested this so far, and they came up with a solution that uses Redis::Distributed:

datastore_config = YAML.load(ERB.new(File.read(File.join(Rails.root, "config", "redis.yml"))).result)

datastore_config = datastore_config["defaults"].merge(datastore_config[::Rails.env])

if datastore_config[:host].is_a?(Array)
  if datastore_config[:host].length == 1
    datastore_config[:host] = datastore_config[:host].first
  else
    datastore_config = datastore_config[:host].map do |host|
      host_has_port = host =~ /:\d+\z/

      if host_has_port
        "redis://#{host}/#{datastore_config[:db] || 0}"
      else
        "redis://#{host}:#{datastore_config[:port] || 6379}/#{datastore_config[:db] || 0}"
      end
    end
  end
end

Sidekiq.configure_server do |config|
  config.redis = ::ConnectionPool.new(:size => Sidekiq.options[:concurrency] + 2, :timeout => 2) do
    redis = if datastore_config.is_a? Array
      Redis::Distributed.new(datastore_config)
    else
      Redis.new(datastore_config)
    end

    Redis::Namespace.new('resque', :redis => redis)
  end
end

Another thing to consider in your quest to get high-availability and fail-over is to get Sidekiq Pro which includes reliability features: "The Sidekiq Pro client can withstand transient Redis outages. It will enqueue jobs locally upon error and attempt to deliver those jobs once connectivity is restored." Since sidekiq is for background processes anyway, a short delay if a Redis instance goes down should not affect your application. If one of your two Redis instances goes down and you're using round robin, you've still lost some jobs unless you're using this feature.

这篇关于使用 sidekiq 使用两个单独的 redis 实例?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！