仅使用sidekiq执行许多重复作业中的一项? [英] execute only one of many duplicate jobs with sidekiq?

问题描述

我有一个后台工作，在MongoDB上执行地图/归约工作.当用户向文档发送更多数据时，它将触发在文档上运行的后台作业.如果用户发送多个请求，它将为同一文档启动多个后台作业，但实际上只需要运行一个.有什么办法可以防止多个重复的实例?我正在考虑为每个文档创建一个队列，并确保在提交新作业之前该队列为空.或者，也许我可以以某种方式设置与我的文档ID相同的工作ID，并在提交之前检查是否存在不存在的工作?

I have a background job that does a map/reduce job on MongoDB. When the user sends in more data to the document, it kicks of the background job that runs on the document. If the user sends in multiple requests, it will kick off multiple background jobs for the same document, but only one really needs to run. Is there a way I can prevent multiple duplicate instances? I was thinking of creating a queue for each document and making sure it is empty before I submit a new job. Or perhaps I can set a job id somehow that is the same as my document id, and check that none exists before submitting it?

此外，我刚刚找到了sidekiq-unique-jobs宝石.但是该文档不存在.这是我想要的吗?

Also, I just found a sidekiq-unique-jobs gem. But the documentation is non-existent. Does this do what I want?

推荐答案

我最初的建议是针对此特定工作的互斥量.但是由于您可能有多个应用程序服务器在运行sidekiq作业，因此我建议在redis级别进行一些操作.

My initial suggestion would be a mutex for this specific job. But as there's a chance that you may have multiple application servers working the sidekiq jobs, I would suggest something at the redis level.

例如，在sidekiq工作人员中使用 redis-semaphore 定义. 未经测试的示例:

For instance, use redis-semaphore within your sidekiq worker definition. An untested example:

def perform
  s = Redis::Semaphore.new(:map_reduce_semaphore, connection: "localhost")

  # verify that this sidekiq worker is the first to reach this semaphore.
  unless s.locked?

    # auto-unlocks in 90 seconds. set to what is reasonable for your worker.
    s.lock(90)
    your_map_reduce()
    s.unlock
  end
end

def your_map_reduce
  # ...
end

这篇关于仅使用sidekiq执行许多重复作业中的一项?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！