工作人员完成后，Sidekiq 不释放内存 [英] Sidekiq not deallocating memory after workers have finished

问题描述

我有大约 六个 Sidekiq 工作器，它们执行 JSON 抓取.根据端点的数据集大小，它们在 1 分钟到 4 小时之间完成.尤其是观看耗时 4 小时的长视频时，我发现随着时间的推移，内存有非常轻微的增长.

I have about six Sidekiq worker which perform JSON crawling. Dependent on the endpoint's dataset size they finish between 1min and 4h. Especially, watching the long one, which takes 4h, I see a very slight increase of memory over time.

这不是问题，直到我想再次安排相同的工人作业.内存不会被释放并堆积起来，直到我遇到 Linux OOM Killer，它摆脱了我的 Sidekiq 进程.

It's not a problem, until I want to schedule the same worker jobs again. The memory is not deallocated and stacks up, until I run into the Linux OOM Killer which gets rid of my Sidekiq process.

内存泄漏?我观察了 ObjectSpace 中不同对象的数量:

Memory leak? I watched the number of different objects in ObjectSpace:

ObjectSpace.each_object.inject(Hash.new(0)) { |count, o| count[o.class] += 1 }

那里没有真正的增加，散列、数组等的集合保持不变，垃圾收集器清除了短暂的增加，gc.stat[:count] 告诉我，垃圾收集器也在工作.

There is not really an increase there, the set of hashes, arrays, etc. stays the same, short increases are swept away by the Garbage Collector and gc.stat[:count] tells me, that the Garbage Collector is working, too.

即使在工人完成之后，例如我记录了 [Done] 并且没有工作人员再忙了，内存没有被释放.原因是什么?我可以对此做些什么吗?编写终结器?

Even after the worker finishes, e.g. I get the [Done] logged and no workers are busy any more, the memory is not deallocated. What are the reasons for that? Can I do something against this? Write a finalizer?

唯一的当前解决方案:重启 Sidekiq 进程.

The only current solution: Restart the Sidekiq process.

我使用 Ruby 2.0.0 并使用 Ruby MRI.

I am on Ruby 2.0.0 and use Ruby MRI.

对于 JSON 解析，我使用 Yajl，因此是 C 绑定.我需要它，因为它似乎是唯一正确实现流式读取和写入的快速 JSON 解析器.

For the JSON parsing I use Yajl, thus a C binding. I need it because it seems the only fast JSON parser that properly implements streamed reading and writing.

推荐答案

Mike Perham 在这里解决了这个问题:http://www.mikeperham.com/2009/05/25/memory-hungry-ruby-daemons/

Mike Perham who wrote Sidekiq addressed this here: http://www.mikeperham.com/2009/05/25/memory-hungry-ruby-daemons/

tl;dr 版本: MRI 不会返还内存，您最多只能控制堆，而要做到这一点，Ruby 企业版 被推荐.

tl;dr version: MRI will not give the memory back, the most you can do is control the heap, and to do that, Ruby Enterprise Edition was suggested.

不知道这些有什么帮助，但情况就是这样 - 直接从马嘴里说出来.

Don't know that any of this helps, but that is the situation - straight from the horse's mouth.

这篇关于工作人员完成后，Sidekiq 不释放内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！