循环遍历大量记录时 Ruby on Rails 内存泄漏；find_each 没有帮助 [英] Ruby on Rails memory leak when looping through large number of records; find_each doesn't help

问题描述

我有一个 Rails 应用程序，可以处理 mysql 数据库中的大量(数百万)条记录.一旦它开始工作，它的内存使用量就会以每秒 50MB 的速度快速增长.使用 oink 等工具，我能够将根本原因缩小到一个循环，该循环遍历数据库中一个大表中的所有记录.

I have a Rails app that processes a large (millions) number of records in a mysql database. Once it starts working, its memory use quickly grows at a speed of 50MB per second. With tools like oink I was able to narrow down the root cause to one loop that goes through all the records in a big table in the database.

我知道如果我使用像 Person.all.each 这样的东西，所有的记录都会被加载到内存中.但是，如果我切换到 find_each，我仍然会看到相同的内存问题.为了进一步隔离问题，我创建了以下测试控制器，它只循环遍历记录.我想 find_each 每次只在内存中保留少量对象，但内存使用量会随着它的执行而线性增长.

I understand if I use something like Person.all.each, all the records will be loaded into memory. However if I switch to find_each, I still see the same memory issue. To further isolate the problem I created the following test controller, which does nothing but looping through the records. I suppose find_each only keeps a small number of objects in memory each time, but memory use grows linearly as it executes.

class TestController < ApplicationController
  def memory_test
    Person.find_each do |person|
    end
end

我怀疑这与 ActiveRecord 缓存查询结果有关.但是我检查了我的环境设置，并且我确实在开发中将所有与缓存相关的选项设置为 false(我使用的是由 rails 创建的默认设置).我在网上进行了一些搜索，但找不到解决方案.

I suspect it has to do with ActiveRecord caching the query results. But I checked my environment settings and I do have all the caching related options set to false in development (I am using the default settings created by rails). I did some search online but couldn't find a solution.

我使用的是 rails 3.1.0 rc1 和 ruby​​ 1.9.2

I am using rails 3.1.0 rc1 and ruby 1.9.2

谢谢！

推荐答案

我能够自己解决这个问题.有两个地方可以更改.

I was able to figure this out myself. There are two places to change.

首先，禁用 IdentityMap.在 config/application.rb

First, disable IdentityMap. In config/application.rb

config.active_record.identity_map = false

其次，使用uncached来结束循环

class MemoryTestController < ApplicationController
  def go
    ActiveRecord::Base.uncached do
      Person.find_each do |person|
        # whatever operation
      end
    end
  end
end

现在我的内存使用得到控制.希望这对其他人有帮助.

Now my memory use is under control. Hope this helps other people.

这篇关于循环遍历大量记录时 Ruby on Rails 内存泄漏；find_each 没有帮助的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！