Codeigniter处理大文件时允许的内存大小耗尽 [英] Codeigniter Allowed memory size exhausted while processing large files

问题描述

我发布这个，以防其他人寻找相同的解决方案，看到我只是浪费了两天在这个胡说。

我有一个cron工作使用以下代码每天更新一个非常大的文件数据库：

  if（（$ handle = fopen dirname（__ FILE__）。'/ uncompressed'，r））！== FALSE）
 {
 while（（$ data = fgets（$ handle））！== FALSE）
 {
 $ thisline = json_decode（$ data，true）; 
 $ this-> regen（$ thisline）; 
} 
 fclose（$ handle）; 
} 
  

这是一个Codeigniter控制器，仅用于cron作业。 $ this-> regen函数运行一系列不同的检查，并将正确的信息存储在数据库中。

这个文件本身只有超过300MB的JSON，的内存。

解决方案

我花了几个小时来解决这个问题，没有什么明显。我使用fgets，我有$ query-> free_result（）在正确的地方。它没有帮助。所以，我开始检查一个大约100行的循环，并观察memory_get_usage（）的输出。我最后将它缩小到Codeigniter Active Record类 - 每次调用该类都会导致内存使用量增加一小部分。

然后我在< a href =http://ellislab.com/forums/viewthread/88908/ =nofollow> Ellislabs ，我得到了答案。 CI Active Record保存查询，以便如果需要，可以在多个函数中构建查询。 （我甚至不打算默认打开它是多么愚蠢。）

转到/config/database.php并添加

$ db ['default'] ['save_queries'] = FALSE;

到文件的结尾。然后确保使用Active Record在单个函数中构建和执行查询。如果您只需要将其关闭一次，请使用

$ this-> db-> save_queries = FALSE;

在构造函数或任何你需要的地方。

I'm posting this in case someone else is looking for the same solution, seeing as I just wasted two days on this bullshit.

I have a cron job that updates the database using a very large file once a day, using the following code:

if (($handle = fopen(dirname(__FILE__) . '/uncompressed', "r")) !== FALSE) 
{
    while (($data = fgets($handle)) !== FALSE) 
    {
        $thisline = json_decode($data, true);
        $this->regen($thisline);
    }
    fclose($handle);
}

This is in a Codeigniter controller that's only used for cron jobs. The $this->regen function runs through a bunch of different checks and stores the right information from the line in the database. The file itself is over 300MB of JSONs separated by newlines.

The problem: it would only process about 20,000 lines before the whole thing ran out of memory.

解决方案

I spent hours troubleshooting this and got nothing obvious. I'm using fgets, I have $query->free_result() in the right places. It didn't help. So then I started checking a loop of about 100 lines, and watched the output of memory_get_usage(). I finally narrowed it down to the Codeigniter Active Record class - every call to the class caused the memory usage to increase by a tiny amount.

Then I found this thread on Ellislabs and I got the answer. CI Active Record saves queries so that if you want to, you can build a query in multiple functions. (I am not even going to go into how dumb it is to have that switched on by default.)

Go to /config/database.php and add

$db['default']['save_queries'] = FALSE;

to the end of the file. Then make sure you build and execute queries using Active Record in a single function. If you need to switch it off just for one case, use

$this->db->save_queries = FALSE;

in the constructor or wherever you need to put it.

这篇关于Codeigniter处理大文件时允许的内存大小耗尽的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！