Codeigniter处理大文件时允许的内存大小耗尽 [英] Codeigniter Allowed memory size exhausted while processing large files
问题描述
我发布这个,以防其他人寻找同样的解决方案,看到我只是浪费了两天在这个胡说。
I'm posting this in case someone else is looking for the same solution, seeing as I just wasted two days on this bullshit.
我有一个cron工作使用以下代码每天更新一个非常大的文件数据库:
I have a cron job that updates the database using a very large file once a day, using the following code:
if (($handle = fopen(dirname(__FILE__) . '/uncompressed', "r")) !== FALSE)
{
while (($data = fgets($handle)) !== FALSE)
{
$thisline = json_decode($data, true);
$this->regen($thisline);
}
fclose($handle);
}
这是一个Codeigniter控制器,只用于cron作业。 $ this-> regen函数运行一系列不同的检查,并将来自该行的正确信息存储在数据库中。
This is in a Codeigniter controller that's only used for cron jobs. The $this->regen function runs through a bunch of different checks and stores the right information from the line in the database. The file itself is over 300MB of JSONs separated by newlines.
问题:它只会处理大约20,000行,在整个事情消失之前的内存。
The problem: it would only process about 20,000 lines before the whole thing ran out of memory.
推荐答案
我花了几个小时来解决这个问题,没有什么明显。我使用fgets,我有$ query-> free_result()在正确的地方。它没有帮助。所以,我开始检查一个大约100行的循环,并观察memory_get_usage()的输出。我最后把它缩小到Codeigniter Active Record类 - 每次调用该类都会导致内存使用量增加一小部分。
I spent hours troubleshooting this and got nothing obvious. I'm using fgets, I have $query->free_result() in the right places. It didn't help. So then I started checking a loop of about 100 lines, and watched the output of memory_get_usage(). I finally narrowed it down to the Codeigniter Active Record class - every call to the class caused the memory usage to increase by a tiny amount.
然后我在< a href =http://ellislab.com/forums/viewthread/88908/ =noreferrer> Ellislabs ,我得到了答案。 CI Active Record保存查询,以便如果需要,可以在多个函数中构建查询。 (我甚至不打算默认打开它是多么愚蠢。)
Then I found this thread on Ellislabs and I got the answer. CI Active Record saves queries so that if you want to, you can build a query in multiple functions. (I am not even going to go into how dumb it is to have that switched on by default.)
转到/config/database.php并添加
Go to /config/database.php and add
$ db ['default'] ['save_queries'] = FALSE;
到文件的结尾。然后确保使用Active Record在单个函数中构建和执行查询。如果您只需要将其关闭一次,请使用
to the end of the file. Then make sure you build and execute queries using Active Record in a single function. If you need to switch it off just for one case, use
$ this-> db-> save_queries = FALSE;
在构造函数或任何你需要的地方。
in the constructor or wherever you need to put it.
这篇关于Codeigniter处理大文件时允许的内存大小耗尽的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!