用PHP管理大型阵列 [英] Managing mega Arrays in PHP

问题描述

我正在为某人的数百万条旧日志条目进行数据挖掘，并且真的想在此问题中使用PHP来展示将它们轻松链接到现有PHP系统的材料.

I'm doing a data-mine on millions of old log entries for someone and really want to use PHP in this matter to present the materials a link them easily to the existing PHP system.

我在终端机(OSX 10.8)的PHP 5.4.4中运行以下代码:

I run this code in the PHP 5.4.4 in the Terminal (OSX 10.8):

// Settings
ini_set('error_reporting', E_ALL); // Shows all feedback from the parser for debugging
ini_set('max_execution_time', 0); // Changes the 30 seconds parser exit to infinite
ini_set('memory_limit', '512M'); // Sets the memory that may be used to 512MegaBytes


echo 'Start memory usage: '.(memory_get_usage(TRUE) / 1024)."\n";

$x = Array();
for ($i = 0; $i < 1e7; $i++) {
    $x[$i] = 1 * rand(0, 10);
    //unset($x[$i]);
}

echo 'End memory usage: '.(memory_get_usage(TRUE) / 1024)."\n";
echo 'Peak memory usage: '.(memory_get_peak_usage(TRUE) / 1024)."\n";

这是一千万次循环的简单测试.与在Python中使用字典相比，泄漏真的很糟糕:(.

This is a simple test with ten-million cycles. The leakage is really bad compared to using dictionaries in Python :(.

当我取消引用unset()函数以测试其用法时，立即所有的独角兽和彩虹.因此，强制释放内存似乎很顺利.

When I unquote the unset() function to test the usage, it's instantly all unicorns and rainbows. So forcing the release of the memory seems to go well.

有什么办法可以在512M内存限制内维持10到5000万个数组条目吗?

Is there any way I can still maintain 10-50 million array entries within that 512M memory limit?

我也无法想象何时使用此类循环进行一些正则表达式..

I can't imagine when I would do some regex with these kind of loops either..

推荐答案

使用SplFixedArray，因为您确实需要查看

Use SplFixedArray because you really need to see How big are PHP arrays (and values) really? (Hint: BIG!)

$t = 1e6;
$x = array();
for($i = 0; $i < $t; $i ++) {
    $x[$i] = 1 * rand(0, 10);
}

输出

Start memory usage: 256
End memory usage: 82688
Peak memory usage: 82688

和

$t = 1e6;
$x = new SplFixedArray($t);
for($i = 0; $i < $t; $i ++) {
    $x[$i] = 1 * rand(0, 10);
}

输出

Start memory usage: 256
End memory usage: 35584
Peak memory usage: 35584

但是更好的是，我认为您应该考虑基于内存的数据库，例如 REDIS

But better still i think you should consider a memory based database like REDIS

这篇关于用PHP管理大型阵列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！