PHP的for循环耗尽了允许的内存大小 [英] Allowed memory size exhausted in PHP for loop
问题描述
当我尝试在PHP中操作大量数组并返回结果作为 HTTP POST请求的响应时，我遇到了致命错误:
I'm facing a fatal error while I'm trying to manipulate a huge array of arrays in PHP and return the result as a response of an HTTP POST request:
允许的内存大小为536870912字节已耗尽
Allowed memory size of 536870912 bytes exhausted
我已经尝试设置ini_set('memory_limit', '-1');以便查看是否得到结果，但是没有得到任何类型的响应.邮递员在我尝试发出POST请求的所有时间都崩溃了.
I have already tried to set ini_set('memory_limit', '-1'); in order to see if I get the result, but I didn't get any type of response. Postman crashed all the times that I tried to make the POST request.
数组的起始结构就是这个.机身大小约为25mb.主数组包含大约22k具有这种结构的数组，我刚刚添加了2:
The starting structure of the array is this one. The body size is around 25mb. The main array contains around 22k arrays with this structure, I have just included 2:
Array
(
[0] => Array
(
[id] => 14
[isActive] => 1
[personId] => 0023fff16d353d16a052a267811af53bc8bd42f51f2266a2904ca41db19dfd32_0
[gender] => m
[age] => 69
[linedata_0] => 2018-03-01 17:15:18, 155.59, 294.076; 2018-03-01 17:16:04, 502.968, 249.947; 2018-03-01 17:16:44, 276.837, 270.593; 2018-03-01 17:17:28, 431.68, 371.14; 2018-03-01 17:17:34, 851.622, 355.915
)
[1] => Array
(
[id] => 180
[isActive] => 1
[personId] => 02659982ae8286409cc5bb283089871b62f2bafbbad517941d64e77ecf2b62b1_0
[gender] => m
[age] => 69
[linedata_0] => 2018-03-01 13:20:05, 155.599, 293.841; 2018-03-01 13:20:48, 495.468, 249.582; 2018-03-01 13:21:28, 258.791, 260.748; 2018-03-01 13:23:20, 859.061, 352.237; 2018-03-01 13:23:32, 56.1404, 269.858
)
)
在php部分下面，通过分解时间戳和每个用户的坐标来操纵数组，以期获得预期的最终结果:
Here below the php part for manipulate the array in order to have the expected final result by explode the timestamp and coordinates for each user:
$final_result = [];
foreach($query_result as $row)
{
$line_datas =explode(";",$row["linedata_0"]);
$linedata = [];
$final = [];
$d = [];
for($s =0; $s < count($line_datas); $s++){
$line_data = explode(",",$line_datas[$s]);
$d["timestamp"] = utf8_encode($line_data[0]);
$d["x"]= utf8_encode($line_data[1]);
$d["y"] = utf8_encode($line_data[2]);
array_push($linedata,$d);
}
$final["id"]= $row["id"];
$final["isActive"]= $row["isActive"];
$final["personId"]= utf8_encode($row["personId"]);
$final["name"] = NULL;
$final["gender"] = utf8_encode($row["gender"]);
$final["age"] = utf8_encode($row["age"]);
$final["linedata"]=$linedata;
array_push($final_result, $final);
}
return $final_result;
在我看来，没有无限循环或不良做法可以证明存在内存问题.唯一真正的问题可能是需要操纵的数组的大小.
As it seems to me there are no infinite loop or bad practices that can justify a memory issue. The only real problem could be the size of the array that need to be manipulated.
有什么建议吗?
推荐答案
此答案是一个示例，说明如何在代码中实现缓冲区(内存中的有限数组)，并在缓冲区填充后将其内容刷新到磁盘上，最后，您会在磁盘上发现一个巨大的JSON格式数组.我在与您类似的情况下使用了这种方式，并且在内存使用"方面取得了不错的成绩，但是正如我在评论中告诉您的那样，您需要重新考虑为什么首先需要使用HUGE数组，以及是否有避免的方法它，随它去吧.
This answer is an example of how to implement a buffer(a limited array in memory) in your code and when it is filled, flush it's contents to disk, at the end you will find a huge array on disk in JSON format. I used this way in a situation similar to yours and got great result regarding "memory usage", but as I told you in comments you need to rethink why you need that HUGE array in the first place, and if there is a way to avoid it, go with it.
使用此功能可以节省您的$final_result数组使用的内存，并将其替换为$ final_result字符串缓冲区，但我们正在控制它对内存的使用.但是，您的$query_result阵列仍会占用其所需的内存.
using this function will save you the memory used by your $final_result array and replace it with $final_result string buffer but we are controlling it's use of memory. However your $query_result array will still taking the memory it needs.
请注意，您需要根据需要更改功能，因为我使用了代码中未定义的变量.
Note that you need to alter the function as you need because I used your variables which are undefined in my code.
/**
* proccess the HUGE array and save it to disk in json format [element,element]
*
* @param string $fileName absulote file name path you want to save the proccessed array in
* @return int processed elements count
*/
function buildMyArrayInFile($fileName)
{
$flushCheckPoint = 100;// set the buffer size as needed, depending on the size of PHP allowed memory and average element size
$processedElements = 0;
$final_result = "[";
file_put_contents($fileName, "");//prepare the file and erase anything in it
foreach($query_result as $row)
{
$line_datas =explode(";",$row["linedata_0"]);
$linedata = [];
$final = [];
$d = [];
for($s =0; $s < count($line_datas); $s++){
$line_data = explode(",",$line_datas[$s]);
$d["timestamp"] = utf8_encode($line_data[0]);
$d["x"]= utf8_encode($line_data[1]);
$d["y"] = utf8_encode($line_data[2]);
array_push($linedata,$d);
}
$final["id"]= $row["id"];
$final["isActive"]= $row["isActive"];
$final["personId"]= utf8_encode($row["personId"]);
$final["name"] = NULL;
$final["gender"] = utf8_encode($row["gender"]);
$final["age"] = utf8_encode($row["age"]);
$final["linedata"]=$linedata;
$final_result .= json_encode($final) . ",";
$processedElements ++;
if($processedElements % $flushCheckPoint === 0){
//the array has reached the limit, flush the array to disk
file_put_contents($fileName, $final_result, FILE_APPEND);
$final_result = "";
}
}
$final_result = rtrim($final_result, ",");//trim the last comma
$final_result .= "]";
//flush the remaning data in $final_result
file_put_contents($fileName, $final_result, FILE_APPEND);
return $processedElements;
}
这是测试功能的另一个简单版本
this is another simple version of the function for testing
// test
var_dump(buildMyArrayInFile2("/home/myuser/myArray.json"));
// outputs int(7)
function buildMyArrayInFile2($fileName)
{
$flushCheckPoint = 2;// set the buffer size as needed, depending on the size of PHP allowed memory and average element size
$processedElements = 0;
$final_result = "[";
file_put_contents($fileName, "");//prepare the file and erase anything in it
$test_result = [1,2,3,4,"wee","hello\nworld",5];
foreach($test_result as $row)
{
$final_result .= json_encode($row) . ",";
$processedElements ++;
if($processedElements % $flushCheckPoint === 0){
//the array has reached the limit, flush the array to disk
file_put_contents($fileName, $final_result, FILE_APPEND);
$final_result = "";
}
}
$final_result = rtrim($final_result, ",");//trim the last comma
$final_result .= "]";
//flush the remaning data in $final_result
file_put_contents($fileName, $final_result, FILE_APPEND);
return $processedElements;
}
这篇关于PHP的for循环耗尽了允许的内存大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
