分割可变大小的块的最有效内存方式? [英] Most memory-efficient way to split a variable sized chunks?

问题描述

是否有办法像fread一样对变量进行操作? 也就是说，我想一次读取"另一个1MB的内存变量.

Is there a way to do something like an fread, but on a variable? That is, I want to "read" another in-memory variable 1MB at a time.

这样，我可以得到这样的东西:

That way I could have something like this:

$data = ... ; // 10MB of data

$handle = fopen($data, "rb"); // Need something instead of fopen here

while (!feof($handle))
{
    $chunk = fread($handle, 1048576); // Want to read 1MB at a time

    doSomethingWithChunk($chunk);
}

fclose($handle);

我有一个大的二进制文件加载到内存中，大约10MB.我想将其拆分为1MB的大块数组.我不需要一次全部用完1MB的内存，因此我认为我可以比仅使用PHP内置的

I have a large binary file loaded into memory, about 10MB. I'd like to split it into an array of 1MB chunks. I don't need all 1MB chunks in memory at once, so I think I could do something like the above more efficiently than just using PHP's built-in str_split function.

推荐答案

无法顺序读取"已加载到内存中的字符串.拆分它实际上不是更有效.多个变量的开销也将比单个变量占用更多的内存.理想情况下，您可以将字符串加载到流中，但是PHP实际上并没有字符串流.

There's no way to sequentially 'read' a string that's already loaded into memory; it's not really more efficient to split it up. The overhead of multiple variables will use more memory than a single one as well. Ideally you would load the string into a stream, but PHP doesn't really have a string stream.

如果只想分块处理字符串，则可以循环遍历它的子字符串:

If you just want to deal with the string in chunks, you can just loop over substrings of it:

$data;
$pointer = 0, $size = strlen($data);

$chunkSize = 1048576;
while ($pointer < $size)
{
    $chunk = substr($data, $pointer, $chunkSize);
    doSomethingWithChunk($chunk);
    $pointer += $chunkSize;
}

我不确定PHP在内部如何处理大字符串，但根据

I'm not sure how PHP handles large strings internally, but according to the string documentation, a string can only be "as large as up to 2GB (2147483647 bytes maximum)". If your file is about 10MB, it shouldn't be a problem for PHP.

另一个选择(可能是更好的选择)是将$data加载到

Another option (probably the better option) is to load $data into a memory or temporary stream. If you want to spare the environment from excessive memory, you can use the php://temp stream wrapper, where some of the data is stored in a temporary file if it exceeds 2MB. Just load the string into the stream as soon as possible to conserve memory, and then you can use the file stream functions on it.

$dataStream = fopen("php://temp", "w+b");
fwrite($dataStream, funcThatGetsData()); // try not to put data into a variable to save memory

while (!feof($dataStream))
{
    $chunk = fread($dataStream, 1048576); // want to read 1MB at a time
    doSomethingWithChunk($chunk);
}

fclose($dataStream);

如果从另一个函数获取$data，则可以绕过$dataStream.如果必须事先在字符串中包含$data，请确保在其上调用unset()以释放内存:

If you get $data from another function you could pass around $dataStream instead. If you must have $data in a string beforehand, be sure to call unset() on it to free the memory:

$data = getData(); // string from some other function
$dataStream = fopen("php://temp", "w+b");
fwrite($dataStream, $data);
unset($data); // free 10MB of memory!
...

如果要将其全部保留在内存中，可以使用php://memory，但在这种情况下也可以只使用字符串.

If you want to keep it all in memory you can use php://memory, but you might as well just use a string in that case.

这篇关于分割可变大小的块的最有效内存方式?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！