phpleague flysystem读取和写入服务器上的大文件 [英] phpleague flysystem read and write to large file on server

问题描述

我正在将flysystem与IRON IO队列一起使用，并且尝试运行一个数据库查询，该查询将获取约180万条记录，同时一次执行5000条记录.这是我收到的错误消息，文件大小为50+ MB:

I am using flysystem with IRON IO queue and I am attempting to run a DB query that will be taking ~1.8 million records and while doing 5000 at at time. Here is the error message I am receiving with file sizes of 50+ MB:

PHP Fatal error:  Allowed memory size of ########## bytes exhausted

以下是我要采取的步骤:

Here are the steps I would like to take:

1)获取数据

2)将其转换为CSV适当的字符串(即implode(',', $dataArray) . "\r\n")

2) Turn it into a CSV appropriate string (i.e. implode(',', $dataArray) . "\r\n")

3)从服务器(在本例中为S3)获取文件

3) Get the file from the server (in this case S3)

4)读取该文件的内容，并将此新字符串附加到该文件中，然后将该内容重新写入S3文件中.

4) Read that files' contents and append this new string to it and re-write that content to the S3 file

以下是我所拥有的代码的简短摘要:

Here is a brief run down of the code I have:

public function fire($job, $data)
{
    // First set the headers and write the initial file to server
    $this->filesystem->write($this->filename, implode(',', $this->setHeaders($parameters)) . "\r\n", [
        'visibility' => 'public',
        'mimetype' => 'text/csv',
    ]);

    // Loop to get new sets of data
    $offset = 0;

    while ($this->exportResult) {
        $this->exportResult = $this->getData($parameters, $offset);

        if ($this->exportResult) {
            $this->writeToFile($this->exportResult);

            $offset += 5000;
        }
    }
}

private function writeToFile($contentToBeAdded = '')
{
    $content = $this->filesystem->read($this->filename);

    // Append new data
    $content .= $contentToBeAdded;

    $this->filesystem->update($this->filename, $content, [
        'visibility' => 'public'
    ]);
}

我假设这不是最有效的?我要去这些文档: PHPLeague Flysystem

I'm assuming this is NOT the most efficient? I am going off of these docs: PHPLeague Flysystem

如果有人可以将我指向更合适的方向，那就太好了！

If anyone can point me in a more appropriate direction, that would be awesome!

推荐答案

Flysystem支持读取/写入/更新流

Flysystem supports read/write/update stream

请检查最新的API https://flysystem.thephpleague.com/api/

Please check latest API https://flysystem.thephpleague.com/api/

$stream = fopen('/path/to/database.backup', 'r+');
$filesystem->writeStream('backups/'.strftime('%G-%m-%d').'.backup', $stream);

// Using write you can also directly set the visibility
$filesystem->writeStream('backups/'.strftime('%G-%m-%d').'.backup', $stream, [
    'visibility' => AdapterInterface::VISIBILITY_PRIVATE
]);

if (is_resource($stream)) {
    fclose($stream);
}

// Or update a file with stream contents
$filesystem->updateStream('backups/'.strftime('%G-%m-%d').'.backup', $stream);

// Retrieve a read-stream
$stream = $filesystem->readStream('something/is/here.ext');
$contents = stream_get_contents($stream);
fclose($stream);

// Create or overwrite using a stream.
$putStream = tmpfile();
fwrite($putStream, $contents);
rewind($putStream);
$filesystem->putStream('somewhere/here.txt', $putStream);

if (is_resource($putStream)) {
    fclose($putStream);
}

这篇关于phpleague flysystem读取和写入服务器上的大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！