使用PHP将大量数据写入CSV文件 [英] Write large amount of data to CSV file using PHP

问题描述

我需要将近109,392行写入CSV文件，目前我正在这样做

I need to write almost 109,392 rows to CSV file and currently I am doing it like this

    //open raw memory as file, no need for temp files
    $tempMemory = fopen('php://temp', 'w');

    //default php csv handler
    foreach ($inputArray as $line) {
        fputcsv($tempMemory, $line, $delimiter);
    }

    //rewrind the "file" with the csv lines
    fseek($tempMemory, 0);

    //modify header to be downloadable csv file
    header('Content-Type: application/csv');
    header('Content-Disposition: attachement; filename="' . $outputFileName . '";');

    //send file to browser for download
    fpassthru($tempMemory);

这里的问题是它非常慢，并且我没有插入所有行。 $ inputArray 的数组大小为109,392，但是在生成的CSV文件中只有5万行。最好的方法是什么？

The problem here is that it is very slow and I am not getting all the rows inserted. The array size of $inputArray is 109,392 but only 50K rows are there in the CSV file produced. What is the best way of doing this ?

推荐答案

现在，您正在生成（或尝试生成）整个文件

As it is now, you are generating (or trying to) the whole file in memory first, and only then start sending the data to the browser.

由于结果文件（在内存中？）可能很大，因此至少应检查返回值 fputcsv（）的值。也许它开始是 FALSE ，这可能是您没有获得所有行的原因。

Since the resulting file (in memory?) could be large, you should at least check the return value of fputcsv(). Maybe it starts being FALSE which could be the reason you don't get all the rows.

无论如何，这这不是正确的方法（假设10个用户同时向您的服务器发送相同的请求）。

Anyway, this is not the right way to do this (just imagine 10 users sending the same request to your server at the same time).

我不了解php，但是生成（按需）并发送此类文件的理想方法是：

I don't know about php, but the ideal way to generate (on demand) and send such a file would be:

• 首先发送http标头，包括分块传输模式的标头。这样，您就无需预先知道文件的大小。

• first send the http headers, including a header for chunked transfer mode. That way you don't need to know the size of the file up front.

而 $ inputArray 中有数据code>，用行填充缓冲区（例如，最大64kB），然后将该缓冲区发送到浏览器（也是分块模式）。您最终可以首先压缩数据。

while there is data in $inputArray, fill a buffer with rows (say max. 64kB) and send that buffer to the browser (also in chunked mode). You could eventually compress the data first.

这样，浏览器将立即开始接收数据，而您不会t不需要超过服务器上64Kb的内存来生成数据。

That way the browser will start receiving the data immediately, and you won't need much more than 64Kb of memory on the server to generate the data.

这篇关于使用PHP将大量数据写入CSV文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！