php用ftell解析csv [英] php parsing csv with ftell

问题描述

我有一个500mb的csv文件，其中包含超过500,000行，每行80个字段。我正在使用fget逐行处理文件。

I have a 500mb csv file with over 500,000 lines, each with 80 fields. I am using fget to process the file line by line.

$col1 = array();
while (($row = fgetcsv($handle, 1000, ",")) !== FALSE) {
  $col1[] = $row[0];
}

由于我的托管服务提供商对PHP文件执行时间限制（120秒），无法一次处理整个文件。

Because of an execution time limit on the PHP file by my hosting provider (120 seconds), I can't process the whole file in one run.

我尝试使用ftell（）和fseek（）来记住重新启动的最后位置。问题是，有时ftell（）位置在一行的中间，而恢复则意味着丢失了该行的前半部分。

I tried using ftell() and fseek() to remember the last position for restart. The trouble is, sometimes the ftell() position is in the middle of a row, and resuming means missing the first half of the row.

有没有一种优雅的方法可以知道最后一行已成功处理，并从后一行继续吗？我知道我可以做一个简单的计数器，然后再次循环到该点，但这会在我可以处理到文件末尾的行上产生递减的收益。

Is there an elegant way to know the last line successfully processed, and resume from the one after it? I realize I can do a simple counter, and then loop through to that point again, but that would produce diminishing returns on the rows I can process towards the end of the file.

是否有适用于我的情况的ftell（）和fseek（）之类的东西？还是一种限制ftell（）返回上一行结尾的指针的方法？

Is there something like ftell() and fseek() that would work in my case? Or a way to limit ftell() to return the pointer for the end of the previous line?

推荐答案

当我需要工作时对于很大的文件，我总是使用分而治之的前提。对于您的情况，我将：

When i needed to work with files that big I always use the 'divide and conquer' premise. For your case I would:

动态创建一个文件夹

将这个大文件复制到其中

将其拆分（在从php调用的Linux分割上）分割命令

使用php
中的 shell_exec 命令

将其拆分后，删除它（大文件）

然后依次循环读取文件夹中的文件。

对于我完成的每个文件，我都将其删除。因此，如果出现时间限制，则只需继续读取文件夹中剩余的文件即可。

Dynamicaly create a folder

Copy this big file inside it

Split it (on linux split called from php) split command

use the shell_exec command in php

After split it, delete it (the big file)

Then loop through the files in the folder reading one by one.

And for every file I finish I delete it. So if the time limit occurs you will need just to continue reading the files left in the folder.

这篇关于php用ftell解析csv的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！