分解CSV中的大量行并将其作为批处理php处理 [英] break down the large number of rows in CSV and process them as a batch wise php

问题描述

我可以编辑和上传包含超过50000条记录的CSV文件(购物车产品).并且它应该更新系统中的表数.所以我在购物车中使用了zend框架.

I am upto edit and upload a CSV with more then 50000 records (shopping cart produts). and it should update the number of tables in the system. So i m using zend framework with my shopping cart.

我打算在使用PHP/MYSQL分批处理它们之前先将其分解(50000个CSV记录)在内存中

im planing to break them down (50000 CSV records) in memory before processing them batch-wise using PHP/MYSQL

请任何人可以给我建议

目前为止我是什么

public function getDataFromPath($path=null) {
        if($path == null) {
            $path = $this->_path;
        }
        ini_set("auto_detect_line_endings", 1);
        $fp = fopen($path, "r");
        while(($line = fgetcsv($fp, 5000, ",")) !== FALSE) {
            $line = self::trimArray($line);
            $this->data[] = $line;
        }
        fclose($fp);          
        return $this->data;
    }

致谢 肉山

推荐答案

每个RDBMS不应对50.000行有问题.没什么.无需批量处理它们.

Every RDBMS out there shouldn't have problems with 50.000 rows. That's nothing. There's no need to process them batch wise.

只需使用 LOAD DATA INFILE 命令你会没事的.

Just use the LOAD DATA INFILE command and you will be fine.

有关示例，请参见此处: LOAD DATA INFILE容易地将YYYYMMDD转换为YYYY-MM-DD?

For an example see here: LOAD DATA INFILE easily convert YYYYMMDD to YYYY-MM-DD?

更新(由于对Ion Wood的回答有评论):要创建CSV文件，可以使用SELECT .. INTO OUTFILE命令.

UPDATE (cause of comment to Ion Wood's answer): To create a CSV file you can use the SELECT .. INTO OUTFILE command.

SELECT a,b,a+b INTO OUTFILE '/tmp/result.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM test_table;

有关更多信息，请参见手册.

For more info, see the manual.

这篇关于分解CSV中的大量行并将其作为批处理php处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！