从php中的csv文件读取大数据 [英] Read large data from csv file in php

问题描述

我正在阅读csv&用mysql检查记录是否存在于我的表中或不在php中。

csv有大约25000条记录，当我运行我的代码，它显示服务不可用错误后2m 10s（onload：2m 10s）

这里我添加了代码

  //设置内存限制&执行时间
 ini_set（'memory_limit'，'512M'）; 
 ini_set（'max_execution_time'，'180'）; 
 
 //读取csv文件的函数
 function readCSV（$ csvFile）
 {
 $ file_handle = fopen（$ csvFile，'r'）; 
 while（！feof（$ file_handle））{
 
 set_time_limit（60）; //你可以启用这个，如果你有很多数据
 
 $ line_of_text [] = fgetcsv（$ file_handle，1024）; 
} 
 fclose（$ file_handle）; 
 return $ line_of_text; 
} 
 
 //设置CSV文件的路径
 $ csvFile ='my_records.csv'; 
 
 $ csv = readCSV（$ csvFile）; 
 
 for（$ i = 1; $ i< count（$ csv）; $ i ++）
 {
 $ user_email = $ csv [$ i] [1] 
 
 $ qry =SELECT u.user_id，u.user_email_id FROM tbl_user as u WHERE u.user_email_id ='。$ user_email。'; 
 
 $ result = @mysql_query（$ qry）或die（Could not execute query：mysql_error（）。''。mysql_errno（））; 
 
 $ rec = @mysql_fetch_row（$ result）; 
 
 if（$ rec）
 {
 echoRecord exists; 
} 
 else 
 {
 echoRecord not exist; 
} 
} 
  

注意：我只想列出记录

解决方案

div>

处理大型文件的出色方法位于： http://stackoverflow.com/a/5249971/797620

此方法用于 http： //www.cuddlycactus.com/knownpasswords/ 在短短几毫秒内搜索超过1.7亿个密码。

I am reading csv & checking with mysql that records are present in my table or not in php.

csv has near about 25000 records & when i run my code it display "Service Unavailable" error after 2m 10s (onload: 2m 10s)

here i have added code

// for set memory limit & execution time
ini_set('memory_limit', '512M');
ini_set('max_execution_time', '180');

//function to read csv file
function readCSV($csvFile)
{
    $file_handle = fopen($csvFile, 'r');
    while (!feof($file_handle) ) {

       set_time_limit(60); // you can enable this if you have lot of data

       $line_of_text[] = fgetcsv($file_handle, 1024);
   }
   fclose($file_handle);
   return $line_of_text;
 }

// Set path to CSV file
$csvFile = 'my_records.csv';

$csv = readCSV($csvFile);

for($i=1;$i<count($csv);$i++)
{
   $user_email= $csv[$i][1];

   $qry = "SELECT u.user_id, u.user_email_id FROM tbl_user as u WHERE u.user_email_id = '".$user_email."'";

   $result = @mysql_query($qry) or die("Couldn't execute query:".mysql_error().''.mysql_errno());

   $rec = @mysql_fetch_row($result);

   if($rec)
   {
      echo "Record exist";
   }
   else
   {
      echo "Record not exist"; 
   }
}

Note: I just want to list out records those are not exist in my table.

Please suggest me solution on this...

解决方案

An excellent method to deal with large files is located at: http://stackoverflow.com/a/5249971/797620

This method is used at http://www.cuddlycactus.com/knownpasswords/ to search through 170+ million passwords in just a few milliseconds.

这篇关于从php中的csv文件读取大数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！