读取巨大的文本文件并将每一行存储在数据库中 [英] Read through huge text files and store each line in database

问题描述

我编写了一个PHP脚本，该脚本运行一个文本文件(实际上是imdb的列表"文件)，并将其存储在本地MySQL数据库中.

I wrote a PHP script which runs through a text file (actually it's a 'list' file from imdb) and stores it on my local MySQL database.

public static function updateMovies( $list ) {
    $offset = 15;               // movies.list start with movie names at line 16
    $handle = fopen($list, "r") or die("Couldn't get handle");
    if ($handle) {
        while (!feof($handle)) {
            $buffer = fgets($handle);
            if($offset!=0)
                $offset--;
            else
                if($buffer[0] != '"'){
                    $title = trim( substr( $buffer, 0, strpos( $buffer, '(' ) ) );
                    $year = intval(trim( substr( $buffer, strpos( $buffer,'(' )+1, 4 ) ));
                    Movie::create( $title, $year );
                }
        }
        fclose($handle);
    }
}

由于这些列表文件最大为200MB，因此需要花费大量时间.默认情况下，PHP的MAX_EXECUTION_TIME设置为30秒.

Since those list-files are up to 200MB it takes a lot of time. By Default PHP's MAX_EXECUTION_TIME is set to 30 seconds.

我将此值设置为300只是为了尝试它是否有效.例如，我的"movies.list"文件约为80MB，并使用此脚本运行了300秒，在数据库中创建了约25000行.这是行不通的，因为我什至没有看过以"B"开头的电影.

I set this value to 300 just to try if it works. For example, my 'movies.list' file is around 80MB and using this script for 300 seconds created around 25000 lines in my database. This doesn't work because I have not even reached the movies starting with 'B'.

我知道我可以将MAX_EXECUTION_TIME设置为0(无限制)，但是将来我不希望该数据库位于我的本地主机上.据我所知，我希望在我的Web服务器上将它放置在我的Web服务器上，并将MAX_EXECUTION_TIME设置为90.

I know I can set the MAX_EXECUTION_TIME to 0 (unlimited) but in the future I don't want this database to be on my localhost. I want it on my webserver and my webserver hosts MAX_EXECUTION_TIME is set to 90 as far as I know.

有什么想法您将如何处理?

Any ideas how you would handle this?

推荐答案

您可以: 使用 set_time_limit(sec) 或者(最好)通过cron条目从命令行中运行脚本.这样，您将避免其他许多非PHP相关的超时问题.

You may either: Use set_time_limit(sec) or (better) run your script from the command line through a cron entry. That way you will avoid many other non-php related timeout issues.

这篇关于读取巨大的文本文件并将每一行存储在数据库中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！