使用缓冲在 C++ 中逐行读取巨大的文本文件 [英] read huge text file line by line in C++ with buffering

问题描述

我需要在 C++ 中逐行从光盘读取巨大的 35G 文件.目前我是这样做的:

I need to read huge 35G file from disc line by line in C++. Currently I do it the following way:

ifstream infile("myfile.txt");
string line;
while (true) {
    if (!getline(infile, line)) break;
    long linepos = infile.tellg();
    process(line,linepos);
}

但它给了我大约 2MB/秒的性能，尽管文件管理器以 100Mb/s 的速度复制文件.我猜 getline() 没有正确进行缓冲.请提出某种缓冲的逐行读取方法.

But it gives me about 2MB/sec performance, though file manager copies the file with 100Mb/s speed. I guess that getline() is not doing buffering correctly. Please propose some sort of buffered line-by-line reading approach.

UPD:process() 不是瓶颈，没有 process() 的代码以相同的速度工作.

UPD: process() is not a bottleneck, code without process() works with the same speed.

推荐答案

我已经从我的 java 项目中翻译了我自己的缓冲代码，它满足了我的需要.我不得不放置定义来克服 M$VC 2010 编译器告诉的问题，它总是在大文件上给出错误的负值.该算法提供了大约 100MB/s 的所需速度，尽管它做了一些无用的 new[].

I've translated my own buffering code from my java project and it does what I need. I had to put defines to overcome problems with M$VC 2010 compiler tellg, that always gives wrong negative values on huge files. This algorithm gives desired speed ~100MB/s, though it does some usless new[].

void readFileFast(ifstream &file, void(*lineHandler)(char*str, int length, __int64 absPos)){
        int BUF_SIZE = 40000;
        file.seekg(0,ios::end);
        ifstream::pos_type p = file.tellg();
#ifdef WIN32
        __int64 fileSize = *(__int64*)(((char*)&p) +8);
#else
        __int64 fileSize = p;
#endif
        file.seekg(0,ios::beg);
        BUF_SIZE = min(BUF_SIZE, fileSize);
        char* buf = new char[BUF_SIZE];
        int bufLength = BUF_SIZE;
        file.read(buf, bufLength);

        int strEnd = -1;
        int strStart;
        __int64 bufPosInFile = 0;
        while (bufLength > 0) {
            int i = strEnd + 1;
            strStart = strEnd;
            strEnd = -1;
            for (; i < bufLength && i + bufPosInFile < fileSize; i++) {
                if (buf[i] == '\n') {
                    strEnd = i;
                    break;
                }
            }

            if (strEnd == -1) { // scroll buffer
                if (strStart == -1) {
                    lineHandler(buf + strStart + 1, bufLength, bufPosInFile + strStart + 1);
                    bufPosInFile += bufLength;
                    bufLength = min(bufLength, fileSize - bufPosInFile);
                    delete[]buf;
                    buf = new char[bufLength];
                    file.read(buf, bufLength);
                } else {
                    int movedLength = bufLength - strStart - 1;
                    memmove(buf,buf+strStart+1,movedLength);
                    bufPosInFile += strStart + 1;
                    int readSize = min(bufLength - movedLength, fileSize - bufPosInFile - movedLength);

                    if (readSize != 0)
                        file.read(buf + movedLength, readSize);
                    if (movedLength + readSize < bufLength) {
                        char *tmpbuf = new char[movedLength + readSize];
                        memmove(tmpbuf,buf,movedLength+readSize);
                        delete[]buf;
                        buf = tmpbuf;
                        bufLength = movedLength + readSize;
                    }
                    strEnd = -1;
                }
            } else {
                lineHandler(buf+ strStart + 1, strEnd - strStart, bufPosInFile + strStart + 1);
            }
        }
        lineHandler(0, 0, 0);//eof
}

void lineHandler(char*buf, int l, __int64 pos){
    if(buf==0) return;
    string s = string(buf, l);
    printf(s.c_str());
}

void loadFile(){
    ifstream infile("file");
    readFileFast(infile,lineHandler);
}

这篇关于使用缓冲在 C++ 中逐行读取巨大的文本文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！