如何用cstream中的fstream将数据写入大于2GB的文件 [英] how to write data to file larger than 2GB with fstream in c++

问题描述

我搜索了一个TON，找到一种方法，使用fstream或其他方法，将数据写入一个大文件，但没有找到任何答案。

I have searched a TON to find a way, using fstream or other methods, to write data into a large file, but have not found any answers.

我的问题是，我从互联网下载数据，然后将数据写入预分配的文件。对于小于2GB的文件，一切似乎都正常，但是当使用大于2GB的预分配文件进行尝试时，将流指针移动到预分配文件中的特定位置（例如文件的最后位置）时会出错。写数据，似乎这个不能做，tellp allway返回-1。

My problem is, i'm downloading data from the internet and then writing data into pre-allocated file. Everything seems OK with a file smaller than 2GB, but when trying with a pre-allocated file larger than 2GB, it gives an error when moving the stream pointer to a specific position (e.g last position of file) in the pre-allocated file to write data, it seems this cannot be done, tellp allway return -1 .

这是我写数据方法，因为写入预分配文件，我不使用FILE或boost：iosstream。我使用visual studio 2008与.Net 3.5。感谢您的阅读和帮助^。^

Here is my writing data method, because write to pre-allocate file, i'm not using FILE, or boost:iosstream. I'm using visual studio 2008 with .Net 3.5. Thanks for reading and helping me ^.^

size_t writeData(void *ptr, size_t size, size_t nmemb, Downloader *objDownloader){
    size_t written;     

    if (objDownloader->_stop == false && objDownloader->mystream.is_open()){

        objDownloader->mystream.seekp(objDownloader->_posSeek, ios::beg);           
        written=size * nmemb;
        objDownloader->mystream.write(static_cast<char*>(ptr), written);

        objDownloader->_posSeek += size*nmemb;
        objDownloader->_downloaded += size*nmemb;

        objDownloader->mystream.seekp(objDownloader->_currentPosition, ios::beg);   
        objDownloader->mystream.write((char*)&objDownloader->_downloaded, sizeof(long long));

        long long  pos = objDownloader->mystream.tellp();           
        cout << "POS TEL: " << pos << endl;

        return size * nmemb;
    }
    else{

        if (objDownloader->mystream.is_open())
            objDownloader->mystream.close();            
        return -1;
    }

}

推荐答案

您的问题与fstreams无关，而与进程内存无关。

Your problem is not related to fstreams but to process memory.

以下代码将生成3GB的文件，没有任何问题，请使用 tellp（）并将seekp（）置于文件末尾：

The following code will generate a file of 3GB without any problem, use tellp() to show the progress, and positions with seekp() to the end of the file:

cout << "size_t is limted to " << numeric_limits<size_t>::max() << " bytes (" << numeric_limits<size_t>::max() /1024/1024<<" MB)"<< endl;
cout << "streamsize is limited to " << numeric_limits<streamsize>::max() << " bytes (" << numeric_limits<streamsize>::max() / 1024 / 1024/1024/104 << " TB)" << endl;

static char mbloc[1024*1024]; 
fill_n(mbloc, sizeof(mbloc), 'A'); 
const size_t blocs = 3*1024; 
ofstream os("test.out"); 
for(int i = 0; i < blocs && os; i++) {
    os.write(mbloc, sizeof(mbloc));
    if(i % 512)
        cout << "tellp() -> " << os.tellp() << "\r";
}
cout << endl;
os.seekp(0, ios::end); 
cout << os.tellp() << " end"<<endl;  
os.close();
cout << "File generation finished"; 

fstreams实际上被认为是非常大的。这就是为什么 ostream :: write（） 使用类型 streamsize 的计数。

fstreams are in fact thought to be incredibly large. This is why ostream::write() uses a count of type streamsize.

tellp男士因此，另一个失败发生。您可以通过检查fstream操作的状态来识别此故障的原因，以确定哪个成功和哪个失败。

The fact that tellp() returns -1 mens therefore that another failure took place. You could identify the origin of this failure by checking the status of your fstream operations, to identify which succeeds and which fails.

这里的声明表示您尝试从内存缓冲区写入所有内容：

Your statement here shows that you try to write everything from a memory buffer:

     objDownloader->mystream.write(static_cast<char*>(ptr), written);

不幸的是，您将遇到2GB限制的问题这可能导致写入失败，而不是因为fstream限制，而是底层操作系统的内存限制。

Unfortunately you will experience a problem with the limit of 2GB that applies to a 32 bit process under windows. This might cause a failure of writing, not because of fstream limits, but memory limits of the underlying OS.

如果您编译为64位并在64位进程上运行，您可以将此限制扩展到8TB，如果您的计算机上有足够的内存。但是，如果你操纵这样巨大的数据，你应该更好地写在磁盘上较小的块，而不必保持完整的文件在内存中。

If you compile for 64bit and run it on a 64 bit process, you could extend this limit to 8TB, if you have enough memory on your computer. But if you manipulate such huge data you should better write on disk in smaller chunks, without having to keep the full file in memory.

这篇关于如何用cstream中的fstream将数据写入大于2GB的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！