用C#处理大文本文件 [英] Processing large text file in C#

问题描述

我有 4GB 以上的文本文件(csv 格式)，我想在 c# 中使用 linq 处理这个文件.

I have 4GB+ text files (csv format) and I want to process this file using linq in c#.

我在加载 csv 并转换为类后运行复杂的 linq 查询?

I run complex linq query after load csv and convert to class?

虽然应用程序内存是文件大小的两倍，但文件大小为 4gb.

but file size is 4gb although application memory double size of file.

我如何处理(linq 和新结果)大文件?

how can i process (linq and new result) large files?

谢谢

推荐答案

您可以逐行读取和处理文件，而不是将整个文件加载到内存中.

Instead of loading whole file into memory, you could read and process the file line-by-line.

using (var streamReader = new StreamReader(fileName))
{
    string line;
    while ((line = streamReader.ReadLine()) != null)
    {
        // analize line here
        // throw it away if it does not match
    }
}

如果您需要对文件中的数据运行复杂的查询，正确的做法是将数据加载到数据库中，让 DBMS 负责数据检索和内存管理.

If you need to run complex queries against the data in the file, the right thing to do is to load the data to database and let DBMS to take care of data retrieval and memory management.

这篇关于用C#处理大文本文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！