如何使用.net读取具有2900万行数据的巨大CSV文件 [英] How to read huge CSV file with 29 million rows of data using .net

问题描述

我有一个很大的.csv文件，具体来说是一个具有2900万行的.TAB文件，文件大小约为600 MB.我需要将其读入IEnumerable集合.

I have a huge .csv file, to be specific a .TAB file with 29 million rows and the file size is around 600 MB. I would need to read this into an IEnumerable collection.

我尝试过CsvHelper，GenericParser和其他一些解决方案，但总是以内存不足"异常结束

I have tried CsvHelper, GenericParser, and few other solutions but always ending up with an Out of Memory exception

请提出一种实现方法

我尝试过

var deliveryPoints = new List<Point>();

using (TextReader csvreader1 = File.OpenText(@"C:\testfile\Prod\PCDP1705.TAB")) //StreamReader csvreader1 = new StreamReader(@"C:\testfile\Prod\PCDP1705.TAB"))
using (var csvR1 = new CsvReader(csvreader1, csvconfig))
{
     csvR1.Configuration.RegisterClassMap<DeliveryMap>();
     deliveryPoints = csvR1.GetRecords<Point>().ToList();
}

using (GenericParser parser = new GenericParser())
{
     parser.SetDataSource(@"C:\testfile\Prod\PCDP1705.TAB");

     parser.ColumnDelimiter = '\t';
     parser.FirstRowHasHeader = false;
     //parser.SkipStartingDataRows = 10;
     //parser.MaxBufferSize = 4096;
     //parser.MaxRows = 500;
     parser.TextQualifier = '\"';

     while (parser.Read())
     {
         var address = new Point();
         address.PostCodeID = int.Parse(parser[0]);
         address.DPS = parser[1];
         address.OrganisationFlag = parser[2];
         deliveryPoints.Add(address);
     }
}

和

var deliveryPoints = new List<Point>();
csvreader = new StreamReader(@"C:\testfile\Prod\PCDP1705.TAB");
csv = new CsvReader(csvreader, csvconfig);

while (csv.Read())
{
     var address = new Point();
     address.PostCodeID = int.Parse(csv.GetField(0));
     address.DPS = csv.GetField(1);                
     deliveryPoints.Add(address);
}

推荐答案

通过在x64模式下运行并添加 <gcAllowVeryLargeObjects enabled="true" />在app.config中.

It worked by running in x64 mode, and by adding <gcAllowVeryLargeObjects enabled="true" /> in app.config.

这篇关于如何使用.net读取具有2900万行数据的巨大CSV文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！