并行处理C＃ - 将数据合并到全局数据表 [英] Parallel Processing C# - Merge data to global Datatable

问题描述

我在strFilePath数组中有各种csv文件的位置（几乎1000个文件，每个文件中有100万条记录）。从文件中读取并将所有数据合并到单个数据表需要花费大量时间。所以我决定继续进行并行处理。问题是，如何将数据表dt中的所有数据合并到全局数据表dtMerge？

I have location of various csv files in "strFilePath" array(almost 1000 files which have 1 million records in each file). It takes a lot of time to read from files and merge all the data to single datatable. So I decided to go forward with Parallel processing. The question is, how can I merge all the data from datatable "dt" to a global datatable "dtMerge"?

DataTable dtMerge=new DataTable();
for(int i=0;i<strFilePath.Count;i++)
{
     Parallel.For(0, 3,m =>
          clsNewClass objCls=new clsNewClass();
          DataTable dt=objCls.ReadCSV(strFilePath[m+i]);
      });
      m+=3;
}

推荐答案

您可以尝试此代码段。

You can try this code snippet.

object padlock = new object();
DataTable dtMerge = new DataTable();
ParallelLoopResult result = Parallel.For(0, strFilePath.Count, index =>
{
    Debug.WriteLine("Iteration {0} : {1}", index, strFilePath[index]);
    clsNewClass objCls = new clsNewClass();
    DataTable dt = objCls.ReadCSV(strFilePath[index]);
    lock (padlock) // Makes sure only one merge operation is done at a time
    {
        dtMerge.Merge(dt);
    }
});
Debug.WriteLine("Result: {0}", result.IsCompleted ? "Completed Normally" : String.Format("Completed to {0}", result.LowestBreakIteration));

您可能还应该查看从CSV文件中读取数据的方式。

如果有这么多行的文件很多，那么小代码中的低效率会增加很多时间损失。

即使每个文件一毫秒，也会减少1秒的时间。

You should probably also look into the way you read data from the CSV file.
When you have so many files with that many lines, small inefficiencies here and there in the code will add up to a lot of time losses.
Even a millisecond per file will add up to 1 second slower time.

这篇关于并行处理C＃ - 将数据合并到全局数据表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！