批处理ADO .NET数据集中的行. [英] Batch processing of rows in a ADO .NET dataset.

问题描述

我有一个数据集，其中要提取大量行，即约100万行.

现在，我将在此数据集中获取的行发送到一个批量大小为100的方法中.

我的代码如下:

Hi,

I have a Dataset in which I am fetching a very large number of rows i.e. around 1 million.

Now, I am sending the rows fetched in this Dataset into a method in batch sizes of 100 each.

My code is as below :

List<Zsfpdate> myDataList = null;
foreach (DataRow dRow in ds.Tables[0].Rows)
{

	if ((count++ % 100) == 0)
	{
		myDataList = new List<Zsfpdate>();
		copyTable = new DataTable();
		ds1 = new DataSet();
		copyTable = ds.Tables[0].Clone();
		ds1.Tables.Add(copyTable);

	}

	copyTable.ImportRow(dRow);

	if (copyTable.Rows.Count == 100)
	{
		Program obj = new Program();
		obj.SendForProcessing(copyTable, myDataList);
	}

}

现在的问题是这种情况:

Now the problem is this condition :

if (copyTable.Rows.Count == 100) i.e. everytime I need to check this condition & then send the rows to the method for processing.

但是，这样一来，我们将无法捕获剩余的行，也就是说，在数据集中，我们有503行.
现在，该批处理将运行5次，每次将发送100行到该方法.但是最后3行不会以这种方式捕获.

有人可以建议解决方法吗?

谢谢，
Aaakar

However, in this way we are not able to catch the remaining rows i.e. Lets say we have 503 rows in the Dataset.
Now, the batch will run 5 times each with a 100 rows sent to the method each time. But the last 3 rows will not get captured in this way.

Can anybody suggest a workaround?

Thanks,
Aaakar

推荐答案

尝试类似的方法:

Try something like this:

int count = 0;
DataTable copyTable = null;
List<Zsfpdate> myDataList = null;

foreach (DataRow dRow in ds.Tables[0].Rows)
{
    if ((count % 100) == 0)
    {
        if (copyTable != null && copyTable.Rows.Count != 0)
        {
            Program obj = new Program();
            obj.SendForProcessing(copyTable, myDataList);
        }
        
        myDataList = new List<Szfpdate>();
        copyTable = ds.Tables[0].Clone();
        ds1 = new DataSet();
        ds1.Tables.Add(copyTable);
    }
    
    copyTable.ImportRow(dRow);
    count++;
}

if (copyTable != null && copyTable.Rows.Count != 0)
{
    Program obj = new Program();
    obj.SendForProcessing(copyTable, myDataList);
}

您可以在foreach循环下面检查copyTable是否包含任何剩余的行.

同样，您似乎在第一个if块中做了一些不必要的事情:您正在对copyTable进行两次初始化，并且从不将DataSet ds1用于任何事情.我想知道myDataList有什么用，因为您将其空发送到SendForProcessing(..)中，即使该方法向其中插入了某些内容，之后也不再在此处使用它.因此，对我来说似乎是多余的，至少在这里-如果您需要SendForProcessing(..)中的一个，则可以在其中创建它. (除非您在这里省略了代码.)

我已注释掉我认为不是必需的或已替换的行:

You can just check below the foreach-loop if the copyTable contains any left-over rows.

Also, it appears you''re doing some unneccessary stuff in the first if-block: You''re double-initializing copyTable and you never use the DataSet ds1 for anything. And I''m wondering what the myDataList is good for because you''re sending it empty into SendForProcessing(..) and even if that method inserts something into it, after that you don''t use it here. So it seems redundant to me, at least here - if you need one in SendForProcessing(..) then you could create it there. (Unless you''ve left out code here.)

I''ve commented out the lines which I think aren''t required or which I have replaced:

//List<Zsfpdate> myDataList = null;

copyTable = ds.Tables[0].Clone();

foreach (DataRow dRow in ds.Tables[0].Rows)
{
    //if ((count++ % 100) == 0)
    //{
        //myDataList = new List<Zsfpdate>();
        //copyTable = new DataTable();
        //ds1 = new DataSet();
        //copyTable = ds.Tables[0].Clone();
        //ds1.Tables.Add(copyTable);
    //}
 
    copyTable.ImportRow(dRow);
 
    if (copyTable.Rows.Count == 100)
    {
        //Program obj = new Program();
        //obj.SendForProcessing(copyTable, myDataList);
        new Program().SendForProcessing(copyTable);
        copyTable = ds.Tables[0].Clone();
    }
}

if (copyTable.Rows.Count > 0)
{
    new Program().SendForProcessing(copyTable);
}

您还可以尝试另一种循环:

You can also try another kind of loop:

for(first=0; first< ds.Tables[0].Rows.Count ; first+=100)
{
    last= min(first+100, ds.Tables[0].Rows.Count);

    myDataList = new List<zsfpdate>();
    copyTable = new DataTable();
    ds1 = new DataSet();
    copyTable = ds.Tables[0].Clone();
    ds1.Tables.Add(copyTable);   

    for cnt= first; cnt < last; cnt++)
    {
        copyTable.ImportRow(ds.Tables[0].Rows[cnt]);
    }

    //Program obj = new Program();
    //obj.SendForProcessing(copyTable, myDataList);
    new Program().SendForProcessing(copyTable);
    copyTable = null;
}

未经测试
从原理上讲，您可以建立一个外部循环，告诉您要处理哪个博客.
注意:您的原始循环看起来有点复杂.

Untested
The principle, you make an outer loop that tell you which blog to process.
Nota: your original loop look a little complicated.

这篇关于批处理ADO .NET数据集中的行.的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！