C＃处理循环中的大项目列表，如果失败，则重试 [英] C# process large list of items in a loop and retry if fails

问题描述

我有一个大型的数据库，我使用实体框架来访问它。我把所有的项目都作为 List< Item> 。当我循环处理它们时：

  List< Item> lstItems = UnitOfWork.Items.GetAllAsync（）; //存储库模式和工作单元模式
 foreach（物料项目in lstItems）
 {
 //对物品做
} 
  
 
 
任何我可以用于这个目的的模式的想法？
 
 
我正在考虑使用try catch，如果处理失败，然后在catch部分添加到一个新的 List< Item> 调用 processAgain ，一旦主列表完成处理，然后处理 processAgain  list 
解决方案
当您要完成并行处理时，使用 code>和异步处理将是一个很好的方法。 
因此，您需要定义一个方法或表达式，它们将是您的任务 / 动作，并包含一个内部重试模式： 
 
 
  public const int MAX_RETRY_COUNT = 3; 
 
 private void ProcessItemsAsync（Item item，int retryCount）
 {
 //注意：抛出的异常会弹出
 //作为Task.WaitAll中的AggregateException ）
 if（retryCount> = MAX_RETRY_COUNT）
抛出新的InvalidOperationException（
超过最大重试次数）; 
 
 retryCount ++; 
 //要么执行try-catch，要么使用条件运算符。 
尝试
 {
 //执行项目
的任何操作} 
 catch（Exception ex）
 {
 //异常记录相关？如果没有，只需重试
 ProcessItemsAsync（item，retryCount）; 
 
 
 
 
 $ b $ p 
 $ b 
一旦你定义了你的任务方法，您可以一次处理大量任务： 
 
 
  public const int BULK_AMOUNT = 10; 
 
 private async void ProcessSqlData（）
 {
 List< Item> lstItems =等待UnitOfWork.Items.GetAllAsync（）; 
 for（var i = 0; i< lstItems.Count; i + = BULK_AMOUNT）
 {
 //运行与多个项目并行的进程
 //可能会慢（BULK_AMOUNT）.ToArray（）;我们可以通过下面的例子来说明如何使用BULK_AMOUNT。 
 var tasks = new Task [bulk.Length]; 
 for（var j = 0; j <= bulk.Length; j ++）
 {
 //创建并启动任务，使用ProcessItemsAsync作为动作
 tasks [j] = Task.Factory.StartNew（（）=> ProcessItemsAsync（bulk [j]，0））; 
} 
 //等待批量完成
尝试
 {
 Task.WaitAll（tasks）; 
 
 catch（AggregateException e）
 {
 Log.WriteLine（String.Format（
批量＃{0}错误信息：{1}，
i，
 e.InnerException！= null 
？e.InnerException.Message 
：e.Message））; 
 
 
 
 
 
 $ p $但是，如果你知道你的计算机运行这个有足够的性能，你可以增加 BULK_AMOUNT 。你应该测试它，在这里找到最佳的金额。
 
I have a large database and I access it using Entity Framework. I get all items as a List<Item>. When I process them in a loop:
List<Item> lstItems = UnitOfWork.Items.GetAllAsync(); // repository pattern and Unit Of Work patterns
foreach (Item item in lstItems)
{
  //do something with the item
}
What I need is if one of the items fail processing in the loop I want to be able to retry that action 3 times before ignoring it in this loop. Also I don't want the loop to brake, so no matter if an item processing is success or fail the loop should continue until the last item in the list.

Any idea of a pattern I can use for this purpose?

I was thinking about using try catch and if processing fails then in the catch section add it to a new List<Item> called processAgain and once the main list is finished processing then handle the processAgain list.
解决方案
As you want to achive parallel processing, using a Task and async processing would be a good approach.
Therefore, you need to define a method or expression that will be your Task/Action and includes a retry pattern internally:
public const int MAX_RETRY_COUNT = 3;

private void ProcessItemsAsync(Item item, int retryCount)
{        
    // Note: Exceptions thrown here will pop up
    // as AggregateException in Task.WaitAll()
    if (retryCount >= MAX_RETRY_COUNT)
        throw new InvalidOperationException(
            "The maximum amount of retries has been exceeded");

    retryCount++;
    // Either implement try-catch, or use conditional operator.
    try
    {
        // Do stuff with item
    }
    catch(Exception ex)
    {
        // Exception logging relevant? If not, just retry
        ProcessItemsAsync(item, retryCount);
    }
}
Once you defined your Task method, you can process a bulk of tasks at once:
public const int BULK_AMOUNT = 10;

private async void ProcessSqlData()
{
    List<Item> lstItems = await UnitOfWork.Items.GetAllAsync();
    for (var i = 0; i < lstItems.Count; i += BULK_AMOUNT)
    {
        // Running the process parallel with to many items
        // might slow down the whole process, so just take a bulk
        var bulk = lstItems.Skip(i).Take(BULK_AMOUNT).ToArray();
        var tasks = new Task[bulk.Length];
        for (var j = 0; j <= bulk.Length; j++)
        {
            // Create and start tasks, use ProcessItemsAsync as Action
            tasks[j] = Task.Factory.StartNew(() => ProcessItemsAsync(bulk[j], 0));
        }
        // Wait for the bulk to complete
        try
        {
            Task.WaitAll(tasks);
        }
        catch (AggregateException e)
        {
            Log.WriteLine(String.Format(
                "The maximum amount of retries has been exceeded in bulk #{0}. Error message: {1}",
                i,
                e.InnerException != null
                    ? e.InnerException.Message
                    : e.Message));
        }
    }
}
However, if you know that your computer running this has enough performance, you might increase the BULK_AMOUNT. You should test it, to find the optimal amount here.

                        
这篇关于C＃处理循环中的大项目列表，如果失败，则重试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！