块分区IEnumerable in Parallel.Foreach [英] Chunk partitioning IEnumerable in Parallel.Foreach

问题描述

有没有人知道获得Parallel.Foreach循环使用块分区的方法，而我认为默认情况下是范围分区.使用数组似乎很简单，因为您可以创建一个自定义分区程序并将负载平衡设置为true.

Does anyone know of a way to get the Parallel.Foreach loop to use chunk partitioning versus, what i believe is range partitioning by default. It seems simple when working with arrays because you can just create a custom partitioner and set load-balancing to true.

由于直到运行时才知道IEnumerable中的元素数量，所以我似乎无法找出一种使块分区正常工作的好方法.

Since the number of elements in an IEnumerable isn't known until runtime I can't seem to figure out a good way to get chunk partitioning to work.

任何帮助将不胜感激.

谢谢！

我要在每个对象上执行的任务花费的时间明显不同.最后，我通常要等待几个小时才能等待最后一个线程完成其工作.我要实现的目标是沿途具有并行循环请求块，而不是将项目预先分配给每个线程.

The tasks i'm trying to perform on each object take significantly different times to perform. At the end i'm usually waiting hours for the last thread to finish its work. What I'm trying to achieve is to have the parallel loop request chunks along the way instead of pre-allocating items to each thread.

推荐答案

如果您的IEnumerable确实具有索引器(例如，您可以执行obj[1]来取出项目)，则可以执行以下操作

If your IEnumerable was really something that had a an indexer (i.e you could do obj[1] to get a item out) you could do the following

    var rangePartitioner = Partitioner.Create(0, source.Length);
    Parallel.ForEach(rangePartitioner, (range, loopState) =>
    {
        // Loop over each range element without a delegate invocation. 
        for (int i = range.Item1; i < range.Item2; i++)
        {
            var item = source[i]
            //Do work on item
        }
    });

但是，如果不能这样做，则必须通过创建一个从System.Collections.Concurrent.Partitioner<TSource>派生的新类来编写自定义分区程序.该主题范围太广，无法涵盖在SO答案中，但是您可以在 MSDN 让您入门.

However if it can't do that you must write a custom partitioner by creating a new class derived from System.Collections.Concurrent.Partitioner<TSource>. That subject is too broad to cover in a SO answer but you can take a look at this guide on the MSDN to get you started.

更新:从.NET 4.5开始，他们添加了

UPDATE: As of .NET 4.5 they added a Partitioner.Create overload that does not buffer data, it has the same effect of making a custom partitioner with a range max size of 1. With this you won't get a single thread that has a bunch of queued up work if it got unlucky with a bunch of slow items in a row.

var partitoner = Partitioner.Create(source, EnumerablePartitionerOptions.NoBuffering);
Parallel.ForEach(partitoner, item =>
{
    //Do work
}

这篇关于块分区IEnumerable in Parallel.Foreach的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！