将交叉束作用域线程与矢量块一起使用 [英] Using crossbeam scoped threads with vector chunks

问题描述

我有一个元组向量，我将其分成多个块，然后将每个块交给自己的线程进行处理，最后将它们重组. 我一直在使用下面的代码，该代码使用std::thread，并且运行良好，但是需要大量克隆和重构，我想消除它们.

I have a vector of tuples, which I split into chunks, then hand each chunk off to its own thread for processing, and recombine them at the end. I've been using the following code which uses std::thread, and it's working well, but requires a lot of cloning and reconstituting which I'd like to eliminate.

// convert_bng is just an expensive function
let orig: Vec<(&f32, &f32)>
// orig gets populated here
let mut guards: Vec<JoinHandle<Vec<(i32, i32)>>> = vec![];
    // split into slices
    let mut size = orig.len() / NUMTHREADS;
    if orig.len() % NUMTHREADS > 0 {
        size += 1;
    }
    // if orig.len() == 0, we need another adjustment
    size = std::cmp::max(1, size);
    for chunk in orig.chunks(size) {
        let chunk = chunk.to_owned();
        let g = thread::spawn(move || {
            chunk.into_iter()
                 .map(|elem| convert_bng(elem.0, elem.1))
                 .collect()
        });
        guards.push(g);
    }
    let mut result: Vec<IntTuple> = Vec::with_capacity(orig.len());
    for g in guards {
        result.extend(g.join()
                       .unwrap()
                       .into_iter()
                       .map(|ints| {
                           IntTuple {
                               a: ints.0 as u32,
                               b: ints.1 as u32,
                           }
                       }));
    }

是否可以使用crossbeam或scoped_threadpool简化此操作?像这样:

Is is possible to simplify this with crossbeam or scoped_threadpool? Something like:

let mut size = orig.len() / NUMTHREADS;
if orig.len() % NUMTHREADS > 0 {
    size += 1;
}
size = std::cmp::max(1, size);
    crossbeam::scope(|scope| {
        for chunk in orig.chunks_mut(size) {
            scope.spawn(move || chunk.iter().map(|elem| convert_bng(elem.0, elem.1)).collect());
        }
    });
    let mut result = orig.into_iter()
                     .map(|ints| {
                         IntTuple {
                             a: ints.0 as u32,
                             b: ints.1 as u32,
                         }
                     })
                     .collect();
}

(使用shepmaster链接到的问题中的代码进行编辑)

(edited with code from the question to which shepmaster linked)

但是，这给了我let result …错误:
casting &f32 as u32 is invalid，指示scope.spawn()内部的map调用未将其结果返回到块中，或者由于类型不匹配而被丢弃(该函数返回(i32, i32)，但orig保留(&f32, &f32)) .如果将有效的伪向量替换为result，则会得到与spawn有关的完全不同的错误:

However, this gives me an error in let result …:
casting &f32 as u32 is invalid, which indicates that the map call inside scope.spawn() isn't returning its result into the chunk, or is being discarded due to a type mismatch (the function returns (i32, i32), but orig holds (&f32, &f32)). If I substitute a valid dummy vector for result, I get a completely different error related to spawn:

error: unable to infer enough type information about `_`; type annotations or generic parameter binding required [E0282]
scope.spawn(move || chunk.iter().map(|elem| convert_bng(elem.0, elem.1)).collect());

推荐答案

表示scope.spawn()内部的map调用未将其结果返回到块中

which indicates that the map call inside scope.spawn() isn't returning its result into the chunk

是真的，因为那不是该方法的工作方式.通常的想法是先建立一个新的作用域，然后可以生成保证在该作用域之前终止的线程.

That's true, because that's not how that method works. The general idea is you establish a new scope and then you can spawn threads that are guaranteed to end before that scope does.

crossbeam::scope确定了范围，并且定义为:

pub fn scope<'a, F, R>(f: F) -> R 
    where F: FnOnce(&Scope<'a>) -> R

也就是说，您给scope加上了结尾.该闭包将被引用为Scope.闭包返回的任何内容都将从`scope返回.

That is, you give a closure to scope. The closure will be given a reference to a Scope. Whatever the closure returns will be returned from `scope.

在该封闭中，您可以使用 Scope::spawn :

Inside that closure, you can spawn threads with Scope::spawn:

fn spawn<F, T>(&self, f: F) -> ScopedJoinHandle<T> 
    where F: FnOnce() -> T + Send + 'a,
          T: Send + 'a

spawn采用不带参数的闭包，并返回某种类型.该类型将是spawn方法的结果(对某些错误进行处理).

spawn takes a closure that takes no arguments and returns some type. That type will be the result of the spawn method (modulo some error handling).

横梁不会固有地修改任何数据，这取决于您的代码.因此，让我们看看您在线程中正在做什么:

Nothing of crossbeam inherently modifies any data, that's up to your code. So let's look at what you are doing in the thread:

chunk.iter().map(|elem| convert_bng(elem.0, elem.1)).collect()

您将您的&mut [T]进行遍历，将每个元素传递给convert_bng进行转换.您没有convert_bng的定义，因此可以说它返回bool.然后，您collect这个bool的迭代器.但是，您可能要针对许多可能的集合，因此我们需要知道需要什么具体的集合.在大多数情况下，这是通过let foo: ConcreteCollection = iter.collect()之类的方法完成的.

You take your &mut [T] and iterate over it, converting each element by passing it to convert_bng. You don't have the definition of convert_bng, so let's say it returns a bool. You then collect this iterator of bools. However, there's a multitude of possible collections that you could be targeting, so we need to know what concrete collection is desired. In most cases, this is done by something like let foo: ConcreteCollection = iter.collect().

由于此collect是最后一个表达式，所以它也是spawn的返回值，因此我们可以看看如何使用该返回值.事实证明，不是，这相当于只说iterator.collect();，没有足够的信息要编译.

Since this collect is the last expression, it is also the return value of the spawn, so we can look to see how that return value is used. Turns out it isn't, which would be equivalent to just saying iterator.collect();, which doesn't have enough information to be compiled.

除此之外，您还需要确定是否要就地修改集合.您执行chunks_mut的事实似乎表明您希望每个线程在没有任何额外分配的情况下对向量执行工作.但是，您忽略它的可变性并返回新分配的(不确定类型的)集合.由于我不知道convert_bng会返回什么，因此很难说是否有可能.另外，map仅在将类型转换为新类型时使用，而不是在适当的位置更改值.您不能在原位更改切片以在其中放置其他类型，因为它将不再是切片！

Beyond that, you need to decide if you are trying to modify a collection in-place or not. The fact that you do chunks_mut seems to indicate that you want each thread to perform work on the vector without any extra allocation. However, you ignore that it's mutable and return newly allocated collections (of an indeterminate type). Since I don't know what convert_bng returns, it's hard to say if it's even possible. Additionally, map is only used when transforming a type to a new type, not to mutate a value in place. You can't mutate a slice in-place to put a different type in there, as it would no longer be a slice!

这篇关于将交叉束作用域线程与矢量块一起使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！