我怎么知道Java中的Fork and Join是否具有足够的池大小? [英] How do I know if Fork and Join has enough pool size in Java?

问题描述

我正在尝试对某些大数据实施分而治之的解决方案.我使用fork和join将事物分解为线程.但是我对派生机制有一个疑问:如果我将分治条件设置为:

@Override
protected SomeClass compute(){
    if (list.size()<LIMIT){
        //Do something here
        ...
    }else{
        //Divide the list and invoke sub-threads
        SomeRecursiveTaskClass subWorker1 = new SomeRecursiveTaskClass(list.subList());
        SomeRecursiveTaskClass subWorker2 = new SomeRecursiveTaskClass(list.subList());
        invokeAll(subWorker1, subWorker2);
        ...
    }
}

如果没有足够的资源来调用subWorker(例如，池中没有足够的线程)，将会发生什么? Fork/Join框架是否维护可用线程的池大小?还是应该将此条件添加到我的分而治之"逻辑中?

解决方案

每个ForkJoinPool具有已配置的目标 默认为"CPU核心数减一"，因此，在将启动的非池线程作为辅助程序合并时，所产生的并行性将利用所有CPU核心.

当您提交的作业多于线程时，它们将被排队.将一些作业排入队列可以帮助利用线程，因为并非所有作业都可以在同一时间运行，因此耗尽工作的线程可能会从其他线程中窃取作业，但是过多地拆分工作可能会造成不必要的开销.

因此，您可以使用 解决方案

Each ForkJoinPool has a configured target parallelism. This isn’t exactly matching the number of threads, i.e. if a worker thread is going to wait via a ManagedBlocker, the pool may start even more threads to compensate. The parallelism of the commonPool defaults to "number of CPU cores minus one", so when incorporating the initiating non-pool thread as helper, the resulting parallelism will utilize all CPU cores.

When you submit more jobs than threads, they will be enqueued. Enqueuing a few jobs can help utilizing the threads, as not all jobs may run exactly the same time, so threads running out of work may steal jobs from other threads, but splitting the work too much may create an unnecessary overhead.

Therefore, you may use ForkJoinTask.getSurplusQueuedTaskCount() to get the current number of pending jobs that are unlikely to be stolen by other threads and split only when it is below a small threshold. As its documentation states:

This value may be useful for heuristic decisions about whether to fork other tasks. In many usages of ForkJoinTasks, at steady state, each worker should aim to maintain a small constant surplus (for example, 3) of tasks, and to process computations locally if this threshold is exceeded.

So this is the condition to decide whether to split your jobs further. Since this number reflects when idle threads steal your created jobs, it will cause balancing when the jobs have different CPU load. Also, it works the other way round, if the pool is shared (like the common pool) and threads are already busy, they will not pick up your jobs, the surplus count will stay high and you will automatically stop splitting then.

这篇关于我怎么知道Java中的Fork and Join是否具有足够的池大小?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！