为什么stream parallel()不使用所有可用线程? [英] Why does stream parallel() not use all available threads?

问题描述

我尝试使用Java8(1.8.0_172)stream.parallel()并行运行100个 Sleep 任务，这些任务是在具有100多个可用线程的自定义ForkJoinPool中提交的.每个任务都会睡眠 1秒.考虑到100个睡眠可以并行进行，我预计整个工作将在大约1秒钟后完成.但是我观察到7s的运行时间.

I tried to run 100 Sleep tasks in parallel using Java8(1.8.0_172) stream.parallel() submitted inside a custom ForkJoinPool with 100+ threads available. Each task would sleep for 1s. I expected the whole work would finish after ~1s, given the 100 sleeps could be done in parallel. However I observe a runtime of 7s.

    @Test
    public void testParallelStream() throws Exception {
        final int REQUESTS = 100;
        ForkJoinPool forkJoinPool = null;
        try {
            // new ForkJoinPool(256): same results for all tried values of REQUESTS
            forkJoinPool = new ForkJoinPool(REQUESTS);
            forkJoinPool.submit(() -> {

                IntStream stream = IntStream.range(0, REQUESTS);
                final List<String> result = stream.parallel().mapToObj(i -> {
                    try {
                        System.out.println("request " + i);
                        Thread.sleep(1000);
                        return Integer.toString(i);
                    } catch (InterruptedException e) {
                        throw new RuntimeException(e);
                    }
                }).collect(Collectors.toList());
                // assertThat(result).hasSize(REQUESTS);
            }).join();
        } finally {
            if (forkJoinPool != null) {
                forkJoinPool.shutdown();
            }
        }
    }

输出指示〜16个流元素在暂停1秒之前执行，然后再执行〜16，依此类推.因此，即使forkjoinpool是使用100个线程创建的，似乎也只有约16个被使用.

With output indicating ~16 stream elements are executed before a pause of 1s, then another ~16 and so on. So it seems even though the forkjoinpool was created with 100 threads, only ~16 get used.

当我使用超过23个线程时，就会出现这种模式:

This pattern emerges as soon as I use more than 23 threads:

1-23 threads: ~1s
24-35 threads: ~2s
36-48 threads: ~3s
...

System.out.println(Runtime.getRuntime().availableProcessors());
// Output: 4

推荐答案

由于Stream实现对Fork/Join池的使用是实现细节，因此也没有强制其使用其他Fork/Join池的技巧.并且似乎偶然地工作，即有一个

Since the Stream implementation’s use of the Fork/Join pool is an implementation detail, the trick to force it to use a different Fork/Join pool is undocumented as well and seems to work by accident, i.e. there’s a hardcoded constant determining the actual parallelism, depending on the default pool’s parallelism. So using a different pool was not foreseen, originally.

但是，已经认识到，即使未记录此技巧，使用带有不适当的目标并行性的其他池也是一个错误，请参见

However, it has been recognized that using a different pool with an inappropriate target parallelism is a bug, even if this trick is not documented, see JDK-8190974.

它已在Java 10中修复，并反向移植到Java 8，更新222.

It has been fixed in Java 10 and backported to Java 8, update 222.

因此，一个简单的解决方案世界就是更新Java版本.

So a simple solution world be updating the Java version.

您还可以更改默认池的并行度，例如

You may also change the default pool’s parallelism, e.g.

System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "100");

进行任何Fork/Join活动之前.

before doing any Fork/Join activity.

但这可能会对其他并行操作产生意想不到的影响.

But this may have unintended effects on other parallel operations.

这篇关于为什么stream parallel()不使用所有可用线程?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！