使用分区的 Spring 批处理多线程 [英] Spring batch multithreading using partitioning

问题描述

我的问题陈述是-我必须将多个文件传递给 spring 批处理读取器，并且读取器以并行方式运行.如果我们使用 grid-size = 100，那么将有 100 个不合逻辑的线程.解决此问题的方法是什么，即以有限的线程数处理许多文件.

My problem statement is that- I have to pass multiple numbers of files to spring batch reader and reader runs in parellel.if we use grid-size = 100 then there will be 100 threads which is not logical. what is the way to solve this issue i.e. process many files with limited number of threads.

@Bean
    public Step orderStep1() throws IOException {
        return stepBuilderFactory.get("orderStep1")
                .partitioner("slaveStep", partitioner())
                .step(slaveStep())
                .gridSize(100)
                .taskExecutor(taskExecutor())
                 .build();
    }

Task executor will be
@Bean
    public TaskExecutor taskExecutor() {
        SimpleAsyncTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor();
        return taskExecutor;
    }

partitoner will be
public Map<String, ExecutionContext> partition(int gridSize) {

      Map<String, ExecutionContext> partitionData = new HashMap<String, ExecutionContext>();  
      for (int i = 0; i < gridSize; i++) {
            ExecutionContext executionContext = new ExecutionContext();
            executionContext.putString("file", fileList.get(i)); //passing filelist
            executionContext.putString("name", "Thread" + i);
           partitionData.put("partition: " + i, executionContext);
        }

        return partitionData;
    }

并使用 stepExecutionContext 动态传递文件

and passing files dynamically using stepExecutionContext

推荐答案

如果我们使用 grid-size = 100 那么就会有 100 个不合逻辑的线程

if we use grid-size = 100 then there will be 100 threads which is not logical

网格大小和线程池大小是两个不同的东西.您可以处理 100 个分区，但只有 10 个工作线程可用.

The grid size and thread pool size are two different things. You can have 100 partitions to process but only 10 worker threads available.

您的问题是您使用的是 SimpleAsyncTaskExecutor 不重用线程(请参阅其 Javadoc).因此，对于每个分区，都会创建一个新线程，您最终会看到 100 个分区有 100 个线程.

The issue in your case is that you are using the SimpleAsyncTaskExecutor which does not re-use threads (See its Javadoc). So for each partition, a new thread will be created and you end up seeing 100 threads for the 100 partitions.

解决此问题的方法是什么，即以有限的线程数处理多个文件.

what is the way to solve this issue i.e. process many files with limited number of threads.

考虑使用 ThreadPoolTask​​Executor 这样您就可以限制工作线程的数量.

Consider using a ThreadPoolTaskExecutor so you can limit the number of worker threads.

这篇关于使用分区的 Spring 批处理多线程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！