为什么Spring Batch为每个线程使用1个数据库连接? [英] Why does Spring Batch use 1 database connection for each thread?

问题描述

为什么Spring Batch为每个线程使用1个数据库连接?

Why does Spring Batch use 1 database connection for each thread?

堆栈:

• Java 8
• Spring Boot 1.5
• 春季批次3.0.7
• HikariCP 2.7.6

数据源配置:

• batcdb(postgres)
• readdb(oracle)
• writedb(postgres)

每个数据源都使用HikariCP，每个默认10个连接.

Each datasource is using HikariCP with default 10 connections each.

Spring Batch配置: ThreadExecutor-1:

Spring Batch config: ThreadExecutor-1:

core-pool-size: 10
max-pool-size: 10
throttle-limit: 10

Job-1 Config/ThreadPoolTask​​Executor: (通过application.yml设置的油门尺寸和油门限制)

Job-1 Config / ThreadPoolTaskExecutor: (pool sizes and throttle limit set via application.yml)

@Bean
public Step job1Step() {
    return stepBuilderFactory.get("job1Step")
            .<ReadModel, WriteModel>chunk(chunkSize)
            .reader(itemReader())
            .processor(compositeProcessor())
            .writer(itemWriter())
            .faultTolerant()
            .taskExecutor(job1TaskExecutor())
            .throttleLimit(throttleLimit)
            .build();
}

@Bean
public ThreadPoolTaskExecutor job1TaskExecutor() {
     ThreadPoolTaskExecutor pool = new ThreadPoolTaskExecutor();
     pool.setCorePoolSize(poolSize);
     pool.setMaxPoolSize(maxPoolSize);
     pool.setWaitForTasksToCompleteOnShutdown(false);
     return pool;
 }

@Bean
@StepScope
public Job1ItemReader job1ItemReader() {
    return new Job1ItemReader(readdb, pageSize);
}

Job1-ItemReader的缩写代码

Abbreviated Code for Job1-ItemReader

public class Job1ItemReader extends JdbcPagingItemReader<ReadModel> {
...
}

ThreadExecutor-2:

ThreadExecutor-2:

core-pool-size: 5
max-pool-size: 5
throttle-limit: 5

Job-2 Config/ThreadPoolTask​​Executor:

Job-2 Config / ThreadPoolTaskExecutor:

@Bean
public Step job2Step() throws Exception {
    return stepBuilderFactory.get("job2Step")
            .<ReadModel2, WriteModel2>chunk(chunkSize)
            .reader(job2ItemReader())
            .processor(job2CompositeProcessor())
            .writer(job2ItemWriter())
            .faultTolerant()
            .taskExecutor(job2TaskExecutor())
            .throttleLimit(throttleLimit)
            .build();
}

@Bean
public ThreadPoolTaskExecutor job2TaskExecutor() {
    ThreadPoolTaskExecutor pool = new ThreadPoolTaskExecutor();
    pool.setCorePoolSize(corePoolSize);
    pool.setMaxPoolSize(maxPoolSize);
    pool.setQueueCapacity(queueCapacity);
    pool.setWaitForTasksToCompleteOnShutdown(false);
    return pool;
}

@Bean
@StepScope
public Job2ItemReader job2ItemReader() {
    return new Job2ItemReader(readdb, pageSize);    
}

Job2-ItemReader的缩写代码

Abbreviated Code for Job2-ItemReader

public class Job2ItemReader extends JdbcPagingItemReader<ReadModel2> {
...
}

• 有2个工作
• 工作1持续时间很长(几天)
• 工作2通常在一个小时或两小时内完成，并且每天按计划运行
• 作业位于相同的应用程序"中，并在相同的JVM上运行
• 每个作业都有自己定义的ThreadPoolTask​​Executor
• There are 2 jobs
• Job-1 is long-running (multiple days)
• Job-2 usually completes in a hour or 2, and runs on a schedule every day
• Jobs are in the same 'application', running on the same JVM
• Each Job has its own ThreadPoolTaskExecutor defined

运行Job-1并启动Job-2时，Job-2无法获得与readdb的连接. Job-2的批处理读取器将引发以下错误.

When Job-1 is running, and Job-2 starts, Job-2 is unable to get a connection to the readdb. The following error is thrown by the Batch Reader of Job-2.

Caused by: org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-3 - Connection is not available, request timed out after 30000ms.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:339)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:366)
at org.springframework.batch.support.DatabaseType.fromMetaData(DatabaseType.java:97)
at org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean.getObject(SqlPagingQueryProviderFactoryBean.java:158)
... 30 common frames omitted
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-3 - Connection is not available, request timed out after 30000ms.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:326)
... 33 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-3 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:666)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:182)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:123)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)

(为保护无辜者而编辑)

(edited to protect the innocent)

参考:

• 多线程步骤
• 类似问题/问题，但我的交易应该在writedb上，在这种情况下，HikariPool-3是readdb.
• multi-threaded step
• Similar question/issue, except my transaction should be on the writedb, where in this case the HikariPool-3 is the readdb.

推荐答案

Spring Batch每个线程使用一个数据库连接的原因(在某些情况下实际上可以使用更多)是由于事务引起的.春季交易被绑定到一个线程. Spring Batch中的几乎所有事情都在事务中发生.因此，当您有一个带有单个线程的单一作业时，最多只能使用几个连接.但是，如果您有一个多线程步骤，则每个线程至少要有一个连接来进行事务处理.

The reason Spring Batch uses one database connection per thread (it can actually use more in certain situations), is due to transactions. Spring transactions are tied to a thread. Just about everything within Spring Batch happens within a transaction. So when you have a single job with a single thread, you'll use only a couple connections at most. However, if you have a multithreaded step, expect at least one connection per thread for the transaction handling.

这篇关于为什么Spring Batch为每个线程使用1个数据库连接?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！