Spring Batch Job hangs - concurrent steps and each step using multiple threads
I'm using Spring Batch to process records from database tables with the following scenario:
- Processing data from 5 tables concurrently using 5 parallel steps.
- Each parallel step has 5 threads to process records from a single table.
Summary of job configuration: TestJob -> Parallel Step1 & Step2 -> Step 1 using 2 threads, Step 2 using 2 threads.
Encountered issues with Spring Batch when selecting from BATCH_STEP_EXECUTION_SEQ or inserting into BATCH_STEP_EXECUTION table. Tried different configurations, including create isolation levels. Problem persists.
Working fine when:
- Processing concurrent tables with one thread per table.
- Processing a single table with multiple threads.
Appreciate any help or pointers to resolve this issue.
I am using Spring Batch for processing records from database tables using the below scenario:
- Processing data from 5 tables concurrently using 5 parallel steps
- Each parallel step has further 5 threads to process records from single table
Here is the summary of job configuration: TestJob -> Parallel Step1 & Step2 -> Step 1 using 2 threads, Step 2 using 2 threads
For Spring Batch tables I tried using SQL Server database, HSQL in memory database but somehow Spring batch stucks when selecting from BATCH_STEP_EXECUTION_SEQ
Spring batch trying to INSERT into BATCH_STEP_EXECUTION table so trying to get ID from BATCH_STEP_EXECUTION_SEQ table where it hangs.
I am using Spring Boot 2.2.2.RELEASE version. I tried override jobrepository configuration with different create isolation levels but problem always persists.
Everything is working as expected when:
- Processing concurrent tables at a time and each table processed by single thread
- Processing single table at a time and single table processed by multiple threads
Any help/pointer to fix the problem is highly appreciated.
Just for the sake of others who may be facing the same issue. The issue seems with the configurations of database specific tables and sequences. I tried with SQL Server, still issue persists with the default provided database scripts. Then I tried with Hsql memory database again issue persists. Then I tried using H2 memory database and it worked with that. It also works with MapJobRepositoryFactoryBean.
So you may need to tweak DDL as per database.