英文:
Spring Batch Job hangs - concurrent steps and each step using multiple threads
问题
I'm using Spring Batch to process records from database tables with the following scenario:
- Processing data from 5 tables concurrently using 5 parallel steps.
- Each parallel step has 5 threads to process records from a single table.
Summary of job configuration: TestJob -> Parallel Step1 & Step2 -> Step 1 using 2 threads, Step 2 using 2 threads.
Encountered issues with Spring Batch when selecting from BATCH_STEP_EXECUTION_SEQ or inserting into BATCH_STEP_EXECUTION table. Tried different configurations, including create isolation levels. Problem persists.
Working fine when:
- Processing concurrent tables with one thread per table.
- Processing a single table with multiple threads.
Appreciate any help or pointers to resolve this issue.
Thanks,
Har Krishan
英文:
I am using Spring Batch for processing records from database tables using the below scenario:
- Processing data from 5 tables concurrently using 5 parallel steps
- Each parallel step has further 5 threads to process records from single table
Here is the summary of job configuration: TestJob -> Parallel Step1 & Step2 -> Step 1 using 2 threads, Step 2 using 2 threads
For Spring Batch tables I tried using SQL Server database, HSQL in memory database but somehow Spring batch stucks when selecting from BATCH_STEP_EXECUTION_SEQ
Spring batch trying to INSERT into BATCH_STEP_EXECUTION table so trying to get ID from BATCH_STEP_EXECUTION_SEQ table where it hangs.
I am using Spring Boot 2.2.2.RELEASE version. I tried override jobrepository configuration with different create isolation levels but problem always persists.
NOTE:
Everything is working as expected when:
- Processing concurrent tables at a time and each table processed by single thread
- Processing single table at a time and single table processed by multiple threads
Any help/pointer to fix the problem is highly appreciated.
Thanks,
Har Krishan
答案1
得分: 1
只为了那些可能遇到相同问题的人。问题似乎出在数据库特定表和序列的配置上。我尝试了SQL Server,但问题仍然存在,默认提供的数据库脚本仍然有问题。然后我尝试了Hsql内存数据库,问题仍然存在。然后我尝试了使用H2内存数据库,它在那种情况下可以工作。同时,MapJobRepositoryFactoryBean也可以正常工作。
因此,你可能需要根据数据库进行DDL调整。
谢谢!
英文:
Just for the sake of others who may be facing the same issue. The issue seems with the configurations of database specific tables and sequences. I tried with SQL Server, still issue persists with the default provided database scripts. Then I tried with Hsql memory database again issue persists. Then I tried using H2 memory database and it worked with that. It also works with MapJobRepositoryFactoryBean.
So you may need to tweak DDL as per database.
Thanks!
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论