“Spring Batch作业挂起 – 并发步骤,每个步骤使用多个线程”

huangapple go评论70阅读模式
英文:

Spring Batch Job hangs - concurrent steps and each step using multiple threads

问题

I'm using Spring Batch to process records from database tables with the following scenario:

  1. Processing data from 5 tables concurrently using 5 parallel steps.
  2. Each parallel step has 5 threads to process records from a single table.

Summary of job configuration: TestJob -> Parallel Step1 & Step2 -> Step 1 using 2 threads, Step 2 using 2 threads.

Encountered issues with Spring Batch when selecting from BATCH_STEP_EXECUTION_SEQ or inserting into BATCH_STEP_EXECUTION table. Tried different configurations, including create isolation levels. Problem persists.

Working fine when:

  1. Processing concurrent tables with one thread per table.
  2. Processing a single table with multiple threads.

Appreciate any help or pointers to resolve this issue.

Thanks,
Har Krishan

英文:

I am using Spring Batch for processing records from database tables using the below scenario:

  1. Processing data from 5 tables concurrently using 5 parallel steps
  2. Each parallel step has further 5 threads to process records from single table

Here is the summary of job configuration: TestJob -> Parallel Step1 & Step2 -> Step 1 using 2 threads, Step 2 using 2 threads

For Spring Batch tables I tried using SQL Server database, HSQL in memory database but somehow Spring batch stucks when selecting from BATCH_STEP_EXECUTION_SEQ
Spring batch trying to INSERT into BATCH_STEP_EXECUTION table so trying to get ID from BATCH_STEP_EXECUTION_SEQ table where it hangs.

I am using Spring Boot 2.2.2.RELEASE version. I tried override jobrepository configuration with different create isolation levels but problem always persists.

NOTE:
Everything is working as expected when:

  1. Processing concurrent tables at a time and each table processed by single thread
  2. Processing single table at a time and single table processed by multiple threads

Any help/pointer to fix the problem is highly appreciated.

Thanks,
Har Krishan

答案1

得分: 1

只为了那些可能遇到相同问题的人。问题似乎出在数据库特定表和序列的配置上。我尝试了SQL Server,但问题仍然存在,默认提供的数据库脚本仍然有问题。然后我尝试了Hsql内存数据库,问题仍然存在。然后我尝试了使用H2内存数据库,它在那种情况下可以工作。同时,MapJobRepositoryFactoryBean也可以正常工作。

因此,你可能需要根据数据库进行DDL调整。

谢谢!

英文:

Just for the sake of others who may be facing the same issue. The issue seems with the configurations of database specific tables and sequences. I tried with SQL Server, still issue persists with the default provided database scripts. Then I tried with Hsql memory database again issue persists. Then I tried using H2 memory database and it worked with that. It also works with MapJobRepositoryFactoryBean.

So you may need to tweak DDL as per database.

Thanks!

huangapple
  • 本文由 发表于 2020年7月31日 21:26:26
  • 转载请务必保留本文链接:https://go.coder-hub.com/63192769.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定