英文:
how to improve the performance of bulk transaction in spring mvc
问题
我正在使用Java Spring MVC与JPA以及Hibernate。如果我想要处理2000个事务，可以将它们分成每批500个事务进行处理，使用线程，并考虑到每个单独的事务应该由同一个线程处理吗？还有其他的处理方式吗？
英文:
i am working with java spring mvc with JPA and hibernate. If i want to process 2000 transactions, can i process them in batches of 500 transactions using threads putting in consideration that the single transaction should be processed by the same thread?is there any other way to do this?
答案1
得分: 0
Hibernate默认情况下不启用批处理。这意味着它将为每个插入操作发送单独的SQL语句。
为了一次性插入大量数据集，你应该使用批量插入。以下是Hibernate的属性设置：
hibernate.jdbc.batch_size = 50 和 hibernate.order_inserts = true
第一个属性告诉Hibernate将插入操作按批次收集，每批次包含50条（这里的50只是示例）。order_inserts属性告诉Hibernate花费时间将插入操作按实体进行分组，以创建更大的批次。
然后你应该在显式刷新（explicit flushing）和隐式刷新（implicit flushing）之间做出选择。
- 隐式刷新：只需设置批处理大小属性，Hibernate将完成其余操作。
- 显式刷新
flush和clear
for (int i = 0; i < 10; i++) {
if (i > 0 && i % BATCH_SIZE == 0) {
entityManager.flush();
entityManager.clear();
}
A a = new A();
entityManager.persist(a);
}
那么为什么选择第二个选项呢？
当我们持久化一个实体时，Hibernate将其存储在持久化上下文中。例如，如果我们在一个事务中持久化了2000个实体，最终会在内存中拥有2000个实体实例，可能会导致内存溢出...
另一个问题... 正如你所说，你正在使用MySQL。
Hibernate的批处理选项与IDENTITY标识生成器不兼容，因此你无法在MySQL的AUTO-INCREMENT特性下使用批量插入。由于MySQL中不存在序列生成，你只能使用TABLE生成器。但是与标识生成器相比，表生成器的性能较低。
引用Vlad Mihalcea（摘自他的著名书籍《High Performance Java Persistence》）：
在使用MySQL并且需要大量插入操作时，使用JOOQ框架是一个不错的选择。
英文:
Hibernate doesn't enable batching by default. This means that it'll send a separate SQL statement for each insert.
To insert so massive dataset in once you should use batch insert.
Here the hibernate properties :
hibernate.jdbc.batch_size = 50 and hibernate.order_inserts = true
The first property tells Hibernate to collect inserts in batches of 50 (50 is only used here for as example..). The order_inserts property tells Hibernate to take the time to group inserts by entity, creating larger batches.
Then you should choose between explicit or implicit flushing.
- Implicit: just set the batch size property and hibernate will do the rest.
- Explicit
flushandclear
for (int i = 0; i < 10; i++) {
if (i > 0 && i % BATCH_SIZE == 0) {
entityManager.flush();
entityManager.clear();
}
A a = new A();
entityManager.persist(a);
}
So why using 2nd option?
When we persist an entity, Hibernate stores it in the persistence context. For example, if we persist 2,000 entities in one transaction, we'll end up having 2,000 entity instances in memory, possibly causing an OutOfMemory
...
Other problem... as you said you're using MySQL.
hibernate batch option not work with IdGenerator IDENTITY so you can't use batch insert with AUTO-INCREMENT feature of mysql. As sequence generation doesn't exist in MySQL you're stuck with TABLE generator. But table generator is less performant than identity one.
To mention Vlad mihalcea (quote from his famous book High Performance Java Persistence)
> When using MySQL and need lot of inserts It is a good idea to use JOOQ framework for that.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论