JPA starts to consume more and more memory after each iteration

Currently I try to store some news from a web api with the help of JPA.
I have 3 entities i need to store: Webpage, NewsPost and the Query that returned the news post. I have one table for each of the three. My simpliefied JPA entities looking like the following ones:

@Entity
@Data
@Table(name = "NewsPosts", schema = "data")
@EqualsAndHashCode
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class NewsPost {

	@Id
	@Column(name = "id")
	private long id;
	@Basic
	@Column(name = "subject")
	private String subject;
	@Basic
	@Column(name = "post_text")
	private String postText;

	@ManyToOne(fetch = FetchType.LAZY, cascade = CascadeType.MERGE)
	@JoinColumn(name = "newsSite")
	private NewsSite site;

	@ManyToMany(fetch = FetchType.EAGER, cascade = CascadeType.MERGE)
	@JoinTable(name = "query_news_post", joinColumns = @JoinColumn(name = "newsid"), inverseJoinColumns = @JoinColumn(name = "queryid"))
	private Set<QueryEntity> queries;
}


@Entity
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@Table(name = "queries", schema = "data")
@EqualsAndHashCode
public class QueryEntity {

	@Id
	@GeneratedValue(strategy = GenerationType.IDENTITY)
	@Column(name = "id")
	private int id;
	@EqualsAndHashCode.Exclude
	@Basic
	@Column(name = "query")
	private String query;

	// needs to be exclueded otherwise we can create stack overflow, because of circular references...
	@EqualsAndHashCode.Exclude
	@ToString.Exclude
	@ManyToMany(mappedBy = "queries", fetch = FetchType.LAZY, cascade = CascadeType.MERGE)
	Set<PostsEntity> posts;

}



@Entity
@Data
@Table(name = "sites", schema = "data")
@EqualsAndHashCode
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class newsSite {
	@Id
	@Column(name = "SiteId")
	private long id;
	@Basic
	@Column(name = "SiteName")
	private String site;

}

Currently I'm doing the following: I create the query and retrieve the of the query. Then i start crawling:
I get the objects from the web api back in paginated fashion with pagesize of 100 newsPosts i use an object mapper to map the json response to my entity classes.

Afterwards i tried two different thins:

1. I added the query ID as Set to the NewsPost and wrote it back to DB with the EntityManager's merge option. This works quite well until i came to the point, where I got a NewsPost again for another query, then the new query is overwritten by the old one. To solve this it tried 2.
2. I check if the NewsPost already exists if it does i retrieved the post added the new query to the existing one and merged it back to the database as i did before. When doing this i works quite well and i get the expected result for the first batches but then suddenly the application start to consume more and more memory for the thrid batch. I attachted a screenshot from JavaVisualVM. Has somebody an idea why this happens?

Edit:
As some Questions were raised in the comments i would like to provide the answers to the questions here.

I think with the crawling everything works fine. The return of the Webapi comes as json. I'm using jackson mapper to map this to a POJO and afterwards I'm using the Dozer mapper to convert to POJO to the Entity. (Yes i need the step to POJO first for other purposes in the application this is workin fine).

Regarding the writing with the EntityManager I'm not sure if I'm doing that correctly.

At first i created a JPA repo for checking if the post already exists (to get the old query ids and avoid the issue with the overwriting in the queryid, postid table). My JPA repo looks as follows.

@Repository
public interface PostRepo extends JpaRepository<NewsPost, Long> {

	NewsPost getById(long id);
}

To update the posts I'm doing this as follows:

private void updatePosts(List<NewsPost> posts){
	posts.forEach(post->{
				NewsPost foundPost = postRepo.getById(post.getId());
				if(foundPost!=null){
					post.getQueries().addAll(foundPost.getQueries());	
				}});
}

I'm currently writing my entities as follows i have a list of entities the contains also the updated posts and i have an autowired EntityManagerFactory in my class that handles the writing.

EntityManager em = entityManagerFactory.createEntityManager();
		try {
			EntityTransaction transaction = em.getTransaction();
			transaction.begin();
            entities.forEach(entity->em.merge(entity))
            em.flush();
			transaction.commit();
		} finally {
			em.clear();
			em.close();
		}

I'm pretty sure that it is the writing process. If i keep the logic of my software the same but only skip the merge or just printing or dumping the entities to a file everything works and fast and no error appears so it seems to be an issue with the merge comment?

Regarding the question if my program dies because of the memory consumption it depends. If I run it on my mac is consumes up to 8+ gigabytes of ram but MAC OS is handling this and swaps the ram to disk. If I run it as a docker container von CentOS the process is killed due to to less memory.

Don't now if this is relevant, but I'm using OpenJDK 11, Springboot 2.2.6, and a MYSQL 8 Database.

I configured jpa as follows in my application.yml:

spring:
  main:
    allow-bean-definition-overriding: true
  datasource:
    url: "jdbc:mysql://db"
    username: user
    password: secret
    driver-class-name: com.mysql.cj.jdbc.Driver
    test-while-idle: true
    validation-query: Select 1
  jpa:
    database-platform: org.hibernate.dialect.MySQL8Dialect
    hibernate:
      ddl-auto: none
    properties:
      hibernate:
        event:
          merge:
            entity_copy_observer: allow
    ```

</details>


# 答案1
**得分**: 1

如果合并过程是问题所在，为了在 `entityManager` 中保持内存消耗低的快速修复方法可能是在每次合并后添加 `em.flush();` 和 `em.clear();`：

```java
EntityTransaction transaction = em.getTransaction();
transaction.begin();
entities.forEach(entity -> {
    em.merge(entity);
    em.flush();
    em.clear();
});
transaction.commit();

然而，我认为你应该改变你的模型。为了仅添加新的关联而加载每篇帖子的所有现有查询是非常低效的。你可以将 N-M 关系建模为一个新实体，然后仅持久化新关系。

EntityTransaction transaction = em.getTransaction();
transaction.begin();
entities.forEach(entity-> {
    em.merge(entity);
    em.flush();
    em.clear();
});
transaction.commit();

答案2

得分: 1

在尝试了一些方法后，我自己解决了这个问题。我为多对多关系创建了一个实体。然后，我为每个实体创建了CRUD存储库，并使用了CRUD存储库中的saveAll方法。这在内存方面也运行良好。现在，垃圾收集器在内存可视化中产生了预期的链锯模式。但是，我仍然不明白为什么我之前使用注释中的联接表创建的多对多关系会在内存管理方面引发问题。是否有人可以解释一下为什么这解决了我的问题？是因为多对多关系创建了循环依赖吗？但据我所知，垃圾收集器也能找到循环依赖。

答案3

得分: 0

一个多对多关系中的 EAGER 关系会获取许多对象。关于 LAZY 关系，请确保获取它们，因为如果不这样做，遍历完整对象以将其转换为 JSON 或 POJO 将为每个未使用 fetch 初始化的对象抛出查询，这可能是一种危险操作。如果不需要全部对象，可以使用 @JsonIgnore 注解。