Spring Boot 在 Kubernetes 中的 Pod 未启动。

huangapple
go评论96阅读模式
英文:

Spring boot pod not coming up in kubernetes

问题

我已经在AWS Kubernetes集群中部署了我的应用程序几天了，部署成功，应用程序一直在运行。最近我们更改了数据库密码。现在有4个应用程序连接到同一个数据库。所以我们重启了所有4个应用程序。3个应用程序正常启动了，但其中一个应用程序无法启动。它总是重新启动，并且我已经捕获了下面的日志：

[省略部分日志]

添加相关的Hikari池初始化日志：

[省略部分日志]

所以可以确定数据库连接不是问题。

没有异常和错误。当我检查Pod事件时，我可以看到：

事件：警告
原因：不健康
消息：活力探测失败。获取“http://....”拨号TCP:....:8080连接：连接被拒绝。
：可读性探测失败。获取“http://....”拨号TCP:....:8080连接：连接被拒绝。
：退后重启失败的容器。

这是我的活力和可读性配置：

[省略部分配置]

应用程序属性：

management.endpoints.web.exposure.include=health,info,metrics,env,flyway
management.endpoint.health.probes.enabled=true
management.health.livenessState.enabled=true
management.health.readinessState.enabled=true

此外，我已禁用了活力和可读性，但结果仍然相同。

Spring Boot版本：2.7.5

相同的配置在开发和预生产集群中应用程序正常运行，但在生产集群中失败。其他应用程序使用相同的配置运行良好。有人能帮助我确定问题可能是什么吗？

英文:

I have deployed my app in aws kubernetes cluster few days back and it has been a successful deploy and the app was up and running. Recently we have changed the db password. Now there are 4 apps which connect to the same db. So we restarted all 4 apps. 3 apps came up but one of the app is not coming up. It always restarts and I have captured the below log :

  1. [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
  2. [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
  3. [nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
  4. [nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
  5. [alina-utility-2] org.apache.catalina.session.ManagerBase  : Start expire sessions StandardManager at 1689843028211 sessioncount 0
  6. [alina-utility-2] org.apache.catalina.session.ManagerBase  : End expire sessions StandardManager processingTime 1 expired sessions: 0
  7. [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
  8. [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
  9. [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
  10. [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
  11. [nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
  12. [nnection-reaper] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-1: Close connection
  13. [nnection-reaper] com.amazonaws.internal.SdkSSLSocket      : closing ssm.us-east-1.amazonaws.com/52.46.145.233:443
  14. [nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
  15. [nnection-reaper] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-0: Close connection
  16. [nnection-reaper] com.amazonaws.internal.SdkSSLSocket      : closing sts.us-east-1.amazonaws.com/67.220.245.46:443
  17. [alina-utility-2] org.apache.catalina.session.ManagerBase  : Start expire sessions StandardManager at 1689843088214 sessioncount 0
  18. [alina-utility-2] org.apache.catalina.session.ManagerBase  : End expire sessions StandardManager processingTime 0 expired sessions: 0

Adding relevant hikari pool initialization log :

  1. [           main] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@4dd90166
  2. [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
  3. [           main] o.h.e.j.e.i.JdbcEnvironmentInitiator     : Database ->
  4. 2023-07-20T12:15:50.405833777Z stdout F        name : PostgreSQL
  5. 2023-07-20T12:15:50.405837321Z stdout F     version : 13.8
  6. 2023-07-20T12:15:50.405840197Z stdout F       major : 13
  7. 2023-07-20T12:15:50.405843007Z stdout F       minor : 8
  8. [           main] o.h.e.j.e.i.JdbcEnvironmentInitiator     : Driver ->
  9. 2023-07-20T12:15:50.405900698Z stdout F        name : PostgreSQL JDBC Driver
  10. 2023-07-20T12:15:50.405904562Z stdout F     version : 42.3.7
  11. 2023-07-20T12:15:50.405907398Z stdout F       major : 42
  12. 2023-07-20T12:15:50.405910368Z stdout F       minor : 3
  13. [           main] o.h.e.j.e.i.JdbcEnvironmentInitiator     : JDBC version : 4.2
  14. [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=1, active=1, idle=0, waiting=0)
  15. [onnection adder] org.postgresql.Driver                    : Connecting with URL: jdbc:postgresql://XXXXXX:5432/ics?useSSL=true
  16. [onnection adder] org.postgresql.jdbc.PgConnection         : PostgreSQL JDBC Driver 42.3.7
  17. [onnection adder] org.postgresql.jdbc.PgConnection         :   setDefaultFetchSize = 0
  19. [onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Trying to establish a protocol version 3 connection to XXXXXX
  20. [onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Receive Buffer Size is 65,536
  21. [onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Send Buffer Size is 166,400
  22. [onnection adder] org.postgresql.ssl.MakeSSL               : converting regular socket connection to ssl
  23. [           main] org.hibernate.dialect.Dialect            : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL10Dialect
  24. [           main] o.h.e.j.env.spi.IdentifierHelperBuilder  : JDBC driver metadata reported database stores quoted identifiers in 
  25.                                                               neither upper, lower nor mixed case
  26. [onnection adder] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@a1713f8
  27. [onnection adder] org.postgresql.Driver                    : Connecting with URL: jdbc:postgresql://XXXXXX:5432/ics?useSSL=true
  28. [onnection adder] org.postgresql.jdbc.PgConnection         : PostgreSQL JDBC Driver 42.3.7
  29. [onnection adder] org.postgresql.jdbc.PgConnection         :   setDefaultFetchSize = 0
  30. [onnection adder] org.postgresql.jdbc.PgConnection         :   setPrepareThreshold = 5
  31. [onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Trying to establish a protocol version 3 connection to XXXXXX:5432
  32. [onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Receive Buffer Size is 65,536
  33. [onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Send Buffer Size is 166,400
  35. [           main] o.h.type.spi.TypeConfiguration$Scope     : Scoping TypeConfiguration [org.hibernate.type.spi.TypeConfiguration@7c369270] to MetadataBuildingContext [org.hibernate.boot.internal.MetadataBuildingContextRootImpl@9f86dc3]
  36. [           main] org.hibernate.type.BasicTypeRegistry     : HHH000270: Type registration [java.util.UUID] overrides previous : org.hibernate.type.UUIDBinaryType@25f15f50
  37. [           main] org.hibernate.type.BasicTypeRegistry     : Adding type registration java.util.UUID -> org.hibernate.type.PostgresUUIDType@4ed15347
  38. [           main] org.hibernate.type.BasicTypeRegistry     : Adding type registration pg-uuid -> org.hibernate.type.PostgresUUIDType@4ed15347
  42. [onnection adder] org.postgresql.ssl.MakeSSL               : converting regular socket connection to ssl
  44. [           main] o.h.boot.model.relational.Namespace      : Created database namespace [logicalName=Name{catalog=null, schema=null}, physicalName=Name{catalog=null, schema=null}]
  45. [onnection adder] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@4d0fcca
  46. [onnection adder] org.postgresql.Driver                    : Connecting with URL: jdbc:postgresql://XXXXXX:5432/ics?useSSL=true
  47. [onnection adder] org.postgresql.jdbc.PgConnection         : PostgreSQL JDBC Driver 42.3.7
  48. [onnection adder] org.postgresql.jdbc.PgConnection         :   setDefaultFetchSize = 0
  49. [onnection adder] org.postgresql.jdbc.PgConnection         :   setPrepareThreshold = 5
  50. [onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Trying to establish a protocol version 3 connection to XXXXXX:5432
  51. [onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Receive Buffer Size is 65,536
  52. [onnection adder] o.p.core.v3.ConnectionFactoryImpl        : Send Buffer Size is 166,400
  54. [           main] org.hibernate.cfg.AnnotationBinder       : Binding entity from annotated class: x.y.z.entity.MyEntity
  55. [onnection adder] org.postgresql.ssl.MakeSSL               : converting regular socket connection to ssl
  56. [           main] org.hibernate.cfg.Ejb3Column             : Binding column: Ejb3DiscriminatorColumn{logicalColumnName'DTYPE', discriminatorTypeName='string'}
  57. [           main] o.h.cfg.annotations.EntityBinder         : Import with entity name MyEntity
  58. [           main] o.h.cfg.annotations.EntityBinder         : Bind entity x.y.z.entity.MyEntity on table my_table

So this is for sure that database connectivity is not an issue.

There is no exception and no error. When I checked the pod events I can see :

  1. Event : Warning
  2. Reason : Unhealthy
  3. Message : Liveness probe failed. Get " http:// ...." dial tcp : ....:8080 connect : connection refused.
  4.         : Readiness probe failed. Get " http:// ...." dial tcp : ....:8080 connect : connection refused.
  5.         : Backoff restarting failed container.

This is my liveness-readiness configuration :

  1.   - op: add
  2.     path: /spec/template/spec/containers/0/readinessProbe/initialDelaySeconds
  3.     value: 180
  4.   - op: add
  5.     path: /spec/template/spec/containers/0/livenessProbe/initialDelaySeconds
  6.     value: 180

application properties :

  1. management.endpoints.web.exposure.include=health,info,metrics,env,flyway
  2. management.endpoint.health.probes.enabled=true
  3. management.health.livenessState.enabled=true
  4. management.health.readinessState.enabled=true

Also I have disabled the liveness and readiness but result still same.

Spring Boot version : 2.7.5

Same configuration the app is up and running in dev and preprod cluster but failing in prod cluster. Also same configuration other apps running fine. Can someone help me to identify what can be the issue here.

答案1

得分: 0

在生产环境中设置以下内容解决了我的问题。

  1. spring.jpa.hibernate.ddl-auto=none

这阻止了DDL生成，很可能是应用程序被卡住并且不断重启的原因。

英文:

Setting the following in prod resolved the issue for me.

  1. spring.jpa.hibernate.ddl-auto=none

This prevented the ddl generation most probably where the app was stuck and getting restarted.

答案2

得分: -1

请按照以下步骤解决问题：

从部署中移除存活性探测和就绪性探测，以缩小问题范围。检查在此更改后，Pod 是否能正常启动。如果问题得到解决，可能与您的探测定时有关。

如果前一步骤未解决问题，请以调试模式部署应用程序。然后，检查日志以查看是否能够获取 JDBC 连接。

英文:

Please follow these steps to troubleshoot the issue:

Remove the liveness and readiness probes from the deployment to narrow down the problem. Check if the pods come up after this change. If the issue is resolved, there might be a problem with the timing of your probes.

If the previous step doesn't resolve the issue, deploy the application in debug mode. Then, check the logs to see if it is able to acquire a JDBC connection.

huangapple
  • 本文由 发表于 2023年7月20日 20:05:12
  • 转载请务必保留本文链接：https://go.coder-hub.com/76729669.html
  • amazon-web-services
  • kubernetes
  • spring-boot
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定