英文:
Spring boot pod not coming up in kubernetes
问题
我已经在AWS Kubernetes集群中部署了我的应用程序几天了，部署成功，应用程序一直在运行。最近我们更改了数据库密码。现在有4个应用程序连接到同一个数据库。所以我们重启了所有4个应用程序。3个应用程序正常启动了，但其中一个应用程序无法启动。它总是重新启动，并且我已经捕获了下面的日志：
[省略部分日志]
添加相关的Hikari池初始化日志：
[省略部分日志]
所以可以确定数据库连接不是问题。
没有异常和错误。当我检查Pod事件时，我可以看到：
事件：警告
原因：不健康
消息：活力探测失败。获取“http://....”拨号TCP:....:8080连接：连接被拒绝。
：可读性探测失败。获取“http://....”拨号TCP:....:8080连接：连接被拒绝。
：退后重启失败的容器。
这是我的活力和可读性配置：
[省略部分配置]
应用程序属性：
management.endpoints.web.exposure.include=health,info,metrics,env,flyway
management.endpoint.health.probes.enabled=true
management.health.livenessState.enabled=true
management.health.readinessState.enabled=true
此外，我已禁用了活力和可读性，但结果仍然相同。
Spring Boot版本：2.7.5
相同的配置在开发和预生产集群中应用程序正常运行，但在生产集群中失败。其他应用程序使用相同的配置运行良好。有人能帮助我确定问题可能是什么吗？
英文:
I have deployed my app in aws kubernetes cluster few days back and it has been a successful deploy and the app was up and running. Recently we have changed the db password. Now there are 4 apps which connect to the same db. So we restarted all 4 apps. 3 apps came up but one of the app is not coming up. It always restarts and I have captured the below log :
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
[nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
[nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
[alina-utility-2] org.apache.catalina.session.ManagerBase : Start expire sessions StandardManager at 1689843028211 sessioncount 0
[alina-utility-2] org.apache.catalina.session.ManagerBase : End expire sessions StandardManager processingTime 1 expired sessions: 0
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Fill pool skipped, pool is at sufficient level.
[nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
[nnection-reaper] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-1: Close connection
[nnection-reaper] com.amazonaws.internal.SdkSSLSocket : closing ssm.us-east-1.amazonaws.com/52.46.145.233:443
[nnection-reaper] h.i.c.PoolingHttpClientConnectionManager : Closing connections idle longer than 60000 MILLISECONDS
[nnection-reaper] h.i.c.DefaultManagedHttpClientConnection : http-outgoing-0: Close connection
[nnection-reaper] com.amazonaws.internal.SdkSSLSocket : closing sts.us-east-1.amazonaws.com/67.220.245.46:443
[alina-utility-2] org.apache.catalina.session.ManagerBase : Start expire sessions StandardManager at 1689843088214 sessioncount 0
[alina-utility-2] org.apache.catalina.session.ManagerBase : End expire sessions StandardManager processingTime 0 expired sessions: 0
Adding relevant hikari pool initialization log :
[ main] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@4dd90166
[ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
[ main] o.h.e.j.e.i.JdbcEnvironmentInitiator : Database ->
2023-07-20T12:15:50.405833777Z stdout F name : PostgreSQL
2023-07-20T12:15:50.405837321Z stdout F version : 13.8
2023-07-20T12:15:50.405840197Z stdout F major : 13
2023-07-20T12:15:50.405843007Z stdout F minor : 8
[ main] o.h.e.j.e.i.JdbcEnvironmentInitiator : Driver ->
2023-07-20T12:15:50.405900698Z stdout F name : PostgreSQL JDBC Driver
2023-07-20T12:15:50.405904562Z stdout F version : 42.3.7
2023-07-20T12:15:50.405907398Z stdout F major : 42
2023-07-20T12:15:50.405910368Z stdout F minor : 3
[ main] o.h.e.j.e.i.JdbcEnvironmentInitiator : JDBC version : 4.2
[l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=1, active=1, idle=0, waiting=0)
[onnection adder] org.postgresql.Driver : Connecting with URL: jdbc:postgresql://XXXXXX:5432/ics?useSSL=true
[onnection adder] org.postgresql.jdbc.PgConnection : PostgreSQL JDBC Driver 42.3.7
[onnection adder] org.postgresql.jdbc.PgConnection : setDefaultFetchSize = 0
[onnection adder] o.p.core.v3.ConnectionFactoryImpl : Trying to establish a protocol version 3 connection to XXXXXX
[onnection adder] o.p.core.v3.ConnectionFactoryImpl : Receive Buffer Size is 65,536
[onnection adder] o.p.core.v3.ConnectionFactoryImpl : Send Buffer Size is 166,400
[onnection adder] org.postgresql.ssl.MakeSSL : converting regular socket connection to ssl
[ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL10Dialect
[ main] o.h.e.j.env.spi.IdentifierHelperBuilder : JDBC driver metadata reported database stores quoted identifiers in
neither upper, lower nor mixed case
[onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@a1713f8
[onnection adder] org.postgresql.Driver : Connecting with URL: jdbc:postgresql://XXXXXX:5432/ics?useSSL=true
[onnection adder] org.postgresql.jdbc.PgConnection : PostgreSQL JDBC Driver 42.3.7
[onnection adder] org.postgresql.jdbc.PgConnection : setDefaultFetchSize = 0
[onnection adder] org.postgresql.jdbc.PgConnection : setPrepareThreshold = 5
[onnection adder] o.p.core.v3.ConnectionFactoryImpl : Trying to establish a protocol version 3 connection to XXXXXX:5432
[onnection adder] o.p.core.v3.ConnectionFactoryImpl : Receive Buffer Size is 65,536
[onnection adder] o.p.core.v3.ConnectionFactoryImpl : Send Buffer Size is 166,400
[ main] o.h.type.spi.TypeConfiguration$Scope : Scoping TypeConfiguration [org.hibernate.type.spi.TypeConfiguration@7c369270] to MetadataBuildingContext [org.hibernate.boot.internal.MetadataBuildingContextRootImpl@9f86dc3]
[ main] org.hibernate.type.BasicTypeRegistry : HHH000270: Type registration [java.util.UUID] overrides previous : org.hibernate.type.UUIDBinaryType@25f15f50
[ main] org.hibernate.type.BasicTypeRegistry : Adding type registration java.util.UUID -> org.hibernate.type.PostgresUUIDType@4ed15347
[ main] org.hibernate.type.BasicTypeRegistry : Adding type registration pg-uuid -> org.hibernate.type.PostgresUUIDType@4ed15347
[onnection adder] org.postgresql.ssl.MakeSSL : converting regular socket connection to ssl
[ main] o.h.boot.model.relational.Namespace : Created database namespace [logicalName=Name{catalog=null, schema=null}, physicalName=Name{catalog=null, schema=null}]
[onnection adder] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@4d0fcca
[onnection adder] org.postgresql.Driver : Connecting with URL: jdbc:postgresql://XXXXXX:5432/ics?useSSL=true
[onnection adder] org.postgresql.jdbc.PgConnection : PostgreSQL JDBC Driver 42.3.7
[onnection adder] org.postgresql.jdbc.PgConnection : setDefaultFetchSize = 0
[onnection adder] org.postgresql.jdbc.PgConnection : setPrepareThreshold = 5
[onnection adder] o.p.core.v3.ConnectionFactoryImpl : Trying to establish a protocol version 3 connection to XXXXXX:5432
[onnection adder] o.p.core.v3.ConnectionFactoryImpl : Receive Buffer Size is 65,536
[onnection adder] o.p.core.v3.ConnectionFactoryImpl : Send Buffer Size is 166,400
[ main] org.hibernate.cfg.AnnotationBinder : Binding entity from annotated class: x.y.z.entity.MyEntity
[onnection adder] org.postgresql.ssl.MakeSSL : converting regular socket connection to ssl
[ main] org.hibernate.cfg.Ejb3Column : Binding column: Ejb3DiscriminatorColumn{logicalColumnName'DTYPE', discriminatorTypeName='string'}
[ main] o.h.cfg.annotations.EntityBinder : Import with entity name MyEntity
[ main] o.h.cfg.annotations.EntityBinder : Bind entity x.y.z.entity.MyEntity on table my_table
So this is for sure that database connectivity is not an issue.
There is no exception and no error. When I checked the pod events I can see :
Event : Warning
Reason : Unhealthy
Message : Liveness probe failed. Get " http:// ...." dial tcp : ....:8080 connect : connection refused.
: Readiness probe failed. Get " http:// ...." dial tcp : ....:8080 connect : connection refused.
: Backoff restarting failed container.
This is my liveness-readiness configuration :
- op: add
path: /spec/template/spec/containers/0/readinessProbe/initialDelaySeconds
value: 180
- op: add
path: /spec/template/spec/containers/0/livenessProbe/initialDelaySeconds
value: 180
application properties :
management.endpoints.web.exposure.include=health,info,metrics,env,flyway
management.endpoint.health.probes.enabled=true
management.health.livenessState.enabled=true
management.health.readinessState.enabled=true
Also I have disabled the liveness and readiness but result still same.
Spring Boot version : 2.7.5
Same configuration the app is up and running in dev and preprod cluster but failing in prod cluster. Also same configuration other apps running fine. Can someone help me to identify what can be the issue here.
答案1
得分: 0
在生产环境中设置以下内容解决了我的问题。
spring.jpa.hibernate.ddl-auto=none
这阻止了DDL生成，很可能是应用程序被卡住并且不断重启的原因。
英文:
Setting the following in prod resolved the issue for me.
spring.jpa.hibernate.ddl-auto=none
This prevented the ddl generation most probably where the app was stuck and getting restarted.
答案2
得分: -1
请按照以下步骤解决问题：
从部署中移除存活性探测和就绪性探测，以缩小问题范围。检查在此更改后，Pod 是否能正常启动。如果问题得到解决，可能与您的探测定时有关。
如果前一步骤未解决问题，请以调试模式部署应用程序。然后，检查日志以查看是否能够获取 JDBC 连接。
英文:
Please follow these steps to troubleshoot the issue:
Remove the liveness and readiness probes from the deployment to narrow down the problem. Check if the pods come up after this change. If the issue is resolved, there might be a problem with the timing of your probes.
If the previous step doesn't resolve the issue, deploy the application in debug mode. Then, check the logs to see if it is able to acquire a JDBC connection.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论