Consul-API-Gateway负载均衡器在AWS EKS环境中始终显示注册的目标不健康。

huangapple
go评论95阅读模式
英文:

consul-api-gateway load balancer always shows registered target un healthy in aws eks environment

问题

我正在尝试在AWS EKS集群中为Keycloak服务创建一个Consul API网关（Gateway）资源，使用HttpRoute进行配置。我以一种使API网关和Keycloak服务运行在不同的命名空间的方式进行配置。但是，当使用下面的部署YAML创建Consul API网关资源时，它会创建一个AWS负载均衡器，其中注册的目标显示为不健康。不确定是什么原因导致它始终不健康。

请注意，我只翻译了您的描述，不包括代码部分。

英文:

I am trying to create a consul-api-gateway (Gateway) resource with HttpRoute for keycloak service in aws eks cluster. I am configuring in a way that the api-gateway and keycloak service are running in different namespaces. But when consul-api-gateway resource created with below deployment yaml, it is creating aws load balancer with the registered target showing as unhealthy. Not sure what makes this as unhealthy always.

Consul-server-and-ui-deployment.yaml

  1. ---
  2. global:
  3.   name: consul  
  4.   imagePullSecrets:
  5.     - name: "***"
  6.   image: /hashicorp/consul:1.12.0
  7.   imageEnvoy: /envoyproxy/envoy:v1.22.2
  8.   imageK8S: /hashicorp/consul-k8s-control-plane:0.44.0
  9.   metrics:
  10.     enabled: true
  11. server:
  12.   replicas: 3
  13.   bootstrapExpect: 3
  14.   extraLabels:
  15.     tags.datadoghq.com/source: consul
  16.     tags.datadoghq.com/service: consul-server
  17.   annotations: |
  18.     "tags.datadoghq.com/source":                              "consul"
  19.     "tags.datadoghq.com/service":                             "consul-server"
  20.   updatePartitions: 3
  21.   disruptionBudget:
  22.     maxUnavailable: 1
  23.   tolerations: >
  24.     -
  25.     key:                                                    "taint_for_consul_xor_vault"
  26.       operator:                                               "Equal"
  27.       value:                                                  "true"
  28.       effect:                                                 "NoSchedule"
  29.   nodeSelector: |
  30.     'eks.amazonaws.com/capacityType':                         'ON_DEMAND'
  31.     'purpose':                                                'consul-server'
  32.   storage: 50G
  33.   priorityClassName: high-priority
  34. client:
  35.   enabled: true
  36.   grpc: true
  37.   exposeGossipPorts: false
  38.   priorityClassName: high-priority
  39.   resources:
  40.     requests:
  41.       memory: 1G
  42.       cpu: "1"
  43.     limits:
  44.       memory: 1G
  45.       cpu: "1"
  46.   healthChecks:
  47.     enabled: true
  48.   hostNetwork: false
  49.   extraConfig: |
  50.     {
  51.       "advertise_reconnect_timeout":                          "15m",
  52.       "limits":                                               {
  53.         "http_max_conns_per_client":                          -1
  54.       }
  55.     }
  56.   extraLabels:
  57.     tags.datadoghq.com/source: consul
  58.     tags.datadoghq.com/service: consul-client
  59.   annotations: |
  60.     "tags.datadoghq.com/source":                              "consul"
  61.     "tags.datadoghq.com/service":                             "consul-client"
  62.   updateStrategy: |
  63.     type:                                                     RollingUpdate
  64.     rollingUpdate:
  65.       maxUnavailable:                                         1
  66. ui:
  67.   enabled: true
  68.   service:
  69.     type: LoadBalancer
  70.     additionalSpec: "'ports': [{'name': 'http', 'protocol': 'TCP', 'port': 8500,
  71.       'targetPort': 8500}, {'name': 'https', 'protocol': 'TCP', 'port': 8501,
  72.       'targetPort': 8501}]"
  73.     annotations: |
  74.       'service.beta.kubernetes.io/aws-load-balancer-scheme':  'internal'
  75.       'service.beta.kubernetes.io/aws-load-balancer-type':    'nlb-ip'
  76. controller:
  77.   enabled: true
  78. prometheus:
  79.   enabled: true
  80. grafana:
  81.   enabled: true
  82. terminatingGateways:
  83.   enabled: true
  84.   priorityClassName: high-priority
  85.   defaults:
  86.     replicas: 2
  87. apiGateway:
  88.   enabled: true
  89.   image: /hashicorp/consul-api-gateway:0.3.0
  90.   controller:
  91.     replicas: 2
  92.     priorityClassName: high-priority
  93. connectInject:
  94.   enabled: true
  95.   priorityClassName: high-priority
  96.   transparentProxy:
  97.     defaultEnabled: true
  98.   default: false
  99. syncCatalog:
  100.   enabled: true
  101.   priorityClassName: high-priority
  102.   default: false
  103.   toConsul: true
  104.   toK8S: false

Consul-api-gateway-deployment.yaml

  1. apiVersion:         api-gateway.consul.hashicorp.com/v1alpha1
  2. kind:               GatewayClassConfig
  3. metadata:
  4.   name:             consul-common-gateway-class-config
  5. spec:
  6.   logLevel:         'info'  
  7.   copyAnnotations:
  8.     service:
  9.       - service.beta.kubernetes.io/aws-load-balancer-scheme
  10.       - service.beta.kubernetes.io/aws-load-balancer-type
  11.   consul:
  12.     scheme:         'http'
  13.     ports:
  14.       http:         8500
  15.       grpc:         8502
  16.   serviceType:      LoadBalancer
  17. ---
  18. apiVersion:         gateway.networking.k8s.io/v1alpha2
  19. kind:               GatewayClass
  20. metadata:
  21.   name:             consul-common-gateway-class
  22. spec:
  23.   controllerName:   'hashicorp.com/consul-api-gateway-controller'
  24.   parametersRef:
  25.     group:          api-gateway.consul.hashicorp.com
  26.     kind:           GatewayClassConfig
  27.     name:           consul-common-gateway-class-config
  28. ---
  29. apiVersion:                                                 gateway.networking.k8s.io/v1alpha2
  30. kind:                                                       Gateway
  31. metadata:
  32.   name:                                                     common-api-gateway
  33.   annotations:
  34.      'service.beta.kubernetes.io/aws-load-balancer-scheme':  'internal'
  35.      'service.beta.kubernetes.io/aws-load-balancer-type':    'nlb-ip'           
  36. spec:
  37.   gatewayClassName:                                         consul-common-gateway-class
  38.   listeners:  
  39.   - protocol:                                               HTTP
  40.     port:                                                   80
  41.     name:                                                   http
  42.     allowedRoutes:
  43.       namespaces:
  44.         from:                                               All

HttpRoute-keycloak-deployment.yaml

  1. ---
  2. apiVersion:       gateway.networking.k8s.io/v1alpha2
  3. kind:             HTTPRoute
  4. metadata:
  5.   name:           gateway-keycloak-route
  6.   namespace:      app-ns
  7. spec:
  8.   parentRefs:
  9.   - name:         common-api-gateway
  10.     namespace:    consul
  11.   rules:
  12.   - matches:
  13.     - path:
  14.         type:     PathPrefix
  15.         value:    /auth
  16.     backendRefs:
  17.     - kind:       Service
  18.       name:       keycloak
  19.       port:       80
  20.       namespace:  app-ns
  21. ---
  22. apiVersion:       gateway.networking.k8s.io/v1alpha2
  23. kind:             ReferencePolicy
  24. metadata:
  25.   name:           reference-policy-keycloak
  26. spec:
  27.   from:
  28.     - group:      gateway.networking.k8s.io
  29.       kind:       HTTPRoute
  30.       namespace:  app-ns
  31.   to:
  32.     - group:      ""
  33.       kind:       Service
  34.       name:       keycloak

I wanted api-gateway load balancer running in aws eks to be with healthy registered targets and wanted to run Keycloak service have proper routes to api-gateway

答案1

得分: 0

I've had a very similar issue today, and spend hours trying to figure it out.
我今天遇到了一个非常类似的问题，花了很多时间来解决它。

I also had a very similar setup with a Consul API Gateway behind an internal AWS load balancer.
我也使用了与您类似的设置，将Consul API Gateway放在内部的AWS负载均衡器后面。

What solved it for me, was changing the load balancer type from ip to instance.
对我来说，解决问题的方法是将负载均衡器类型从IP更改为实例。

In my case , I was using the AWS Load balancer controller, which seems to be the recommended way to go.
在我的情况下，我正在使用AWS负载均衡器控制器，这似乎是推荐的方式。

So, I defined the annotations on Gateway like this like this:
所以，我像这样在Gateway上定义了注释：

  1. service.beta.kubernetes.io/aws-load-balancer-type: "external"
  2. service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
  3. service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"

And, of course, adding the service.beta.kubernetes.io/aws-load-balancer-nlb-target-type to the list of annotations to copy.
当然，还要将service.beta.kubernetes.io/aws-load-balancer-nlb-target-type添加到要复制的注释列表中。

If you don't wish to install it, I think you can simply change the value of service.beta.kubernetes.io/aws-load-balancer-type from nlb-ip to nlb. I haven't tested it, but it should be the equivalent setting for the built-in Kubernetes load balancer controller.
如果您不想安装它，我认为您可以简单地将service.beta.kubernetes.io/aws-load-balancer-type的值从nlb-ip更改为nlb。我没有测试过，但这应该是内置的Kubernetes负载均衡器控制器的等效设置。

Hope this helps!
希望这有所帮助！

Also I have a few suggestions which are probably not related to the issue but might help you maintain a cleaner code.
另外，我有一些建议，可能与问题无关，但可能有助于保持更干净的代码。

You should change the deprecated ReferencePolicy object to a ReferenceGrant.
您应该将已弃用的ReferencePolicy对象更改为ReferenceGrant。

In addition, you might not need to define a custom GatewayClass and GatewayClassConfig, since you can control many of the settings of the load balancer through the values.yaml file you use to deploy Consul, making your configuration shorter and simpler.
此外，您可能不需要定义自定义的GatewayClass和GatewayClassConfig，因为您可以通过用于部署Consul的values.yaml文件来控制负载均衡器的许多设置，从而使您的配置更短更简单。

For example, this is the relevant part from my file:
例如，这是我的文件中相关的部分：

  1. apiGateway:
  2.   enabled: true
  3.   image: "hashicorp/consul-api-gateway:0.5.1"
  4.   # Configuration settings for the default GatewayClass which Consul will use.
  5.   managedGatewayClass:
  6.     serviceType: LoadBalancer
  7.     copyAnnotations:
  8.       service:
  9.         annotations: |
  10.           - service.beta.kubernetes.io/aws-load-balancer-type
  11.           - service.beta.kubernetes.io/aws-load-balancer-scheme
  12.           - service.beta.kubernetes.io/aws-load-balancer-nlb-target-type

And then in the Gateway I just set gatewayClassName: consul-api-gateway which is the default class.
然后，在Gateway中，我只需设置gatewayClassName: consul-api-gateway，这是默认的类。

英文:

I've had a very similar issue today, and spend hours trying to figure it out.
I also had a very similar setup with a Consul API Gateway behind an internal AWS load balancer.

What solved it for me, was changing the load balancer type from ip to instance.
In my case , I was using the AWS Load balancer controller, which seems to be the recommended way to go.
So, I defined the annotations on Gateway like this like this:

  1. service.beta.kubernetes.io/aws-load-balancer-type: "external"
  2. service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
  3. service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"

And, of course, adding the service.beta.kubernetes.io/aws-load-balancer-nlb-target-type to the list of annotations to copy.

If you don't wish to install it, I think you can simply change the value of
'service.beta.kubernetes.io/aws-load-balancer-type from 'nlb-ip' to 'nlb'. I haven't tested it, but it should be the equivalent setting for the built-in kubernetes load balancer controller.

Hope this helps!

Also I have a few suggestion which are probably not related to the issue but might help you maintain a cleaner code.
You should change the deprecated ReferencePolicy object to a ReferenceGrant.
In addition, you might not need to define a custom GatewayClass and GatewayClassConfig, since you can control many of the settings of the load balancer through the values.yaml file you use to deploy consul, making your configuration shorter and simpler.
For example, this is the relevant part from my file

  1. apiGateway:
  2.   enabled: true
  3.   image: "hashicorp/consul-api-gateway:0.5.1"
  4.   # Configuration settings for the default GatewayClass which consul will use.
  5.   managedGatewayClass:
  6.     serviceType: LoadBalancer
  7.     copyAnnotations:
  8.       service:
  9.         annotations: |
  10.           - service.beta.kubernetes.io/aws-load-balancer-type
  11.           - service.beta.kubernetes.io/aws-load-balancer-scheme
  12.           - service.beta.kubernetes.io/aws-load-balancer-nlb-target-type

And then in the Gateway I just set gatewayClassName: consul-api-gateway which is the default class.

huangapple
  • 本文由 发表于 2023年4月11日 03:36:44
  • 转载请务必保留本文链接：https://go.coder-hub.com/75980150.html
  • amazon-eks
  • api-gateway
  • consul
  • keycloak
  • kubernetes
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定