consul-api-gateway load balancer always shows registered target un healthy in aws eks environment

I am trying to create a consul-api-gateway (Gateway) resource with HttpRoute for keycloak service in aws eks cluster. I am configuring in a way that the api-gateway and keycloak service are running in different namespaces. But when consul-api-gateway resource created with below deployment yaml, it is creating aws load balancer with the registered target showing as unhealthy. Not sure what makes this as unhealthy always.

Consul-server-and-ui-deployment.yaml

---
global:
  name: consul  
  imagePullSecrets:
    - name: "***"
  image: /hashicorp/consul:1.12.0
  imageEnvoy: /envoyproxy/envoy:v1.22.2
  imageK8S: /hashicorp/consul-k8s-control-plane:0.44.0
  metrics:
    enabled: true
server:
  replicas: 3
  bootstrapExpect: 3
  extraLabels:
    tags.datadoghq.com/source: consul
    tags.datadoghq.com/service: consul-server
  annotations: |
    "tags.datadoghq.com/source":                              "consul"
    "tags.datadoghq.com/service":                             "consul-server"
  updatePartitions: 3
  disruptionBudget:
    maxUnavailable: 1
  tolerations: >
    -
    key:                                                    "taint_for_consul_xor_vault"
      operator:                                               "Equal"
      value:                                                  "true"
      effect:                                                 "NoSchedule"
  nodeSelector: |
    'eks.amazonaws.com/capacityType':                         'ON_DEMAND'
    'purpose':                                                'consul-server'
  storage: 50G
  priorityClassName: high-priority
client:
  enabled: true
  grpc: true
  exposeGossipPorts: false
  priorityClassName: high-priority
  resources:
    requests:
      memory: 1G
      cpu: "1"
    limits:
      memory: 1G
      cpu: "1"
  healthChecks:
    enabled: true
  hostNetwork: false
  extraConfig: |
    {
      "advertise_reconnect_timeout":                          "15m",
      "limits":                                               {
        "http_max_conns_per_client":                          -1
      }
    }
  extraLabels:
    tags.datadoghq.com/source: consul
    tags.datadoghq.com/service: consul-client
  annotations: |
    "tags.datadoghq.com/source":                              "consul"
    "tags.datadoghq.com/service":                             "consul-client"
  updateStrategy: |
    type:                                                     RollingUpdate
    rollingUpdate:
      maxUnavailable:                                         1
ui:
  enabled: true
  service:
    type: LoadBalancer
    additionalSpec: "'ports': [{'name': 'http', 'protocol': 'TCP', 'port': 8500,
      'targetPort': 8500}, {'name': 'https', 'protocol': 'TCP', 'port': 8501,
      'targetPort': 8501}]"
    annotations: |
      'service.beta.kubernetes.io/aws-load-balancer-scheme':  'internal'
      'service.beta.kubernetes.io/aws-load-balancer-type':    'nlb-ip'
controller:
  enabled: true
prometheus:
  enabled: true
grafana:
  enabled: true
terminatingGateways:
  enabled: true
  priorityClassName: high-priority
  defaults:
    replicas: 2
apiGateway:
  enabled: true
  image: /hashicorp/consul-api-gateway:0.3.0
  controller:
    replicas: 2
    priorityClassName: high-priority
connectInject:
  enabled: true
  priorityClassName: high-priority
  transparentProxy:
    defaultEnabled: true
  default: false
syncCatalog:
  enabled: true
  priorityClassName: high-priority
  default: false
  toConsul: true
  toK8S: false

Consul-api-gateway-deployment.yaml

apiVersion:         api-gateway.consul.hashicorp.com/v1alpha1
kind:               GatewayClassConfig
metadata:
  name:             consul-common-gateway-class-config
spec:
  logLevel:         'info'  
  copyAnnotations:
    service:
      - service.beta.kubernetes.io/aws-load-balancer-scheme
      - service.beta.kubernetes.io/aws-load-balancer-type
  consul:
    scheme:         'http'
    ports:
      http:         8500
      grpc:         8502
  serviceType:      LoadBalancer
---
apiVersion:         gateway.networking.k8s.io/v1alpha2
kind:               GatewayClass
metadata:
  name:             consul-common-gateway-class
spec:
  controllerName:   'hashicorp.com/consul-api-gateway-controller'
  parametersRef:
    group:          api-gateway.consul.hashicorp.com
    kind:           GatewayClassConfig
    name:           consul-common-gateway-class-config
---
apiVersion:                                                 gateway.networking.k8s.io/v1alpha2
kind:                                                       Gateway
metadata:
  name:                                                     common-api-gateway
  annotations:
     'service.beta.kubernetes.io/aws-load-balancer-scheme':  'internal'
     'service.beta.kubernetes.io/aws-load-balancer-type':    'nlb-ip'           
spec:
  gatewayClassName:                                         consul-common-gateway-class
  listeners:  
  - protocol:                                               HTTP
    port:                                                   80
    name:                                                   http
    allowedRoutes:
      namespaces:
        from:                                               All

HttpRoute-keycloak-deployment.yaml

---
apiVersion:       gateway.networking.k8s.io/v1alpha2
kind:             HTTPRoute
metadata:
  name:           gateway-keycloak-route
  namespace:      app-ns
spec:
  parentRefs:
  - name:         common-api-gateway
    namespace:    consul
  rules:
  - matches:
    - path:
        type:     PathPrefix
        value:    /auth
    backendRefs:
    - kind:       Service
      name:       keycloak
      port:       80
      namespace:  app-ns
---
apiVersion:       gateway.networking.k8s.io/v1alpha2
kind:             ReferencePolicy
metadata:
  name:           reference-policy-keycloak
spec:
  from:
    - group:      gateway.networking.k8s.io
      kind:       HTTPRoute
      namespace:  app-ns
  to:
    - group:      ""
      kind:       Service
      name:       keycloak

I wanted api-gateway load balancer running in aws eks to be with healthy registered targets and wanted to run Keycloak service have proper routes to api-gateway

I've had a very similar issue today, and spend hours trying to figure it out.
I also had a very similar setup with a Consul API Gateway behind an internal AWS load balancer.

What solved it for me, was changing the load balancer type from ip to instance.
In my case , I was using the AWS Load balancer controller, which seems to be the recommended way to go.
So, I defined the annotations on Gateway like this like this:

service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"

And, of course, adding the service.beta.kubernetes.io/aws-load-balancer-nlb-target-type to the list of annotations to copy.

If you don't wish to install it, I think you can simply change the value of
'service.beta.kubernetes.io/aws-load-balancer-type from 'nlb-ip' to 'nlb'. I haven't tested it, but it should be the equivalent setting for the built-in kubernetes load balancer controller.

Hope this helps!

Also I have a few suggestion which are probably not related to the issue but might help you maintain a cleaner code.
You should change the deprecated ReferencePolicy object to a ReferenceGrant.
In addition, you might not need to define a custom GatewayClass and GatewayClassConfig, since you can control many of the settings of the load balancer through the values.yaml file you use to deploy consul, making your configuration shorter and simpler.
For example, this is the relevant part from my file

apiGateway:
  enabled: true
  image: "hashicorp/consul-api-gateway:0.5.1"
  # Configuration settings for the default GatewayClass which consul will use.
  managedGatewayClass:
    serviceType: LoadBalancer
    copyAnnotations:
      service:
        annotations: |
          - service.beta.kubernetes.io/aws-load-balancer-type
          - service.beta.kubernetes.io/aws-load-balancer-scheme
          - service.beta.kubernetes.io/aws-load-balancer-nlb-target-type

And then in the Gateway I just set gatewayClassName: consul-api-gateway which is the default class.