CockroachDB Kubernetes Cluster Ingress Setup

So my goal is to be able to access my CockroachDB from domain like db.test.com with cert.

I want to use cert-manager letsencrypt to issue keys. And it should work with CF (in non proxy mode as I think they do not support tcp for this)

At first to test everything I used normal kubectl port-forward which worked, but now I needed to expose it always.

I have tried using Ingress (using ingress-nginx)

• I know that Ingress is mostly HTTP/HTTPS but I saw it can be used for the thing I need and IN CF I cannot point to port that I needed.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tcp-example-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/tcp-services: "cluster-cockroachdb-public"
    nginx.ingress.kubernetes.io/tcp-service-port: "26257"
    nginx.ingress.kubernetes.io/backend-protocol: "TCP"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
    - hosts:
        - db.test.com
      secretName: db-access-ssl-cert-production
  rules:
    - host: db.test.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: cluster-cockroachdb-public
                port:
                  number: 26257

Attempting to connect does not work, and in logs I can see 400 status code with strange characters like \x20...

No matter what I tried I could not get it to work..

I did manage to get web-ui portion working that was easy enough.

Other resource that might be helpful is the values.yaml that I used

conf:
  cache: "2Gi"
  max-sql-memory: "2Gi"
# My WEB-UI that works
ingress:
  enabled: true
  labels: {}
  annotations: 
     kubernetes.io/ingress.class: nginx
     cert-manager.io/cluster-issuer: letsencrypt-production
  paths: [/]
  hosts:
    - db-ui.test.com
  tls:
    - hosts: [db-ui.test.com]
      secretName: ssl-cert-production

Everything else is default

I solved my issue by following the tutorial below:

https://mailazy.com/blog/exposing-tcp-udp-services-ingress/

also mentioned here

https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

Ingress does not support TCP or UDP services so we use ingress-nginx config for it we patch ingress-nginx values of chart and add custom one (Copy default values.yaml from github helm chart for ingress-nginx)

I just edited this portion:

# -- TCP service key-value pairs
## Ref: https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/exposing-tcp-udp-services.md
##
tcp:
  "26257": "default/cluster-cockroachdb-public:26257"

After that we run helm upgrade command to replace values of ingress-nginx and after that it should work for anyone else as well.

If you are using cloudflare make sure to disable proxy!