Nginx Ingress getting 504 gateway time-out

I’m quite new to k8s in general, only been using for smaller projects but made it work. I hope btw this is the right channel to ask questions (in this case about ingress-nginx). I’m trying to setup a cluster with a gateway-api and a few microservices (all written in NestJs). To give a little background, I first had everything in docker-compose and my entry was also a Nginx container with letsencrypt. The whole docker, works great locally.

This was the config used for my NGinx Docker:

upstream equmedia-api {
    server equmedia-api:3000;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name localhost;
	return 301 https://$server_name$request_uri;
}

server {
    listen 80;
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;
    keepalive_timeout 70;
    server_name subdomain.example.com;

    ssl_session_cache shared:SSR:10m;
    ssl_session_timeout 10m;
    ssl_certificate /etc/letsencrypt/live/equmedia.pixeliner.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/equmedia.pixeliner.com/privkey.pem;

    access_log /var/log/nginx/nginx.access.log;
    error_log /var/log/nginx/nginx.error.log;

    location / {
        proxy_pass http://equmedia-api;
        # proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

As you can see, it upstreamed to my api container.

Eventually I wanted to turn the whole deployment into k8s. Seemed like a good followup practice after the small projects.

I learned about ingress-nginx and gave it my first try, but I seem to have struck a wall.

Here is the setup I'm trying to achieve:

Through DigitalOcean the setup will be behind a LoadBalancer.

Here is my Ingress NGinx controller:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: equmedia-ingress-api
  annotations:
    kubernetes.io/ingress.class: "nginx"    
    cert-manager.io/issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/rewrite-target: "/"
    nginx.ingress.kubernetes.io/proxy-protocol: "true"
    nginx.ingress.kubernetes.io/ssl-proxy-headers: "X-Forwarded-Proto: https"
spec:
  tls:
  - hosts:
    - subdomain.example.com
    secretName: quickstart-example-tls
  rules:
  - host: subdomain.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: equmedia-api
            port:
              number: 3000

And my api service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.22.0 (955b78124)
  creationTimestamp: null
  labels:
    io.kompose.service: equmedia-api
  name: equmedia-api
spec:
  ports:
    - port: 3000
      targetPort: 3000
  selector:
    io.kompose.service: equmedia-api
status:
  loadBalancer: {}

When I try to access "https://subdomain.example.com/api/health", I get a 504 Gateway Time-out. Looking at the ingress controller logs I get the following:

2023/02/17 15:51:44 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
2023/02/17 15:51:49 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
2023/02/17 15:51:54 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"

Anyone that can point me into the right direction, to fix this issue?

EDIT

The outcome for

kubectl get pods -l io.kompose.service=equmedia-api:

NAME           READY   STATUS    RESTARTS   AGE
equmedia-api   1/1     Running   0          2d2h

kubectl get svc:

NAME                                            TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                      AGE
equmedia-api                                    ClusterIP      10.245.173.11   <none>            3000/TCP                     23h
equmedia-api-rabbitmq                           ClusterIP      10.245.17.225   <none>            5672/TCP,15673/TCP           2d17h
equmedia-api-redis                              ClusterIP      10.245.120.11   <none>            6379/TCP                     2d17h
equmedia-auth-db                                ClusterIP      10.245.94.21    <none>            5432/TCP                     2d17h
kubernetes                                      ClusterIP      10.245.0.1      <none>            443/TCP                      2d17h
quickstart-ingress-nginx-controller             LoadBalancer   10.245.36.216   179.128.139.106   80:31194/TCP,443:31609/TCP   2d16h
quickstart-ingress-nginx-controller-admission   ClusterIP      10.245.232.77   <none>            443/TCP                      2d16h

EDIT2:

I've requested my domain https://subdomain.example.com/api/health through browser, curl and postman. All of them return timeouts.

kubectl get pods -A -o wide | grep 10.244.0.228
returns:

default        equmedia-api                                           1/1     Running   0               2d4h    10.244.0.228   temp-pool-qyhii   <none>           <none>

kubectl get svc -A | grep 10.244.0.228 returns nothing

EDIT3:

Here is the logs of my API:

[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [NestFactory] Starting Nest application...
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] AppModule dependencies initialized +136ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] RedisCacheModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] UtilsModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] AxiosWrapperModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] PassportModule dependencies initialized +32ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] JwtModule dependencies initialized +3ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] TerminusModule dependencies initialized +2ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +2ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] BullModule dependencies initialized +61ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ClientsModule dependencies initialized +17ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ClientsModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ClientsModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ClientsModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ClientsModule dependencies initialized +7ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ClientsModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] HealthModule dependencies initialized +8ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] CacheModule dependencies initialized +2ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] MailModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] HttpModule dependencies initialized +3ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] BullModule dependencies initialized +24ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] BullQueueModule dependencies initialized +7ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] PaymentModule dependencies initialized +8ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] CustomerModule dependencies initialized +1ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] ContentModule dependencies initialized +2ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] AdserveModule dependencies initialized +3ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] AuthModule dependencies initialized +2ms
[Nest] 1  - 02/17/2023, 10:52:27 AM     LOG [InstanceLoader] OpenIdModule dependencies initialized +65ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] HealthController {/api/health}: +18ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/health, GET} route +5ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/health/check-ping, GET} route +2ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/health/check-disk, GET} route +2ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/health/check-memory, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/health/check-microservice/:name, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] OpenIdController {/api/open-id}: +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/open-id/login, GET} route +2ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/open-id/user, GET} route +2ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/open-id/callback, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/open-id/logout, GET} route +2ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] AuthController {/api/auth}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/auth, GET} route +2ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/auth/signup, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/auth/signin, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/auth/signout, POST} route +2ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/auth/refresh, GET} route +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] UserController {/api/user}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/user/get-user-id/email?, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/user/get-authenticated-user, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/user/:id/change-user-password, PUT} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/user/:id/delete-user-account, DELETE} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/user/confirm/:token, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/user/forgot-password, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/user/set-new-password/:token, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] UsersController {/api/users}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/users, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] PaymentController {/api/payment}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/payment/:id, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/payment/create/:id, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/payment/:id, PUT} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] CustomerController {/api/customer}: +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/customer, GET} route +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/customer/profile/:id, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/customer/create, POST} route +2ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/customer/delete/:id, DELETE} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/customer/update/:id, PUT} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] ContentController {/api/content}: +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/content, GET} route +2ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/content/create, POST} route +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/content/update/:contentId, PUT} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/content/delete/:contentId, DELETE} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/content/category/:categoryId, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/content/slug/:slug, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] CategoryController {/api/category}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/category, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/category/create, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/category/update/:categoryId, PUT} route +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/category/delete/:categoryId, DELETE} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] WidgetController {/api/widget}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/widget, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/widget/create, POST} route +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/widget/update/:widgetId, PUT} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/widget/delete/:widgetId, DELETE} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] AdvertiserController {/api/adserve/advertiser}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/advertiser, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/advertiser/create, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId/campaigns/create, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId/campaigns/:campaignId, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId/campaigns/:campaignId/create, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/advertiser/:advertiserId/campaigns/:campaignId/assign, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] AdserveController {/api/adserve}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/serve, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/redirect, GET} route +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] PublisherController {/api/adserve}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/publisher, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/publisher/:publisherId, GET} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/publisher/create, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/publisher/:publisherId/zone/create, POST} route +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RoutesResolver] ReportController {/api/adserve/report}: +1ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [RouterExplorer] Mapped {/api/adserve/report, GET} route +0ms
[Nest] 1  - 02/17/2023, 10:52:28 AM     LOG [NestApplication] Nest application successfully started +58ms
-- API GATEWAY RUNNING - PORT: 3000  --

No errors are logged, and through a port-forward I also see my api working.

EDIT4:

Here is the gist with all pods/services/claims/...

https://gist.github.com/pixeliner/2c89048294197155b0d4833ab4045f3c

Your output text:

2023/02/17 15:51:44 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
2023/02/17 15:51:49 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"
2023/02/17 15:51:54 [error] 356#356: *336457 upstream timed out (110: Operation timed out) while connecting to upstream, client: 164.92.221.107, server: subdomain.example.com, request: "GET /api/health HTTP/2.0", upstream: "http://10.244.0.228:3000/", host: "subdomain.example.com"

Implies the request is timing out on the IP 10.244.0.228:3000

Things to check:

• Is the service IP 10.244.0.228: kubectl get svc equmedia-api (it will likely be of type ClusterIP)

• Port forward to the service directly: kubectl port-forward svc/equmedia-api 3000:3000 and then try to access localhost:3000 in another terminal or in your browser. Does it respond, does it error or does it timeout?

• Check the pods your service is trying to match: kubectl get pods -l io.kompose.service=equmedia-api -- does this return any pods? If so, are they in Ready state or are they erroring? Do they have a value greater than 0 in the Restarts count?

• Check the logs of the pod(s) kubectl logs -f {pod-name} and see if it is unable to start up, or if it is repeatedly starting.

UPDATE 1

Please add the output of the following commands to your question. Wrap the output with three backticks (`) on a single line before and after to preserve formatting:

kubectl get pods -l io.kompose.service=equmedia-api

kubectl get svc

UPDATE 2

Since the IP that your controller is 10.244.0.228 see if any of your pods or services actually have that IP. Please add the output of these commands:

kubectl get pods -A -o wide | grep 10.244.0.228

kubectl get svc -A | grep 10.244.0.228

UPDATE 3

I've yet to try deploying the gist, but I have noticed something

You have networkpolicies setup and you have labelled your pod

apiVersion: v1
kind: Pod
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.network/backend: "true" # <<--- HERE
io.kompose.service: equmedia-api
name: equmedia-api-pod
spec:
...

This matches your network policy here:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
creationTimestamp: null
name: backend
spec:
ingress:
- from:
- podSelector:
matchLabels:
io.kompose.network/backend: "true"
podSelector:
matchLabels:
io.kompose.network/backend: "true"

Now, this network policy reads (based in the information off this link)

"Allow connections from Pods with the label io.kompose.network/backend="true" (last three lines) to pods that match the labels io.kompose.network/backend="true" (the ingress.from.podSelector bit)

Sooo.... assuming I'm reading this correct, the reason the ingress controller is not able to talk to the pod, is because the controller pod does not have a label io.kompose.network/backend="true", and since you did not include that in your gist, I'm assuming you're using the ingress controller chart as a subchart/dependency. And if so, then out of the box, that chart won't have this label. This would explain why we were able to port-forward to the pod and the service directly, but the controller pod was not able to talk to the pod.

And easy way to verify this is to either delete the backend networkpolicy, or modify it to allow all ingress traffic as a test (something like the example here)

If this works, it will confirm the networkpolicy is blocking the traffic.