Kubernetes Ingress-Controllers 在 Ingress 中争夺地址。

huangapple
go评论84阅读模式
英文:

Kubernetes Ingress-Controllers "fighting" over Address in Ingress

问题

基础设施背景：

我在我的Kubernetes（K3s）集群中有4个节点。

  1. k3s-server 位置=家（VM），主要用于etcd复制
  2. k3s-agent 位置=家（VM），运行大多数Pod
  3. mercury 位置=家（RPI4），重要Pod的备份
  4. moon 位置=云（云VM），在公共云中运行某些工作负载

我在两个命名空间中运行ingress-nginx Helm图表的两个不同实例。一个用于在ingressClassName=nginx时在我的家中公开本地服务，使用loadBalancerIP: 192.168.113.230运行在nginx-ingres-home命名空间中。云控制器使用ingressClassName=nginx-cloud和loadBalancerIP: 91.x.x.x，并运行在nginx-ingress-cloud命名空间中。 （Helm图表和示例Ingress的值如下）

问题

现在我有多个定义的Ingress，它们使用不同的类。然而，使用kubectl get ingress -A提供以下输出。

  1. NAMESPACE              NAME                   CLASS         HOSTS                        ADDRESS           PORTS     AGE
  2. kubernetes-dashboard   kubernetes-dashboard   nginx         k3s.local.example.com        192.168.113.230   80, 443   4d3h
  3. longhorn-system        longhorn-ingress       nginx         longhorn.local.example.com   192.168.113.230   80, 443   10d
  4. mailu                  mailu                  nginx-cloud   mail.example.com             192.168.113.230   80, 443   2d23h
  5. pihole                 pihole                 nginx         dns.local.example.com        192.168.113.230   80, 443   10d
  6. ubiquiti               unifi-web-interface     nginx         unifi.local.example.com       192.168.113.230   80, 443   24h

可以看到，不管在Ingress中设置的ingressClassName是什么，地址始终来自其中一个ingress控制器。这些地址定期切换。 “nginx-ingress-home”的日志显示，控制器始终更新Ingress的地址（1分钟间隔）：

  1. I0510 19:46:12.087815       7 status.go:300] "updating Ingress status" namespace="pihole" ingress="pihole" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
  2. I0510 19:46:12.087857       7 status.go:300] "updating Ingress status" namespace="ubiquiti" ingress="unifi-web-interface" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
  3. I0510 19:46:12.088485       7 status.go:300] "updating Ingress status" namespace="mailu" ingress="mailu" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
  4. I0510 19:46:12.088782       7 status.go:300] "updating Ingress status" namespace="longhorn-system" ingress="longhorn-ingress" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
  5. I0510 19:46:12.090051       7 status.go:300] "updating Ingress status" namespace="kubernetes-dashboard" ingress="kubernetes-dashboard" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]

当然，“nginx-ingress-cloud”也执行相同的操作，只是用“91.x.x.x”替换“192.168.113.230”。

有人知道如何阻止它们“占有”所有Ingress，只更新分配了相同IngressClass的Ingress吗？

配置

nginx-ingress-home的值

  1. controller:
  2.   ingressClass: "nginx"
  3.   ingressClassResource:
  4.     name: nginx
  5.     enabled: yes
  6.     default: yes
  7.   service:
  8.     type: "LoadBalancer"
  9.     loadBalancerIP: 192.168.113.230
  10.   nodeSelector:
  11.     location: home
  12.   tolerations: #Allow running on backup nodes
  13.     - key: "backup"
  14.       operator: "Equal"
  15.       value: "true"
  16.       effect: "NoSchedule"
  17.   affinity: #Prefer running on nodes labled type=power
  18.     nodeAffinity:
  19.       preferredDuringSchedulingIgnoredDuringExecution:
  20.         - weight: 1
  21.           preference:
  22.             matchExpressions:
  23.               - key: type
  24.                 operator: In
  25.                 values:
  26.                   - power

nginx-ingress-cloud的值

  1. controller:
  2.   ingressClass: "nginx-cloud"
  3.   ingressClassResource:
  4.     name: nginx-cloud
  5.     enabled: yes
  6.     default: no
  7.   service:
  8.     type: "LoadBalancer"
  9.     loadBalancerIP: 91.x.x.x
  10.   nodeSelector:
  11.     location: cloud

示例home Ingress：Pihole

  1. ---
  2. apiVersion: networking.k8s.io/v1
  3. kind: Ingress
  4. metadata:
  5.   name: kubernetes-dashboard
  6.   namespace: kubernetes-dashboard
  7.   annotations:
  8.     cert-manager.io/cluster-issuer: letsencrypt-prod
  9. spec:
  10.   ingressClassName: nginx
  11.   tls:
  12.     - hosts:
  13.         - k3s.local.example.com
  14.       secretName: kubernetes-dashboard-tls
  15.   rules:
  16.     - host: k3s.local.example.com
  17.       http:
  18.         paths:
  19.           - backend:
  20.               service:
  21.                 name: kubernetes-dashboard
  22.                 port:
  23.                   number: 443
  25.             path: /
  26.             pathType: Prefix
英文:

Infrastructure Background:

I have 4 nodes in my Kubernetes (K3s) cluster.

  1. k3s-server location=home   (VM) maily used for etcd replication 
  2. k3s-agent  location=home   (VM) runs most pods
  3. mercury    location=home   (RPI4) backup for important pods
  4. moon       location=cloud  (Cloud VM) runs certain workloads in a public cloud

I am running 2 different instances of the ingress-nginx helm chart in two namespaces. One to exposes local services at my home when ingressClassName=nginx using loadBalancerIP: 192.168.113.230 running in the nginx-ingres-home namespace. The cloud controller uses ingressClassName=nginx-cloud with loadBalancerIP: 91.x.x.x and runs inside the nginx-ingress-cloud namespace. (Values for the Helm charts & example Ingress below)

The Problem

I now have multiple Ingresses defined, which are using the different classes. However using kubectl get ingress -A provides the following output.

  1. NAMESPACE              NAME                   CLASS         HOSTS                        ADDRESS           PORTS     AGE
  2. kubernetes-dashboard   kubernetes-dashboard   nginx         k3s.local.example.com        192.168.113.230   80, 443   4d3h
  3. longhorn-system        longhorn-ingress       nginx         longhorn.local.example.com   192.168.113.230   80, 443   10d
  4. mailu                  mailu                  nginx-cloud   mail.example.com             192.168.113.230   80, 443   2d23h
  5. pihole                 pihole                 nginx         dns.local.example.com        192.168.113.230   80, 443   10d
  6. ubiquiti               unifi-web-interface     nginx         unifi.local.example.com       192.168.113.230   80, 443   24h

You can see that regardless of the ingressClassName set in the Ingress, the addresses are always the ones from one of the ingress-controllers. The addresses are switching periodically. The logs of nginx-ingress-home show, that the controller permanently updates the addresses of the Ingresses (1m interval):

  1. I0510 19:46:12.087815       7 status.go:300] "updating Ingress status" namespace="pihole" ingress="pihole" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
  2. I0510 19:46:12.087857       7 status.go:300] "updating Ingress status" namespace="ubiquiti" ingress="unifi-web-interface" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
  3. I0510 19:46:12.088485       7 status.go:300] "updating Ingress status" namespace="mailu" ingress="mailu" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
  4. I0510 19:46:12.088782       7 status.go:300] "updating Ingress status" namespace="longhorn-system" ingress="longhorn-ingress" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]
  5. I0510 19:46:12.090051       7 status.go:300] "updating Ingress status" namespace="kubernetes-dashboard" ingress="kubernetes-dashboard" currentValue=[{IP:91.x.x.x Hostname: Ports:[]}] newValue=[{IP:192.168.113.230 Hostname: Ports:[]}]

Of course the nginx-ingress-cloud does the same thing, just replacing 192.168.113.230 with 91.x.x.x.

Does somebody know, how to stop them from "taking owernship" of all Ingresses and only updating the ones with the same IngressClass assigned?

Configs

Values of nginx-ingress-home

  1. controller:
  2.   ingressClass: "nginx"
  3.   ingressClassResource:
  4.     name: nginx
  5.     enabled: yes
  6.     default: yes
  7.   service:
  8.     type: "LoadBalancer"
  9.     loadBalancerIP: 192.168.113.230
  10.   nodeSelector:
  11.     location: home
  12.   tolerations: #Allow running on backup nodes
  13.     - key: "backup"
  14.       operator: "Equal"
  15.       value: "true"
  16.       effect: "NoSchedule"
  17.   affinity: #Prefer running on nodes labled type=power
  18.     nodeAffinity:
  19.       preferredDuringSchedulingIgnoredDuringExecution:
  20.         - weight: 1
  21.           preference:
  22.             matchExpressions:
  23.               - key: type
  24.                 operator: In
  25.                 values:
  26.                   - power

Values of nginx-ingress-cloud

  1. controller:
  2.   ingressClass: "nginx-cloud"
  3.   ingressClassResource:
  4.     name: nginx-cloud
  5.     enabled: yes
  6.     default: no
  7.   service:
  8.     type: "LoadBalancer"
  9.     loadBalancerIP: 91.x.x.x
  10.   nodeSelector:
  11.     location: cloud

Example home Ingress: Pihole

  1. ---
  2. apiVersion: networking.k8s.io/v1
  3. kind: Ingress
  4. metadata:
  5.   name: kubernetes-dashboard
  6.   namespace: kubernetes-dashboard
  7.   annotations:
  8.     cert-manager.io/cluster-issuer: letsencrypt-prod
  9. spec:
  10.   ingressClassName: nginx
  11.   tls:
  12.     - hosts:
  13.         - k3s.local.example.com
  14.       secretName: kubernetes-dashboard-tls
  15.   rules:
  16.     - host: k3s.local.example.com
  17.       http:
  18.         paths:
  19.           - backend:
  20.               service:
  21.                 name: kubernetes-dashboard
  22.                 port:
  23.                   number: 443
  25.             path: /
  26.             pathType: Prefix

答案1

得分: 1

借助Blender Fox的评论，我能够找到解决方案。

IngressClass对象具有属性spec.controller，这似乎是入口控制器在选择入口类是否属于其类集时查看的值。我能够通过在Helm Chart中设置controller.ingressClassResource.controllerValue的不同值来更改该值。在我的情况下，我选择了k8s.io/ingress-nginx/nginx和k8s.io/ingress-nginx/nginx-cloud，而不是默认的k8s.io/ingress-nginx。

英文:

With the help of a comment from Blender Fox I was able to figure out the solution.

The IngressClass objects have the property spec.controller, which seems to be the value, for which a ingress controller looks, when choosing if the ingressClass is part of his set of classes. I was able to change that value in the Helm Chart, by setting the controller.ingressClassResource.controllerValue to different values. In my case I chose k8s.io/ingress-nginx/nginx and k8s.io/ingress-nginx/nginx-cloud instead of the default k8s.io/ingress-nginx.

huangapple
  • 本文由 发表于 2023年5月11日 04:16:39
  • 转载请务必保留本文链接：https://go.coder-hub.com/76222260.html
  • ingress-nginx
  • kubernetes
  • kubernetes-ingress
  • nginx
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定