Pods cannot resolve kubernetes DNS

TL;DR

From a pod I am able to resolve DNS addresses only when I am using CoreDNS endpoint IP. How to make pods use coreDNS IP address?

CoreDNS configuration

• IP address: 10.96.0.10
• endpoint: 10.244.0.222

Kubelet configuration

...
clusterDNS:
  - 10.96.0.10
...

Flannel

Default configuration from: https://github.com/flannel-io/flannel

Cluster

Is up and running and have these pods:

• coredns
• etcd
• apiserver
• controller-manager
• flannel
• scheduler

Also, the cluster consists of only a single master node (that is schedulable).

Pod setup

# run pod
kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools

Test cases

# 1
nslookup kubernetes.default
# output: ;; connection timed out; no servers could be reached

# 2
nslookup kubernetes.default 10.96.0.10
# output: ;; connection timed out; no servers could be reached

# 3
nslookup kubernetes.default 10.244.0.222
# output: resolved address, Address: 10.96.0.1

# 4
ping 10.96.0.1
# output: all packages lost

# 5
ping 10.96.0.10
# output: all packages lost

# 6
ping 10.244.0.222
# output: all packages received

How have that happened?

That issue appeared when I was updating a cluster. I have reinstalled kube-flannel and coreDNS services with their default equivalents:

• https://github.com/coredns/helm#configuration
• https://github.com/flannel-io/flannel

Throughout this process I have also uninstalled kube-proxy.

I don't know if kube-proxy is a necessary component of a k8s or does a flannel can be an alternative but I tried to reinstall it with this command: https://stackoverflow.com/a/71519601/19570235 However, without any luck.

# Error message:
could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt":

I was able to resolve this issue by changing clusterDNS section in kubelet from 10.96.0.10 to 10.244.0.222.

After that I have restarted kubelet service (k8s is working on Ubuntu) and pods. Now pods can resolve hostnames and other services, like google.com.

I am not really sure if that is a workaround or if it is necessary to provide coreDNS endpoint to the kubelet config and because of that I will keep this post open for a while before accepting this answer.