GKE Pod Migration to 2nd Nodepool blocked by PDB and 2nd Nodepool not scaling up

I have an old nodepool with machine type X, and I am migrating them workload to a new nodepool of machine type Y. Both nodepools are up, and I've disabled autoscaling on the old nodepool. I added a cordon to the old nodepool, then drained the nodes using the command provided by the GKE doc:

for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=old-pool -o=name); do
  kubectl drain --force --ignore-daemonsets --delete-emptydir-data --grace-period=300 "$node";
done

Those have a pod disruption budget (PDB) of Min 1 available, and the new nodepool has autoscaling setup with min 1 and max around 10.

My problem is that the drain does not complete, it gets stuck saying Cannot evict pod as it would violate the pod's disruption budget.. It makes sense that it doesn't evict the pod, but at the same time, should it trigger the creation of the same pod on the new nodepool?

As new pods are not getting created on the new nodepool to let the old node drain, I cannot migrate my workload (at least not without disruption, which I cannot have).

What am I missing here?

The log Cannot evict pod as it would violate the pod's disruption budget confirms that the root cause is due to the pod's disruption budget(PDB) configuration.

The main components of the PDB are minavailable and maxunavailable.

spec.minAvailable : this is the total number of pods that must be available after the eviction, in the absence of the evicted pod.

spec.maxUnavailable : this is the maximum number of pods that can go unavailable during an eviction.

According to the official documentation, an application's PodDisruptionBudget can also prevent autoscaling; if deleting nodes would cause the budget to be exceeded, the cluster does not scale down.

When scaling down, the cluster autoscaler respects scheduling and eviction rules set on Pods. These restrictions can prevent a node from being deleted by the autoscaler. A node's deletion could be prevented if it contains a Pod with any of these conditions:

> - The Pod's affinity or anti-affinity rules prevent rescheduling.
> - The Pod is not managed by a Controller such as a Deployment, StatefulSet, Job or ReplicaSet.
> - The Pod has local storage and the GKE control plane version is lower than 1.22. In GKE clusters with control plane version 1.22 or
> later, Pods with local storage no longer block scaling down.
> - The Pod has the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation.
> - The node's deletion would exceed the configured PodDisruptionBudget and the operation is unable to complete because of a deployment which is having PDB .From the behavior
> identified in the PDB you can change the minAvailable or
> maxUnavailable values of the PDB.

Can you check the above conditions which are preventing the node’s deletion in your deployment along with PDB.

Hope the mentioned information is useful to you.