How the node pull decreases after the downscale of services in GKE

I have the workload of 6 nodes in 1 node pool, and some number of services, let's say 3n + k. Then I am downscaling them to the number of n + k, and the load of all nodes reduces from 80% to approximately 45-50%, and I want my services to be reschedulled to reduce the overall number of nodes, but this does not happen. Why? Do I need to wait more? Do I need to make some other actions?

Once the load decreases to 45 to 50%, GKE should automatically rearrange the workloads between nodes to effectively utilize the resources. However, this process may take some time as Kubernetes keeps nodes in a buffer, expecting the same amount of traffic for a certain duration. If the expected traffic doesn't materialize, Kubernetes should eventually take action on its own to rebalance the workloads. If this activity is not happening or if node downscaling is not occurring, it could be due to the following reasons:

1. Pod Disruption Budget: If there is a Pod Disruption Budget (PDB) set for the workloads, GKE will not be able to kill running pods on the node, which would prevent the node drain from happening. PDBs define the minimum number of pods that must be available during a disruption, and they can prevent GKE from draining nodes when the PDB constraints are not met.

2. Pods Using Node Storage: If a pod is using local storage on a node, Kubernetes will avoid killing that pod to avoid data loss. In such cases, the workloads won't be rearranged until Kubernetes can safely terminate the pods without losing data.

3. Adding safe to evict annotation:By adding eviction annotations to deployments, administrators can ensure better pod re-arrangement, mitigating potential problems caused by pods not being evicted as expected.

> apiVersion: v1
> kind: Pod | Deployment
> metadata:
> annotations:
> cluster-autoscaler.kubernetes.io/safe-to-evict: "true"

To resolve issues with node downscaling and workload rearrangement, you should check and address the above points. If there are Pod Disruption Budgets in place, consider adjusting them to allow for more flexibility during node drain. Additionally, if pods are using local storage, consider migrating them to use network-attached storage or other persistent storage solutions that allow for better workload redistribution. Once these issues are addressed, Kubernetes should be able to handle node downscaling based on pod load and resource utilization effectively.