K8s Operator listen to secret change with event filter

We have created a few month ago controller which runs great using kubebuilder.

Few weeks ago we added a “listener” to a secret which when the secret is changing (secret properties)
The reconcile should be invoked, the problem is that it is sometimes working and sometimes not, (you change the secret apply it and the reconcile doesn’t happens) , we are doing it for the exact same secret file.

We tried few days to find the root cause without success, (we change the k8s.io/client-go v0.23.4 and also to v0.22.3 and now v0.22.1 that is only working.
Any idea what the issue could be? any hint will be helpful. or Any other way to do it that we can try out.

func (r *vtsReconciler) SetupWithManager(mgr ctrl.Manager) error {
	manager := ctrl.NewControllerManagedBy(mgr).
		For(&vts.str).
		WithEventFilter(predicate.Or(predicate.AnnotationChangedPredicate{}))
	manager = manager.Watches(&source.Kind{Type: &v1.Secret{}}, handler.EnqueueRequestsFromMapFunc(func(a client.Object) []reconcile.Request {
		return r.SecretRequests.SecretFinder(a.GetName())
	}))
	return manager.Complete(r)
}



func (secm *SecretMapper) SecretFinder(name string) []reconcile.Request {
	v := cli.ObjectKey{Name: name}
	return secm.SecMap[v.String()]
}

Most likely the issue is that the WithEventFIlter applies to all watched objects by the controller. The generation is auto-incremented for CRDs, but this doesn't hold for all resource types.

From the GenerationChangedPredicate docs:

// Caveats:
//
// * The assumption that the Generation is incremented only on writing to the spec does not hold for all APIs.
// E.g For Deployment objects the Generation is also incremented on writes to the metadata.annotations field.
// For object types other than CustomResources be sure to verify which fields will trigger a Generation increment when they are written to.

You can check this by creating a secret / updating a secret you will see that there is no generation set (at least not on my local k3d cluster).

Most likely it works for the creation as initially the controller will sync existing resources with the cluster.

To solve it you can use:

func (r *vtsReconciler) SetupWithManager(mgr ctrl.Manager) error {
    manager := ctrl.NewControllerManagedBy(mgr).
        For(&vts.str, WithPredicates(predicate.Or(predicate.GenerationChangedPredicate{}, predicate.AnnotationChangedPredicate{}))).
    manager = manager.Watches(&source.Kind{Type: &v1.Secret{}}, handler.EnqueueRequestsFromMapFunc(func(a client.Object) []reconcile.Request {
        return r.SecretRequests.FindForSecret(a.GetNamespace(), a.GetName())
    }))
    return manager.Complete(r)
}

which should apply the predicates only to your custom resource.