k8s控制器监视其他控制器的自定义资源(CR)。

huangapple go评论81阅读模式
英文:

k8s controller watch other controller CR

问题

我有一个正常工作的k8s operator,我需要在另一个operator的CRD(不是我的)上添加一个"watch",为了简单起见,我们称之为extCR,我们的operator cr称为inCR

我尝试了以下方法,但是存在一个问题,如何正确触发reconcile。

func (r *Insiconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&Inv1alpha1.Iget{}).
        Watches(&source.Kind{Type: &ext.Se{}}, handler.EnqueueRequestsFromMapFunc(r.FWatch)).
        Complete(r)
}

func (r *Insiconciler) FWatch(c client.Object) []reconcile.Request {
    val := c.(*ivi.Srv)
    req := reconcile.Request{NamespacedName: types.NamespacedName{Name: val.Name, Namespace: val.Namespace}}
    return []reconcile.Request{req}
}

问题在于我使用extCR触发了reconcile,我希望在FWatch中更新inCR并以inCR而不是extCR开始reconcile,我该如何做到?

我的意思是,避免像下面的代码一样,有时reconcile是针对inCR,有时是针对extCR,这样我就会得到一些丑陋的if语句。

func (r *Insiconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var inCR FOO
    var extCR BAR

    if err := r.Get(ctx, req.NamespacedName, &inCR); err != nil {
        return ctrl.Result{}, err
    }

    if err := r.Get(ctx, req.NamespacedName, &extCR); err != nil {
        return ctrl.Result{}, err
    }

    // ...
}

我想知道处理这种情况的正确/清晰的方法是什么?

当您需要监听外部CR(不是您控制器的一部分)和内部CR(来自您的控制器)时。

还有一件事 - CR具有不同的GVK,但是externalCR包含许多不需要的字段,只有其中一些字段是必需的,并且这些字段在两个CR上具有相同的名称。

更新

type inCR struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   inSpec  `json:"spec,omitempty"`  // 这里是不同之处
    Status InsightTargetStatus `json:"status,omitempty"`
}

// 这是在我们不拥有的其他程序中定义的,因此无法“重用”

type Bar struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   extSpec  `json:"spec,omitempty"`    // 这里是不同之处
    Status ServiceStatus `json:"status,omitempty"`
}

inSpec具有以下字段extSpec的子集

type inSpec struct {
    name string
    age  int
}

extSpec具有这些字段和许多其他不相关的字段

type extSpec struct {
    name string   
    age  int
    foo string  // 不相关
    bar string  // 不相关
    bazz string // 不相关
}

最后在reconcile中我需要将"relevant"字段移动到某些函数中完全相同的函数有时从extCR获取字段有时从inCR获取字段根据发生的事件如更新extCR或由用户更新inCR)。

**更新2**

```go
func sharedLogic(r reconciler, ctx context.Context, c client.Object) (ctrl.Result, error) {
    in := c.(*inCR)
    vPass, e := vps.Get(ctx, r.Client, in.Spec.foo, in.Spec.bar)
    return ctrl.Result{}, nil
}

但是对于extCR我应该执行以下操作

func sharedLogic(r reconciler, ctx context.Context, c client.Object) (ctrl.Result, error) {
    ext := c.(*extCR)
    vPass, e := vps.Get(ctx, r.Client, ext.Spec.val.foo, ext.Spec.val.bar)
    return ctrl.Result{}, nil
}
英文:

I’ve k8s operator which works as expected, I need to add a “watch” to other operator CRD (not mine), to make it simple lets call it extCR and our operator cr called inCR,

I tried the following but there is an issue how its right to trigger the reconcile.

func (r *Insiconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&Inv1alpha1.Iget{}}).
Watches(&source.Kind{Type: &ext.Se{}},  handler.EnqueueRequestsFromMapFunc(r.FWatch)).
Complete(r)
}
func (r *Insiconciler) FWatch(c client.Object) []reconcile.Request {
val := c.(*ivi.Srv)
req := reconcile.Request{NamespacedName: types.NamespacedName{Name: val.Name, Namespace: val.Namespace}}
return []reconcile.Request{req}
}

The problem here that I trigger the reconcile with the extCR , I want inside the FWatch to update the inCR and start the reconcile with inCR and not with extCR, how can I do it ?

I mean, to avoid something like the following code as sometimes the reconcile is done for the inCR and sometimes for the extCR and than I can get some ugly if's

func (r *Insiconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
var inCR FOO
var extCR BAR
if err := r.Get(ctx, req.NamespacedName, &inCR); err != nil {
return ctrl.Result{}, err
}
if err := r.Get(ctx, req.NamespacedName, &extCR); err != nil {
return ctrl.Result{}, err
}

I want to know what is the right/clean way to handle such case

case when you need to listen to externalCR (not part of your controller) and also internalCR (from your controller) .

One more thing - the CR are different GVK but the exteranlCR contain lot of fields which is not required, just some of them. but the required fields is having the same names on both cr's

update

type inCR struct {
metav1.TypeMeta   `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec   inSpec  `json:"spec,omitempty"`  / / ————————here is the difference 
Status InsightTargetStatus `json:"status,omitempty"`
}

//————— This is defined on other program which is not owned by us, therefore cannot “reuse”

type Bar struct {
metav1.TypeMeta   `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec   extSpec  `json:"spec,omitempty"`    // ———————here is the difference 
Status ServiceStatus `json:"status,omitempty"`
}

And inSpec is having the following fields (subset of extSpec)

type inSpec struct {
name string
age  int
}

and extSpec have those fields and many more which is not related

type extSpec struct {
name string   
age  int
foo string  // not relevant
bar string  // not relevant
bazz string // not relevant
}

at the end, Inside the reconcile I need to move the relevant fields to some functions. exactly same functions just take sometime the fields from extCR and sometimes for inCR, according to the event that happens (like updating the extCR or update the inCR by users )

Update2

func sharedLogic(r reconciler, ctx context.Context, c client.Object) (ctrl.Result, error) {
in := c.(*inCR)
vPass , e := vps.Get(ctx, r.Client, in.Spec.foo, in.Spec.bar)
return ctrl.Result{}, nil
}
But for extCR I should do the following
func sharedLogic(r reconciler, ctx context.Context, c client.Object) (ctrl.Result, error) {
ext := c.(*extCR)
vPass , e := vps.Get(ctx, r.Client, ext.Spec.val.foo, ext.Spec.val.bar)
return ctrl.Result{}, nil
}

答案1

得分: 2

需要注意的几点:

  • 每个控制器只负责一个资源。
  • 调和请求包含调和 Kubernetes 对象所需的信息。这包括唯一标识对象的信息 - 名称和命名空间。它不包含任何特定事件或对象内容的信息。

你可以创建一个没有资源定义的第二个控制器。在你的主文件中,两个控制器都将被注册。

如果 CRD 完全不相关,或者外部资源引用了内部资源,那么在外部调和器中可以对内部资源进行更改。

kubebuilder create api --group other --version v2 --kind External \
 --resource=false --controller=true

这将给你一个带有以下 SetupWithManager 方法的控制器。

func (r *ExternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		// Uncomment the following line adding a pointer to an instance of the controlled resource as an argument
		// For().
		Complete(r)
}

注意,For 方法被注释掉了,因为你需要从其他地方导入要监视的资源并引用它。

import (
	...
	otherv2 "other.io/external/api/v2"
)
...
func (r *ExternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&otherv2.External{}).
        Complete(r)
}

如果你无法导入外部资源,可以自己模拟它,但这可能不是一个很好的方法。你应该尝试从其他控制器项目中导入它。

kubebuilder edit --multigroup=true
kubebuilder create api --group=other --version v2 --kind External \
  --resource --controller

另一种方法是,当资源彼此相关时,内部资源在其规范中引用外部资源,并且在调和时知道如何获取外部资源。可以在这里找到一个示例:https://book.kubebuilder.io/reference/watching-resources/externally-managed.html

type InternalSpec struct {
    // 外部资源的名称
    ExternalResource string `json:"externalResource,omitempty"`
}

这意味着在每个调和循环中,控制器将查找外部资源并使用它来管理内部资源。

func (r *InternalReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	internal := examplev1.Internal{}
	if err := r.Get(context.TODO(), types.NamespacedName{
		Name:      req.Name,
		Namespace: req.Namespace,
	}, &internal); err != nil {
		return ctrl.Result{}, err
	}

	external := otherv2.External{}
	if err := r.Get(context.TODO(), types.NamespacedName{
        // 注意,名称是从内部规范中获取的
		Name:      internal.Spec.ExternalResource,
		Namespace: req.Namespace,
	}, &internal); err != nil {
		return ctrl.Result{}, err
	}

	// 在这里处理内部和外部资源

	return ctrl.Result{}, nil
}

这种方法的问题是,当内部资源不发生更改时,不会触发调和事件,即使外部资源发生了更改。为了解决这个问题,我们可以通过监视外部资源来触发调和。注意 Watches 方法:

func (r *InternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&examplev1.Main{}).
		Watches(
			&source.Kind{Type: &otherv2.ExternalResource{}},
			handler.EnqueueRequestsFromMapFunc(r.triggerReconcileBecauseExternalHasChanged),
			builder.WithPredicates(predicate.ResourceVersionChangedPredicate{}),
		).
		Complete(r)
}

为了知道我们应该触发哪个内部对象的事件,我们使用映射函数来查找所有具有对外部资源的引用的内部对象。

func (r *InternalReconciler) triggerReconcileBecauseExternalHasChanged(o client.Object) []reconcile.Request {
	usedByInternals := &examplev1.InternalList{}
	listOps := &client.ListOptions{
		FieldSelector: fields.OneTermEqualSelector(".spec.ExternalResource", o.GetName()),
		Namespace:     o.GetNamespace(),
	}
	err := r.List(context.TODO(), usedByInternals, listOps)
	if err != nil {
		return []reconcile.Request{}
	}
	requests := make([]reconcile.Request, len(usedByInternals.Items))
	for i, item := range usedByInternals.Items {
		requests[i] = reconcile.Request{
			NamespacedName: types.NamespacedName{
				Name:      item.GetName(),
				Namespace: item.GetNamespace(),
			},
		}
	}
	return requests
}

由于你更新了问题,我建议按照以下方式进行操作。

我正在创建一个新项目和两个控制器。注意第二个控制器命令中没有创建资源,只创建了控制器。这是因为控制器将监视外部资源。

mkdir demo && cd demo
go mod init example.io/demo
kubebuilder init --domain example.io --repo example.io/demo --plugins=go/v4-alpha
kubebuilder create api --group=demo --version v1 --kind Internal --controller --resource
kubebuilder create api --group=other --version v2 --kind External --controller --resource=false
$ tree controllers
controllers
├── external_controller.go
├── internal_controller.go
└── suite_test.go

现在我们需要一些共享逻辑,例如将其添加到控制器包中。我们将从两个调和器中调用它。

// 接口可能需要调整
// 取决于你想在调和器中做什么
type reconciler interface {
 client.Reader
 client.Writer
 client.StatusClient
}

func sharedLogic(r reconciler, kobj *demov1.Internal) (ctrl.Result, error) {
 // 在这里执行共享逻辑,操作内部对象结构
 // 这样做是可行的,因为外部控制器将调用此函数并传递内部对象
 return ctrl.Result{}, nil
}

这是内部调和器的示例。

func (r *InternalReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 _ = log.FromContext(ctx)
 obj := demov1.Internal{}
 if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
  return ctrl.Result{}, err
 }
 return sharedLogic(r, &obj)
}

外部调和器也是一样。

func (r *ExternalReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 _ = log.FromContext(ctx)
 // 注意,我们可以在这里使用内部对象,只要外部对象包含我们需要的相同字段即可。这意味着在解组时,额外的字段会被丢弃。如果无法这样做,你可以先解组为外部资源,然后将需要的字段分配给内部资源,然后再传递下去
 obj := demov1.Internal{}
 if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
  return ctrl.Result{}, err
 }
 return sharedLogic(r, &obj)
}

func (r *ExternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
 return ctrl.NewControllerManagedBy(mgr).
 // 注意,外部资源是从另一个项目导入的
 // 你可以通过创建具有正确 GKV 的最小类型来在不导入的情况下监视它
  For(otherv2.External{}).
  Complete(r)
}
英文:

Few things to keep in mind:

  • Each controller is responsible for exactly one resource.
  • Reconcile request contains the information necessary to reconcile a Kubernetes object. This includes the information to uniquely identify the object - its Name and Namespace. It does NOT contain information about any specific Event or the object contents itself.

You can create a second controller without the resource definition. In your main file, both controllers will be registered.

This could be useful if the CRDs are not related at all or if the external resource references the internal one, so you can make changes to the internal resource in the external reconciler.

kubebuilder create api --group other --version v2 --kind External \
 --resource=false --controller=true

This gives you a controller with a SetupWithManager method that looks like the below.

func (r *ExternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		// Uncomment the following line adding a pointer to an instance of the controlled resource as an argument
		// For().
		Complete(r)
}

Note how the For method is commented out because you need to import the resource to watch from somewhere else and reference it.

import (
	...
	otherv2 "other.io/external/api/v2"
)
...
func (r *ExternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&otherv2.External{}).
        Complete(r)
}

If you cannot import the external resource you could fall back to mocking it yourself but this is probably not a very clean way. You should really try to import it from the other controller project.

kubebuilder edit --multigroup=true
kubebuilder create api --group=other --version v2 --kind External \
  --resource --controller

Another way is when the resources are related to each other such that the internal resource has a reference in its spec to the external resource and knows how to get the external resource in its spec, when it reconciles. An example of this can be found here https://book.kubebuilder.io/reference/watching-resources/externally-managed.html

type InternalSpec struct {
    // Name of an external resource
    ExternalResource string `json:"externalResource,omitempty"`
}

This means that in each reconciliation loop, the controller will look up the external resource and use it to manage the internal resource.

func (r *InternalReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)

	internal := examplev1.Internal{}
	if err := r.Get(context.TODO(), types.NamespacedName{
		Name:      req.Name,
		Namespace: req.Namespace,
	}, &internal); err != nil {
		return ctrl.Result{}, err
	}

	external := otherv2.External{}
	if err := r.Get(context.TODO(), types.NamespacedName{
        // note how the name is taken from the internal spec
		Name:      internal.Spec.ExternalResource,
		Namespace: req.Namespace,
	}, &internal); err != nil {
		return ctrl.Result{}, err
	}

	// do something with internal and external here

	return ctrl.Result{}, nil
}

The problem with this is, that when the internal resource does not change, no reconciliation event will be triggered, even when the external resource has changed. To work around that, we can trigger the reconciliation by watching the external resource. Note the Watches method:

func (r *InternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&examplev1.Main{}).
		Watches(
			&source.Kind{Type: &otherv2.ExternalResource{}},
			handler.EnqueueRequestsFromMapFunc(r.triggerReconcileBecauseExternalHasChanged),
			builder.WithPredicates(predicate.ResourceVersionChangedPredicate{}),
		).
		Complete(r)
}

In order to know for which internal object we should trigger an event, we use a mapping function to look up all the internal that have a reference to the external resource.

func (r *InternalReconciler) triggerReconcileBecauseExternalHasChanged(o client.Object) []reconcile.Request {
	usedByInternals := &examplev1.InternalList{}
	listOps := &client.ListOptions{
		FieldSelector: fields.OneTermEqualSelector(".spec.ExternalResource", o.GetName()),
		Namespace:     o.GetNamespace(),
	}
	err := r.List(context.TODO(), usedByInternals, listOps)
	if err != nil {
		return []reconcile.Request{}
	}
	requests := make([]reconcile.Request, len(usedByInternals.Items))
	for i, item := range usedByInternals.Items {
		requests[i] = reconcile.Request{
			NamespacedName: types.NamespacedName{
				Name:      item.GetName(),
				Namespace: item.GetNamespace(),
			},
		}
	}
	return requests
}

Since you updated your question, I suggest doing something like below.

I am creating a new project and 2 controllers. Note on the second controller command no resource is created along with the controller. this is because the controller
will watch an external resource.

mkdir demo && cd demo
go mod init example.io/demo
kubebuilder init --domain example.io --repo example.io/demo --plugins=go/v4-alpha
kubebuilder create api --group=demo --version v1 --kind Internal --controller --resource
kubebuilder create api --group=other --version v2 --kind External --controller --resource=false
$ tree controllers
controllers
├── external_controller.go
├── internal_controller.go
└── suite_test.go

Now we need some shared logic, for example by adding this to the controllers package. We will call this from both reconcilers.

// the interface may need tweaking
// depending on what you want to do with
// the reconiler
type reconciler interface {
 client.Reader
 client.Writer
 client.StatusClient
}

func sharedLogic(r reconciler, kobj *demov1.Internal) (ctrl.Result, error) {
 // do your shared logic here operating on the internal object struct
 // this works out because the external controller will call this passing the
 // internal object
 return ctrl.Result{}, nil
}

Here is an example for the internal reconciler.

func (r *InternalReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 _ = log.FromContext(ctx)
 obj := demov1.Internal{}
 if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
  return ctrl.Result{}, err
 }
 return sharedLogic(r, &obj)
}

And in the external reconciler we do the same.

func (r *ExternalReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 _ = log.FromContext(ctx)
 // note, we can use the internal object here as long as the external object
 // does contain the same fields we want. That means when unmarshalling the extra
 // fields are dropped. If this cannot be done, you could first unmarshal into the external
 // resource and then assign the fields you need to the internal one, before passing it down
 obj := demov1.Internal{}
 if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
  return ctrl.Result{}, err
 }
 return sharedLogic(r, &obj)
}

func (r *ExternalReconciler) SetupWithManager(mgr ctrl.Manager) error {
 return ctrl.NewControllerManagedBy(mgr).
 // note the external resource is imported from another project
 // you may be able to watch this without import by creating a minimal
 // type with the right GKV
  For(otherv2.External{}).
  Complete(r)
}

huangapple
  • 本文由 发表于 2022年10月31日 17:22:34
  • 转载请务必保留本文链接:https://go.coder-hub.com/74261141.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定