Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client

huangapple go评论69阅读模式
英文:

Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client

问题

我正在尝试从K8s控制器访问我的ETCD数据库,但在尝试打开ETCD客户端时遇到了rpc错误/EOF。

我的设置:

  • ETCD服务部署在我的K8s集群中,并包含在我的Istio服务网格中(其DNS记录为my-etcd-cluster.my-etcd-namespace.svc.cluster.local
  • 我有一个使用Kubebuilder框架开发的自定义K8s控制器,并部署在同一集群的不同命名空间中,但配置为是相同Istio服务网格的一部分
  • 我正在尝试使用Go客户端SDK库连接到控制器中的ETCD数据库

这是我受影响的Go代码:

cli, err := clientv3.New(clientv3.Config{
    Endpoints:   []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
    DialTimeout: 5 * time.Second,
    Username:    username,
    Password:    password,
})

if err != nil {
    return nil, fmt.Errorf("opening ETCD client failed: %v", err)
}

这是当执行clientv3.New(...)时我遇到的错误:

{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
...
1.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
1.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227

当我传递一些虚拟的无效凭据时,也会出现相同的错误。

然而,当我尝试以HTTP API的方式访问数据库时:

postBody, _ := json.Marshal(map[string]string{
    "name":     username,
    "password": password,
})
responseBody := bytes.NewBuffer(postBody)

resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
if err != nil {
    return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
}
l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
defer resp.Body.Close()

...我得到了200 OK和一个正确的令牌(这是预期的),所以我相信我的Istio配置是正确的,我的控制器应该能够看到ETCD数据库服务。我不知道为什么在使用客户端SDK方法时会出现问题。

当我使用ETCD服务的端口转发并在本地访问时,clientv3.New()和其他客户端SDK方法都能正常工作。我错过了什么?我真的很感谢任何建议。

编辑:
我还添加了一个简单的Pod来尝试通过etcdctl访问我的etcd数据库:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: my-controller-namespace
spec:
  containers:
  - name: etcdctl
    image: bitnami/etcd
    command:
    - sleep
    - infinity

当通过kubectl exec登录到容器时,我能够访问我的数据库:

$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user="user" --password="password" put foo bob
OK

我猜问题出在SDK中?

英文:

I'm trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.

My setup:

  • ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record: my-etcd-cluster.my-etcd-namespace.svc.cluster.local)
  • I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh
  • I'm trying to connect to ETCD database from the controller, using Go client SDK library for ETCD

Here's my affected Go code:

cli, err := clientv3.New(clientv3.Config{
    Endpoints:   []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
    DialTimeout: 5 * time.Second,
    Username:    username,
    Password:    password,
})

if err != nil {
    return nil, fmt.Errorf("opening ETCD client failed: %v", err)
}

And here's an error I'm getting when clientv3.New(...) gets executed:

{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
...
1.647473862175209e+09   INFO    controller.etcdclient   Finish reconcile loop for some-service/test-svc-client  {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
1.6474738621752858e+09  ERROR   controller.etcdclient   Reconciler error        {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227

The same error happens when I'm passing some dummy, invalid credentials.

However, when I tried to access the database in a HTTP API manner:

postBody, _ := json.Marshal(map[string]string{
    "name":     username,
    "password": password,
})
responseBody := bytes.NewBuffer(postBody)

resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
if err != nil {
    return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
}
l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
defer resp.Body.Close()

...I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn't work when following the client SDK approach.

When I'm using port-forwarding of the ETCD service and accessing it locally, clientv3.New() and other client SDK methods work like a charm. What am I missing? I'd really appreciate any suggestions.

EDIT:
I've also added a simple pod to try accessing my etcd db via etcdctl:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: my-controller-namespace
spec:
  containers:
  - name: etcdctl
    image: bitnami/etcd
    command:
    - sleep
    - infinity

When logged into the container via kubectl exec, I was able to access my db:

$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user="user" --password="password" put foo bob
OK

I guess the problem is somewhere in the SDK?

答案1

得分: 1

原来是版本不匹配的问题 - 我的 ETCD 数据库是 v3.5.2,而我使用的 clientv3 库是 v3.5.0。

根据 ETCD 的变更日志(https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md)所示:

Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client

英文:

Turned out to be version mismatch - my ETCD db is v3.5.2 and the clientv3 library that I used was v3.5.0.
As seen in ETCD changelog (https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md):

Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client

huangapple
  • 本文由 发表于 2022年3月17日 16:45:08
  • 转载请务必保留本文链接:https://go.coder-hub.com/71509351.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定