英文:
Getting "rpc error: code = Unavailable desc = error reading from server: EOF" when trying to create a new etcdv3 client
问题
我正在尝试从K8s控制器访问我的ETCD数据库,但在尝试打开ETCD客户端时遇到了rpc错误/EOF。
我的设置:
- ETCD服务部署在我的K8s集群中,并包含在我的Istio服务网格中(其DNS记录为
my-etcd-cluster.my-etcd-namespace.svc.cluster.local
) - 我有一个使用Kubebuilder框架开发的自定义K8s控制器,并部署在同一集群的不同命名空间中,但配置为是相同Istio服务网格的一部分
- 我正在尝试使用Go客户端SDK库连接到控制器中的ETCD数据库
这是我受影响的Go代码:
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
DialTimeout: 5 * time.Second,
Username: username,
Password: password,
})
if err != nil {
return nil, fmt.Errorf("opening ETCD client failed: %v", err)
}
这是当执行clientv3.New(...)
时我遇到的错误:
{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
...
1.647473862175209e+09 INFO controller.etcdclient Finish reconcile loop for some-service/test-svc-client {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
1.6474738621752858e+09 ERROR controller.etcdclient Reconciler error {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
当我传递一些虚拟的无效凭据时,也会出现相同的错误。
然而,当我尝试以HTTP API的方式访问数据库时:
postBody, _ := json.Marshal(map[string]string{
"name": username,
"password": password,
})
responseBody := bytes.NewBuffer(postBody)
resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
if err != nil {
return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
}
l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
defer resp.Body.Close()
...我得到了200 OK和一个正确的令牌(这是预期的),所以我相信我的Istio配置是正确的,我的控制器应该能够看到ETCD数据库服务。我不知道为什么在使用客户端SDK方法时会出现问题。
当我使用ETCD服务的端口转发并在本地访问时,clientv3.New()
和其他客户端SDK方法都能正常工作。我错过了什么?我真的很感谢任何建议。
编辑:
我还添加了一个简单的Pod来尝试通过etcdctl访问我的etcd数据库:
apiVersion: v1
kind: Pod
metadata:
name: test-pod
namespace: my-controller-namespace
spec:
containers:
- name: etcdctl
image: bitnami/etcd
command:
- sleep
- infinity
当通过kubectl exec
登录到容器时,我能够访问我的数据库:
$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user="user" --password="password" put foo bob
OK
我猜问题出在SDK中?
英文:
I'm trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.
My setup:
- ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record:
my-etcd-cluster.my-etcd-namespace.svc.cluster.local
) - I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh
- I'm trying to connect to ETCD database from the controller, using Go client SDK library for ETCD
Here's my affected Go code:
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
DialTimeout: 5 * time.Second,
Username: username,
Password: password,
})
if err != nil {
return nil, fmt.Errorf("opening ETCD client failed: %v", err)
}
And here's an error I'm getting when clientv3.New(...)
gets executed:
{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3@v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
...
1.647473862175209e+09 INFO controller.etcdclient Finish reconcile loop for some-service/test-svc-client {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
1.6474738621752858e+09 ERROR controller.etcdclient Reconciler error {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227
The same error happens when I'm passing some dummy, invalid credentials.
However, when I tried to access the database in a HTTP API manner:
postBody, _ := json.Marshal(map[string]string{
"name": username,
"password": password,
})
responseBody := bytes.NewBuffer(postBody)
resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
if err != nil {
return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
}
l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
defer resp.Body.Close()
...I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn't work when following the client SDK approach.
When I'm using port-forwarding of the ETCD service and accessing it locally, clientv3.New()
and other client SDK methods work like a charm. What am I missing? I'd really appreciate any suggestions.
EDIT:
I've also added a simple pod to try accessing my etcd db via etcdctl:
apiVersion: v1
kind: Pod
metadata:
name: test-pod
namespace: my-controller-namespace
spec:
containers:
- name: etcdctl
image: bitnami/etcd
command:
- sleep
- infinity
When logged into the container via kubectl exec
, I was able to access my db:
$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user="user" --password="password" put foo bob
OK
I guess the problem is somewhere in the SDK?
答案1
得分: 1
原来是版本不匹配的问题 - 我的 ETCD 数据库是 v3.5.2,而我使用的 clientv3 库是 v3.5.0。
根据 ETCD 的变更日志(https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md)所示:
英文:
Turned out to be version mismatch - my ETCD db is v3.5.2 and the clientv3 library that I used was v3.5.0.
As seen in ETCD changelog (https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md):
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论