Got "ERROR:root:Failed to get healthz info attempt 1 of 5." when deploy pipeline in the Kubeflow

huangapple
go评论128阅读模式
英文:

Got "ERROR:root:Failed to get healthz info attempt 1 of 5." when deploy pipeline in the Kubeflow

问题

我正在尝试按照Kubeflow v2快速入门指南进行操作。

首先，我使用以下命令将Kubeflow部署到本地Kubernetes集群中：

  1. export PIPELINE_VERSION="2.0.0-alpha.4"
  3. kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
  4. kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
  5. kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"

然后，我进行了端口转发操作：

  1. kubectl port-forward service/ml-pipeline-ui --namespace=kubeflow 38620:80

我可以在http://localhost:38620看到UI。

接下来，我安装了kfp 2.0.1，并且这是我的代码：

  1. from kfp import client, dsl
  3. @dsl.component
  4. def addition_component(num1: int, num2: int) -> int:
  5.     return num1 + num2
  7. @dsl.pipeline(name="addition-pipeline")
  8. def my_pipeline(a: int, b: int, c: int = 10):
  9.     add_task_1 = addition_component(num1=a, num2=b)
  10.     add_task_2 = addition_component(num1=add_task_1.output, num2=c)
  12. endpoint = "http://localhost:38620"  # <- 不确定是否正确，因为在快速入门文档中缺少此部分。
  13. kfp_client = client.Client(host=endpoint)
  14. run = kfp_client.create_run_from_pipeline_func(
  15.     my_pipeline,
  16.     arguments={"a": 1, "b": 2},
  17. )
  18. url = f"{endpoint}/#/runs/details/{run.run_id}"
  19. print(url)

然而，我遇到了错误：

  1. python src/main.py

我在endpoint中使用的是http://localhost:38620，但是我不确定是否正确，因为在快速入门文档中缺少此部分。

我尝试访问以下地址：

  • http://localhost:38620/apis/v2beta1/healthz
  • http://localhost:38620/#/apis/v2beta1/healthz

但它们都不存在。

我在https://github.com/kubeflow/kubeflow/issues/5989 上找到了类似的问题，但里面没有有用的信息。

感谢任何指导！

英文:

I am trying to follow the Kubeflow v2 quickstart.

First, I deployed Kubeflow to a local Kubernetes cluster by

  1. export PIPELINE_VERSION=&quot;2.0.0-alpha.4&quot;
  3. kubectl apply -k &quot;github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION&quot;
  4. kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
  5. kubectl apply -k &quot;github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION&quot;

I port forwarded by

  1. kubectl port-forward service/ml-pipeline-ui --namespace=kubeflow 38620:80

I can see the UI at http://localhost:38620

Got "ERROR:root:Failed to get healthz info attempt 1 of 5." when deploy pipeline in the Kubeflow

Next, I installed kfp 2.0.1. And here is my code:

  1. from kfp import client, dsl
  4. @dsl.component
  5. def addition_component(num1: int, num2: int) -&gt; int:
  6.     return num1 + num2
  9. @dsl.pipeline(name=&quot;addition-pipeline&quot;)
  10. def my_pipeline(a: int, b: int, c: int = 10):
  11.     add_task_1 = addition_component(num1=a, num2=b)
  12.     add_task_2 = addition_component(num1=add_task_1.output, num2=c)
  15. endpoint = &quot;http://localhost:38620&quot;  # &lt;- Not entirely sure if it is correct as it is missing in the quickstart document.
  16. kfp_client = client.Client(host=endpoint)
  17. run = kfp_client.create_run_from_pipeline_func(
  18.     my_pipeline,
  19.     arguments={&quot;a&quot;: 1, &quot;b&quot;: 2},
  20. )
  21. url = f&quot;{endpoint}/#/runs/details/{run.run_id}&quot;
  22. print(url)

However, I got error

  1. python src/main.py
  3. /Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp/client/client.py:158: FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.
  4.   warnings.warn(
  5. ERROR:root:Failed to get healthz info attempt 1 of 5.
  6. Traceback (most recent call last):
  7.   File &quot;/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp/client/client.py&quot;, line 435, in get_kfp_healthz
  8.     return self._healthz_api.get_healthz()
  9.            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  10.   File &quot;/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api/healthz_service_api.py&quot;, line 63, in get_healthz
  11.     return self.get_healthz_with_http_info(**kwargs)  # noqa: E501
  12.            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  13.   File &quot;/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api/healthz_service_api.py&quot;, line 134, in get_healthz_with_http_info
  14.     return self.api_client.call_api(
  15.            ^^^^^^^^^^^^^^^^^^^^^^^^^
  16.   File &quot;/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api_client.py&quot;, line 364, in call_api
  17.     return self.__call_api(resource_path, method,
  18.            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  19.   File &quot;/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api_client.py&quot;, line 188, in __call_api
  20.     raise e
  21.   File &quot;/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api_client.py&quot;, line 181, in __call_api
  22.     response_data = self.request(
  23.                     ^^^^^^^^^^^^^
  24.   File &quot;/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api_client.py&quot;, line 389, in request
  25.     return self.rest_client.GET(url,
  26.            ^^^^^^^^^^^^^^^^^^^^^^^^^
  27.   File &quot;/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/rest.py&quot;, line 230, in GET
  28.     return self.request(&quot;GET&quot;, url,
  29.            ^^^^^^^^^^^^^^^^^^^^^^^^
  30.   File &quot;/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/rest.py&quot;, line 224, in request
  31.     raise ApiException(http_resp=r)
  32. kfp_server_api.exceptions.ApiException: (404)
  33. Reason: Not Found
  34. HTTP response headers: HTTPHeaderDict({&#39;X-Powered-By&#39;: &#39;Express&#39;, &#39;Content-Security-Policy&#39;: &quot;default-src &#39;none&#39;&quot;, &#39;X-Content-Type-Options&#39;: &#39;nosniff&#39;, &#39;Content-Type&#39;: &#39;text/html; charset=utf-8&#39;, &#39;Content-Length&#39;: &#39;159&#39;, &#39;Date&#39;: &#39;Mon, 03 Jul 2023 03:59:33 GMT&#39;, &#39;Connection&#39;: &#39;keep-alive&#39;, &#39;Keep-Alive&#39;: &#39;timeout=5&#39;})
  35. HTTP response body: &lt;!DOCTYPE html&gt;
  36. &lt;html lang=&quot;en&quot;&gt;
  37. &lt;head&gt;
  38. &lt;meta charset=&quot;utf-8&quot;&gt;
  39. &lt;title&gt;Error&lt;/title&gt;
  40. &lt;/head&gt;
  41. &lt;body&gt;
  42. &lt;pre&gt;Cannot GET /apis/v2beta1/healthz&lt;/pre&gt;
  43. &lt;/body&gt;
  44. &lt;/html&gt;

Inside for endpoint, I am using http://localhost:38620, However, I am not entirely sure if I am using correct one as it is missing in quickstart document.

I tried to access

  • http://localhost:38620/apis/v2beta1/healthz
  • http://localhost:38620/#/apis/v2beta1/healthz

and they do not exist.

I found a similar issue at https://github.com/kubeflow/kubeflow/issues/5989 but there is not any helpful info inside.

Any guide would be appreciate!

答案1

得分: 1

UPDATE: 我已经提交了一个拉取请求来更新文档。今天应该很好。

我仔细查看了Python脚本日志。它有一行：

> 未来警告：此客户端仅与Kubeflow Pipeline v2.0.0-beta.2及更高版本兼容。

原来Kubeflow v2快速入门已过时。

在我将2.0.0-alpha.4替换为2.0.0之后（您可以在https://github.com/kubeflow/pipelines/releases?q=Version找到最新版本），然后重新部署：

  1. export PIPELINE_VERSION="2.0.0"
  3. kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
  4. kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
  5. kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"

现在流水线脚本成功执行：

  1. python src/main.py
  3. /Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp/client/client.py:158: 未来警告：此客户端仅与Kubeflow Pipeline v2.0.0-beta.2及更高版本兼容。
  4.   警告(
  5. 实验详情：http://localhost:38620/#/experiments/details/c9d82bc3-712e-422a-804f-fcbe5d2b8acb
  6. 运行详情：http://localhost:38620/#/runs/details/87398dae-6488-446b-b650-7bbc580aab56
  7. http://localhost:38620/#/runs/details/87398dae-6488-446b-b650-7bbc580aab56

健康端点http://localhost:38620/apis/v2beta1/healthz也返回：

  1. {
  2.   "buildDate": "Tue Jun 20 16:56:00 UTC 2023",
  3.   "frontendCommitHash": "e03e31219387b587b700ba3e31a02df486aa364f",
  4.   "frontendTagName": "2.0.0",
  5.   "apiServerReady": true,
  6.   "apiServerCommitHash": "e03e31219387b587b700ba3e31a02df486aa364f",
  7.   "apiServerTagName": "2.0.0",
  8.   "apiServerMultiUser": false,
  9.   "multi_user": false
  10. }

Got "ERROR:root:Failed to get healthz info attempt 1 of 5." when deploy pipeline in the Kubeflow

英文:

UPDATE: I have opened a pull request to update the document. Today should be good.

I had a closer look of the Python script log. It has a line:

> FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.

It turns out the Kubeflow v2 quickstart is out-of-date.

After I replaced 2.0.0-alpha.4 to 2.0.0 (You can find latest version at https://github.com/kubeflow/pipelines/releases?q=Version) and redeploy by

  1. export PIPELINE_VERSION=&quot;2.0.0&quot;
  3. kubectl apply -k &quot;github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION&quot;
  4. kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
  5. kubectl apply -k &quot;github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION&quot;

Now the pipeline script succeed:

  1. python src/main.py
  3. /Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp/client/client.py:158: FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.
  4.   warnings.warn(
  5. Experiment details: http://localhost:38620/#/experiments/details/c9d82bc3-712e-422a-804f-fcbe5d2b8acb
  6. Run details: http://localhost:38620/#/runs/details/87398dae-6488-446b-b650-7bbc580aab56
  7. http://localhost:38620/#/runs/details/87398dae-6488-446b-b650-7bbc580aab56

Got "ERROR:root:Failed to get healthz info attempt 1 of 5." when deploy pipeline in the Kubeflow

The health endpoint http://localhost:38620/apis/v2beta1/healthz also returns

  1. {
  2.   &quot;buildDate&quot;: &quot;Tue Jun 20 16:56:00 UTC 2023&quot;,
  3.   &quot;frontendCommitHash&quot;: &quot;e03e31219387b587b700ba3e31a02df486aa364f&quot;,
  4.   &quot;frontendTagName&quot;: &quot;2.0.0&quot;,
  5.   &quot;apiServerReady&quot;: true,
  6.   &quot;apiServerCommitHash&quot;: &quot;e03e31219387b587b700ba3e31a02df486aa364f&quot;,
  7.   &quot;apiServerTagName&quot;: &quot;2.0.0&quot;,
  8.   &quot;apiServerMultiUser&quot;: false,
  9.   &quot;multi_user&quot;: false
  10. }

huangapple
  • 本文由 发表于 2023年7月3日 13:15:02
  • 转载请务必保留本文链接：https://go.coder-hub.com/76601980.html
  • kubeflow
  • kubernetes
  • python
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定