英文:
Running apache beam spark runner failed with java.util.concurrent.TimeoutException
问题
我正在尝试使用Apache Beam Spark Runner运行Apache Beam Go SDK的成绩示例。我使用以下命令从Apache Beam Go源代码启动预部署的Spark集群上的JobService端点：./gradlew :runners:spark:job-server:runShadow -PsparkMasterUrl=spark://localhost:7077，然后我按如下方式提交作业：grades =runner=spark -endpoint=host:port -job_name=xxxx我遇到了以下错误：
: 运行Apache Beam Spark Runner失败，错误为java.util.concurrent.TimeoutException: Timed out while waiting for command 'docker run -d --network=host --env=DOCKER_MAC_CONTAINER=null
apache/beam_go_sdk:latest --id=4-1 --provision_endpoint=localhost:*****
此外，Docker正在运行，并且已安装了所需的依赖项。
我还发现了以下错误：
Caused by: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.util.concurrent.TimeoutException: Timed out while waiting for command 'docker run -d --network=host --env=DOCKER_MAC_CONTAINER=null apache/beam_go_sdk:latest --id=2-1 --provision_endpoint=localhost:49392'
我该如何继续进行？
英文:
I'm trying to run grades example of apache beam go sdk using apache beam spark runner.
I start the JobService endpoint on a pre-deployed spark cluster from apache beam go source using the following command: ./gradlew :runners:spark:job-server:runShadow -PsparkMasterUrl=spark://localhost:7077 and,
i submit the job as follows: grades =runner=spark -endpoint=host:port -job_name=xxxx
I got the following error :
: Ruuning apache beam spark runner failed with java.util.concurrent.TimeoutException: Timed out while waiting for command 'docker run -d --network=host --env=DOCKER_MAC_CONTAINER=null
apache/beam_go_sdk:latest --id=4-1 --provision_endpoint=localhost:*****
In qddition, docker is running and the dependencies are installed.
I found the following error also
Caused by: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: java.util.concurrent.TimeoutException: Timed out while waiting for command 'docker run -d --network=host --env=DOCKER_MAC_CONTAINER=null apache/beam_go_sdk:latest --id=2-1 --provision_endpoint=localhost:49392'
How can i proceed?
答案1
得分: 1
看起来你的作业服务器无法找到Go SDK的Docker镜像。以下是一些选项：
本地镜像
你可以在本地机器上获取apache/beam_go_sdk:latest镜像。你可以使用以下命令从Docker Hub拉取容器。
docker pull apache/beam_go_sdk:latest
或者，你可以使用以下命令从Apache Beam存储库的根目录自行构建它。
./gradlew :sdks:go:container:docker
Loopback
Loopback模式是一种快捷方式，用于在本地运行用于测试目的的流水线。当你启用loopback模式时，作业服务器不会启动Docker容器来运行SDK代码，而是将请求循环回最初运行的进程，以创建和提交流水线。这样你就不需要依赖Docker。
在运行Beam流水线时，你可以通过添加流水线选项--environment_type=LOOPBACK来启用loopback模式。
英文:
It looks like your job server is unable to find the Docker image for the Go SDK. Here are some options:
Local Image
You can get the apache/beam_go_sdk:latest image on your local machine. You can pull the container from Docker Hub with the following command.
docker pull apache/beam_go_sdk:latest
Alternatively, you can build it yourself with the following command from the root directory of the Apache Beam repository.
./gradlew :sdks:go:container:docker
Loopback
Loopback mode is a shortcut to aid running pipelines locally for testing purposes. When you enable loopback mode, instead of the job server spinning up a Docker container to run the SDK code, it will instead loop requests back to the process you originally ran to create and submit the pipeline. This way you don't rely on Docker at all.
You can enable loopback mode by adding the pipeline option --environment_type=LOOPBACK when running your Beam pipeline.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论