(Apache Beam) Cannot increase executor memory – it is fixed at 1024M despite using multiple settings

huangapple go评论89阅读模式
英文:

(Apache Beam) Cannot increase executor memory - it is fixed at 1024M despite using multiple settings

问题

我正在 Spark 上运行 Apache Beam 的工作负载。我使用 32GB 的内存初始化了工作节点(从节点使用 -c 2 -m 32G 运行)。Spark 提交将驱动程序内存设置为 30g,执行器内存设置为 16g。然而,执行器失败并显示 java.lang.OutOfMemoryError: Java heap space

主节点界面显示每个执行器的内存为 1024M。另外,我看到所有的 Java 进程都是使用 -Xmx 1024m 启动的。这意味着 spark-submit 没有将它的执行器设置传播给执行器。

管道选项如下:

       --runner PortableRunner \
        --job_endpoint=localhost:8099 \
        --environment_type=PROCESS \
        --environment_config='{ "command": "$HOME/beam/sdks/python/container/build/target/launcher/linux_amd64/boot"}';

作业端点按照默认方式设置:

docker run --rm --network=host --name spark-jobservice apache/beam_spark_job_server:latest --spark-master-url=spark://$HOSTNAME:7077

我该如何确保这些设置传播到执行器?

更新:
我将 conf/spark-defaults.conf 设置为:

    spark.driver.memory 32g
    spark.executor.memory 32g

并将 conf/spark-env.sh 设置为:

    SPARK_EXECUTOR_MEMORY=32g

然后重新启动集群并重新启动了所有内容,但执行器内存仍然限制为 1024M。

英文:

I am running an apache beam workload on Spark. I initialized the workers with 32GB of memory (slave run with -c 2 -m 32G). Spark submit sets driver memory to 30g and executor memory to 16g. However, executors fail with java.lang.OutOfMemoryError: Java heap space.

The master gui indicates that memory per executor is 1024M. In addition, I see that all java processes are launched with -Xmx 1024m. This means spark-submit doesn't propagate it's executor settings to the executors.

Pipeline options are as follows:

   --runner PortableRunner \
    --job_endpoint=localhost:8099 \
    --environment_type=PROCESS \
    --environment_config='{"command": "$HOME/beam/sdks/python/container/build/target/launcher/linux_amd64/boot"}'

Job endpoint is setup in the default way:
docker run --rm --network=host --name spark-jobservice apache/beam_spark_job_server:latest --spark-master-url=spark://$HOSTNAME:7077

How do I make sure the settings propagate to the executors?

Update:
I set conf/spark-defaults.conf to

spark.driver.memory 32g
spark.executor.memory 32g

and conf/spark-env.sh to

SPARK_EXECUTOR_MEMORY=32g

and restarted the cluster and relaunched everything, and executor memory is still limited to 1024M

答案1

得分: 3

我找到了原因并且有一个解决方法。

作业服务器容器在内部运行它自己的 Spark 发行版,因此在您的本地机器上配置的 Spark 发行版设置不起作用。

解决方法是在作业服务器容器中更改配置,例如在启动时传递相应的环境变量:

docker run -e SPARK_EXECUTOR_MEMORY=32g --rm --network=host --name spark-jobservice apache/beam_spark_job_server:latest --spark-master-url=spark://$HOSTNAME:7077
英文:

I found the reason and a workaround.

The jobserver container is running internally its own spark distribution, so the settings configured in the spark distribution on your local machine have no effect.

The solution is thus to change the configuration in the jobserver container, for instance by passing the corresponding environment variable when launching it:

docker run -e SPARK_EXECUTOR_MEMORY=32g --rm --network=host --name spark-jobservice apache/beam_spark_job_server:latest --spark-master-url=spark://$HOSTNAME:7077

huangapple
  • 本文由 发表于 2020年10月23日 02:03:51
  • 转载请务必保留本文链接:https://go.coder-hub.com/64488060.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定