问题

I have been at this for several days now, my objective is simple.

我已经进行了好几天，我的目标很简单。

I am setting up a SparkConf() object inside a Java Application and I need to specify a custom path to the log4j.properties file. The application is meant to run on a Spark Worker which has the custom log4j.properties file required.

我正在Java应用程序内设置一个SparkConf()对象，我需要指定一个自定义路径到log4j.properties文件。这个应用程序是要在一个具有所需的自定义log4j.properties文件的Spark Worker上运行。

It seems like my Spark configuration is unable to find this and is using the default file.

似乎我的Spark配置无法找到这个文件，而是使用默认文件。

I have added the log4j.properties file in several places inside the worker pod like: /app/spark/conf/log4j.properties. But it doesn't seem to work.

我已经在工作节点内的几个位置添加了log4j.properties文件，比如：/app/spark/conf/log4j.properties。但似乎没有起作用。

Here's how I'm trying to set the custom path:

这是我尝试设置自定义路径的方式：

        SparkConf sc = new SparkConf().setMaster(master)
                                      .set("spark.driver.extraJavaOptions", "-Dlog4j.configuration=/app/spark/conf/log4j.properties")
                                      .set("spark.executor.extraJavaOptions", "-Dlog4j.configuration=/app/spark/conf/log4j.properties")

The last two statements are currently having no effect on the Spark Configuration. Any idea what's wrong with this? Is something missing on my end?

最后两个语句目前对Spark配置没有影响。对此有什么想法吗？是我漏掉了什么吗？

Help...

帮帮忙...

        SparkConf sc = new SparkConf().setMaster(master)
                                      .set("spark.driver.extraJavaOptions", "-Dlog4j.configuration=/app/spark/conf/log4j.properties")
                                      .set("spark.executor.extraJavaOptions", "-Dlog4j.configuration=/app/spark/conf/log4j.properties")

答案1

得分: 1

似乎您提供了路径，但没有包含关键字"file"。只需添加以下内容，如果文件位于该路径上，它应该能够引用您的文件...

SparkConf sc = new SparkConf().setMaster(master)
                              .set("spark.driver.extraJavaOptions", "-Dlog4j.configuration=file:/app/spark/conf/log4j.properties")
                              .set("spark.executor.extraJavaOptions", "-Dlog4j.configuration=file:/app/spark/conf/log4j.properties")

        SparkConf sc = new SparkConf().setMaster(master)
                                  .set("spark.driver.extraJavaOptions", "-Dlog4j.configuration=file:/app/spark/conf/log4j.properties")
                                  .set("spark.executor.extraJavaOptions", "-Dlog4j.configuration=file:/app/spark/conf/log4j.properties")