英文:
Spark properties file read
问题
我尝试在Spark中读取属性文件,其中我的文件位置在运行作业时可用,但出现以下错误
代码如下:
object runEmpJob {
def main(args: Array[String]): Unit = {
println("starting emp job")
val props = ConfigFactory.load()
val envProps = props.getConfig("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")
System.setProperty("hadoop.home.directory", "D:\\SHARED\\winutils-master\\hadoop-2.6.3\\bin")
val spark = SparkSession.builder().
appName("emp dept operation").
master(envProps.getString("Dev.executionMode")).
getOrCreate()
val empObj = new EmpOperation
empObj.runEmpOperation(spark, "String", fileType = "csv")
val inPutPath = args(1)
val outPutPath = args(2)
}
}
出现错误:
Exception in thread "main"
com.typesafe.config.ConfigException$BadPath: path parameter: Invalid path C:\Users\mmishra092815\IdeaProjects\use_case_1\src\main\Resource\filepath.properties':
Token not allowed in path expression: ':' (you can double-quote this token if you really want it here)
at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:155)
at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:74)
at com.typesafe.config.impl.PathParser.parsePath(PathParser.java:61)
at com.typesafe.config.impl.Path.newPath(Path.java:230)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:192)
at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
at executor.runEmpJob$.main(runEmpJob.scala:12)
at executor.runEmpJob.main(runEmpJob.scala)
Process finished with exit code 1
英文:
I tried to read a properties file in spark where my file location is available while run the job getting below error
code is
object runEmpJob {
def main(args: Array[String]): Unit = {
println("starting emp job")
val props = ConfigFactory.load()
val envProps = props.getConfig("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")
System.setProperty("hadoop.home.directory", "D:\\SHARED\\winutils-master\\hadoop-2.6.3\\bin")
val spark = SparkSession.builder().
appName("emp dept operation").
master(envProps.getString("Dev.executionMode")).
getOrCreate()
val empObj = new EmpOperation
empObj.runEmpOperation(spark, "String", fileType = "csv")
val inPutPath = args(1)
val outPutPath = args(2)
}
}
getting error:
> Exception in thread "main"
> com.typesafe.config.ConfigException$BadPath: path parameter: Invalid path C:\Users\mmishra092815\IdeaProjects\use_case_1\src\main\Resource\filepath.properties':
> Token not allowed in path expression: ':' (you can double-quote this token if you really want it here)
> at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:155)
> at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:74)
> at com.typesafe.config.impl.PathParser.parsePath(PathParser.java:61)
> at com.typesafe.config.impl.Path.newPath(Path.java:230)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:192)
> at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
> at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
> at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
> at executor.runEmpJob$.main(runEmpJob.scala:12)
> at executor.runEmpJob.main(runEmpJob.scala)
> Process finished with exit code 1
答案1
得分: 1
Loading happens in ConfigFactory.load()
. If you want to load configuration from a specific file, pass it like:
val props = ConfigFactory.load("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")
As described in API documentation, the getConfig
method does not load configuration from a file - it returns a Config
object for a given config path (not a filesystem path!).
英文:
Loading happens in ConfigFactory.load()
. If you want to load configuration from specific file, pass it like:
val props = ConfigFactory.load("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")
As described in API documentation, getConfig
method does not load configuration from file - it returns a Config
object for given config path (not filesystem path!)
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论