Spark属性文件读取

huangapple go评论81阅读模式
英文:

Spark properties file read

问题

我尝试在Spark中读取属性文件,其中我的文件位置在运行作业时可用,但出现以下错误
代码如下:

object runEmpJob {    
  def main(args: Array[String]): Unit = {

    println("starting emp job")
    val props = ConfigFactory.load()
    val envProps = props.getConfig("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")

    System.setProperty("hadoop.home.directory", "D:\\SHARED\\winutils-master\\hadoop-2.6.3\\bin")

    val spark = SparkSession.builder().
         appName("emp dept operation").
         master(envProps.getString("Dev.executionMode")).
         getOrCreate()
    val empObj = new EmpOperation

    empObj.runEmpOperation(spark, "String", fileType = "csv")

    val inPutPath = args(1)
    val outPutPath = args(2)    
  }
}

出现错误:

Exception in thread "main"
com.typesafe.config.ConfigException$BadPath: path parameter: Invalid path C:\Users\mmishra092815\IdeaProjects\use_case_1\src\main\Resource\filepath.properties':
Token not allowed in path expression: ':' (you can double-quote this token if you really want it here)   	
at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:155)  
at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:74)  
at com.typesafe.config.impl.PathParser.parsePath(PathParser.java:61)   
at com.typesafe.config.impl.Path.newPath(Path.java:230)   	
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:192)     
at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)  
at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41) 
at executor.runEmpJob$.main(runEmpJob.scala:12)   	
at executor.runEmpJob.main(runEmpJob.scala)  
Process finished with exit code 1
英文:

I tried to read a properties file in spark where my file location is available while run the job getting below error
code is

object runEmpJob {    
  def main(args: Array[String]): Unit = {

    println("starting emp job")
    val props = ConfigFactory.load()
    val envProps = props.getConfig("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")

    System.setProperty("hadoop.home.directory", "D:\\SHARED\\winutils-master\\hadoop-2.6.3\\bin")

    val spark = SparkSession.builder().
         appName("emp dept operation").
         master(envProps.getString("Dev.executionMode")).
         getOrCreate()
    val empObj = new EmpOperation

    empObj.runEmpOperation(spark, "String", fileType = "csv")

    val inPutPath = args(1)
    val outPutPath = args(2)    
  }
}


getting error:

> Exception in thread "main"
> com.typesafe.config.ConfigException$BadPath: path parameter: Invalid path C:\Users\mmishra092815\IdeaProjects\use_case_1\src\main\Resource\filepath.properties':
> Token not allowed in path expression: ':' (you can double-quote this token if you really want it here)
> at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:155)
> at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:74)
> at com.typesafe.config.impl.PathParser.parsePath(PathParser.java:61)
> at com.typesafe.config.impl.Path.newPath(Path.java:230)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:192)
> at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
> at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
> at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
> at executor.runEmpJob$.main(runEmpJob.scala:12)
> at executor.runEmpJob.main(runEmpJob.scala)
> Process finished with exit code 1

答案1

得分: 1

Loading happens in ConfigFactory.load(). If you want to load configuration from a specific file, pass it like:

val props = ConfigFactory.load("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")

As described in API documentation, the getConfig method does not load configuration from a file - it returns a Config object for a given config path (not a filesystem path!).

英文:

Loading happens in ConfigFactory.load(). If you want to load configuration from specific file, pass it like:

val props = ConfigFactory.load("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")

As described in API documentation, getConfig method does not load configuration from file - it returns a Config object for given config path (not filesystem path!)

huangapple
  • 本文由 发表于 2020年1月3日 15:50:00
  • 转载请务必保留本文链接:https://go.coder-hub.com/59574949.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定