英文:
Spark properties file read
问题
我尝试在Spark中读取属性文件,其中我的文件位置在运行作业时可用,但出现以下错误
代码如下:
object runEmpJob {    
  def main(args: Array[String]): Unit = {
    println("starting emp job")
    val props = ConfigFactory.load()
    val envProps = props.getConfig("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")
    System.setProperty("hadoop.home.directory", "D:\\SHARED\\winutils-master\\hadoop-2.6.3\\bin")
    val spark = SparkSession.builder().
         appName("emp dept operation").
         master(envProps.getString("Dev.executionMode")).
         getOrCreate()
    val empObj = new EmpOperation
    empObj.runEmpOperation(spark, "String", fileType = "csv")
    val inPutPath = args(1)
    val outPutPath = args(2)    
  }
}
出现错误:
Exception in thread "main"
com.typesafe.config.ConfigException$BadPath: path parameter: Invalid path C:\Users\mmishra092815\IdeaProjects\use_case_1\src\main\Resource\filepath.properties':
Token not allowed in path expression: ':' (you can double-quote this token if you really want it here)   	
at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:155)  
at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:74)  
at com.typesafe.config.impl.PathParser.parsePath(PathParser.java:61)   
at com.typesafe.config.impl.Path.newPath(Path.java:230)   	
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:192)     
at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)  
at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41) 
at executor.runEmpJob$.main(runEmpJob.scala:12)   	
at executor.runEmpJob.main(runEmpJob.scala)  
Process finished with exit code 1
英文:
I tried to read a properties file in spark where my file location is     available while run the job getting below error
code is
object runEmpJob {    
  def main(args: Array[String]): Unit = {
    println("starting emp job")
    val props = ConfigFactory.load()
    val envProps = props.getConfig("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")
    System.setProperty("hadoop.home.directory", "D:\\SHARED\\winutils-master\\hadoop-2.6.3\\bin")
    val spark = SparkSession.builder().
         appName("emp dept operation").
         master(envProps.getString("Dev.executionMode")).
         getOrCreate()
    val empObj = new EmpOperation
    empObj.runEmpOperation(spark, "String", fileType = "csv")
    val inPutPath = args(1)
    val outPutPath = args(2)    
  }
}
getting error:
> Exception in thread "main"
> com.typesafe.config.ConfigException$BadPath: path parameter: Invalid path C:\Users\mmishra092815\IdeaProjects\use_case_1\src\main\Resource\filepath.properties':
> Token not allowed in path expression: ':' (you can double-quote this token if you really want it here)
> at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:155)
> at com.typesafe.config.impl.PathParser.parsePathExpression(PathParser.java:74)
> at com.typesafe.config.impl.PathParser.parsePath(PathParser.java:61)
> at com.typesafe.config.impl.Path.newPath(Path.java:230)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:192)  
> at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:268)
> at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:274)
> at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:41)
> at executor.runEmpJob$.main(runEmpJob.scala:12)
> at executor.runEmpJob.main(runEmpJob.scala)
> Process finished with exit code 1
答案1
得分: 1
Loading happens in ConfigFactory.load(). If you want to load configuration from a specific file, pass it like:
val props = ConfigFactory.load("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")
As described in API documentation, the getConfig method does not load configuration from a file - it returns a Config object for a given config path (not a filesystem path!).
英文:
Loading happens in ConfigFactory.load(). If you want to load configuration from specific file, pass it like:
val props = ConfigFactory.load("C:\\Users\\mmishra092815\\IdeaProjects\\use_case_1\\src\\main\\Resource\\filepath.properties")
As described in API documentation, getConfig method does not load configuration from file - it returns a Config object for given config path (not filesystem path!)
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。


评论