英文:
NoSuchMethod Exception thrown when build AvroParquetWriter in java spark
问题
我正在使用Java和Spark，并且有以下代码：
JavaRDD<String> uploadedFiles = resultJavaRDD.map(rdd -> {
String localFilePath = "tmp/myfirst.parquet";
Path dataFile = new Path(filePath);
try (ParquetWriter<MyData> writer = AvroParquetWriter.<MyData>builder(dataFile)
.withSchema(ReflectData.AllowNull.get().getSchema(MyData.class))
.withDataModel(ReflectData.get())
.withConf(new Configuration())
.withCompressionCodec(SNAPPY)
.withWriteMode(OVERWRITE)
.build()) {
for (MyData d : rdd.getDataList()) {
writer.write(d);
}
} catch (Exception e) {
System.err.println("Failed to write to the file. \n" + e.getMessage());
}
// ...
s3Service.uploadFile(s3FilePath, localFlePath);
return s3FilePath;
});
然而，在ParquetWriter的构建步骤中出现了错误。以下是错误信息：
java.lang.NoSuchMethodError: org.apache.parquet.hadoop.ParquetWriter.<init>(Lorg/apache/hadoop/fs/Path;Lorg/apache/parquet/hadoop/ParquetFileWriter$Mode;Lorg/apache/parquet/hadoop/api/WriteSupport;Lorg/apache/parquet/hadoop/metadata/CompressionCodecName;IIIZZLorg/apache/parquet/column/ParquetProperties$WriterVersion;Lorg/apache/hadoop/conf/Configuration;I)V
at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:489)
at com.aktana.api.command.DSESparkDriverCommand.lambda$run$3005c09$1(DSESparkDriverCommand.java:590)
// ...
我的Gradle配置如下：
compile 'com.amazonaws:aws-java-sdk:1.11.747'
compile 'org.apache.parquet:parquet-avro:1.8.1'
问题在于，如果我不使用Spark，而是将相同的代码放入纯Java中，一切似乎都正常工作。请帮忙解决。谢谢。
英文:
I am using Java and Spark and have following code
JavaRDD<String> uploadedFiles = resultJavaRDD.map( rdd -> {
String localFilePath = "tmp/myfirst.parquet";
Path dataFile = new Path(filePath);
// Write as Parquet file.
try (ParquetWriter<MyData> writer = AvroParquetWriter.<MyData>builder(dataFile)
.withSchema(ReflectData.AllowNull.get().getSchema(MyData.class))
.withDataModel(ReflectData.get())
.withConf(new Configuration())
.withCompressionCodec(SNAPPY)
.withWriteMode(OVERWRITE)
.build()) { // this build statment throw exception
for (MyData d : rdd.getDataList()) {
writer.write(d);
}
} catch (Exception e) {
System.err.println("Failed to write to the file. \n" + e.getMessage());
}
......
s3Service.uploadFile(s3FilePath, localFlePath); // my own service to upload file to S3
return s3FilePath;
});
However, I got an error at the ParquetWriter build step. Below is the error.
java.lang.NoSuchMethodError: org.apache.parquet.hadoop.ParquetWriter.<init>(Lorg/apache/hadoop/fs/Path;Lorg/apache/parquet/hadoop/ParquetFileWriter$Mode;Lorg/apache/parquet/hadoop/api/WriteSupport;Lorg/apache/parquet/hadoop/metadata/CompressionCodecName;IIIZZLorg/apache/parquet/column/ParquetProperties$WriterVersion;Lorg/apache/hadoop/conf/Configuration;I)V
at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:489)
at com.aktana.api.command.DSESparkDriverCommand.lambda$run$3005c09$1(DSESparkDriverCommand.java:590)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1015)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
My gradle is like
compile 'com.amazonaws:aws-java-sdk:1.11.747'
compile 'org.apache.parquet:parquet-avro:1.8.1'
The problem is if I don't use Spark, but put the same code in pure java, everything seems working fine. Please help. Thanks.
答案1
得分: 1
似乎对我正在使用的类引用错误。我使用relocate来重新定位org.apache.parquet:parquet-avro的库依赖，现在它已经正常工作。
英文:
There seems to be a mis-reference of the class I am using. I use relocate to relocate the lib dependency of org.apache.parquet:parquet-avro and it works fine now.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论