SparkException cause by java.lang.NoClassDefFoundError: org/apache/htrace/core/HTraceConfiguration

huangapple
go评论206阅读模式
英文:

SparkException cause by java.lang.NoClassDefFoundError: org/apache/htrace/core/HTraceConfiguration

问题

似乎在JAR包中找不到HTraceConfiguration类，但我在JAR包中找到了这个类。

  1. jar ftv app.jar  | grep HTraceConfiguration
  3. 2814 Tue Jul 10 17:26:52 CST 2018 org/apache/hadoop/hbase/trace/HBaseHTraceConfiguration.class
  4. 1202 Thu Sep 15 10:07:46 CST 2016 org/apache/htrace/core/HTraceConfiguration$MapConf.class
  5. 3249 Thu Sep 15 10:07:46 CST 2016 org/apache/htrace/core/HTraceConfiguration.class

但是，为什么在执行任务时抛出了NoClassDefFoundError？

以下是异常堆栈跟踪信息

  1. Caused by: org.apache.spark.SparkException: 由于阶段失败而中止作业：阶段 119.0 中的任务 31 失败了 4 次，最近一次失败：阶段 119.0 中的任务 31.3 失败（TID 1024）（172.20.2.15 执行器 2）：org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException：失败了 1 次动作：java.lang.NoClassDefFoundError：org/apache/htrace/core/HTraceConfiguration：问题服务器 1 次，服务器问题：null
  2. 	at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
  3. 	at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1195)
  4. 	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doFlush(BufferedMutatorImpl.java:298)
  5. 	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:270)
  6. 	at org.apache.hadoop.hbase.spark.HBaseContext.hbaseForeachPartition(HBaseContext.scala:481)
  7. 	at org.apache.hadoop.hbase.spark.HBaseContext.$anonfun$bulkPut$1$adapted(HBaseContext.scala:221)
  8. 	at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1011)
  9. 	at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1011)
  10. 	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2278)
  11. 	at org.apache.spark.scheduler.Task.run(Task.scala:136)
  12. 	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  13. 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  14. 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  15. 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  16. 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  17. 	at java.lang.Thread.run(Thread.java:748)
  19. Driver stacktrace:
  20. 	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)
  21. 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)
  22. 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)
  23. 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)
  24. 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)
  25. 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)
  26. 	at scala.Option.foreach(Option.scala:407)
  27. 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)
  28. 	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
  29. 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2238)
  30. 	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  31. 	at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)
  32. 	at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:1009)
  33. 	at org.apache.hadoop.hbase.spark.HBaseContext.bulkPut(HBaseContext.scala:221)
  34. 	at org.apache.hadoop.hbase.spark.HBaseRDDFunctions$GenericHBaseRDDFunctions.hbaseBulkPut(HBaseRDDFunctions.scala:60)
  35. 	at scala.collection.Iterator.foreach(Iterator.scala:943)
  36. 	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
  37. 	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
  39. 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  40. 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  41. 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

你能告诉我发生了什么吗？

英文:

It seems no class HTraceConfiguration found in jar package. But, I do found this class in jar package.

  1. jar ftv app.jar  | grep HTraceConfiguration
  3. 2814 Tue Jul 10 17:26:52 CST 2018 org/apache/hadoop/hbase/trace/HBaseHTraceConfiguration.class
  4. 1202 Thu Sep 15 10:07:46 CST 2016 org/apache/htrace/core/HTraceConfiguration$MapConf.class
  5. 3249 Thu Sep 15 10:07:46 CST 2016 org/apache/htrace/core/HTraceConfiguration.class

But, why NoClassDefFoundError thrown while exec task?

Here is exception stack trace

  1. aused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 31 in stage 119.0 failed 4 times, most recent failure: Lost task 31.3 in stage 119.0 (TID 1024) (172.20.2.15 executor 2): org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: java.lang.NoClassDefFoundError: org/apache/htrace/core/HTraceConfiguration: 1 time, servers with issues: null
  2. 	at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:50)
  3. 	at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1195)
  4. 	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.doFlush(BufferedMutatorImpl.java:298)
  5. 	at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:270)
  6. 	at org.apache.hadoop.hbase.spark.HBaseContext.hbaseForeachPartition(HBaseContext.scala:481)
  7. 	at org.apache.hadoop.hbase.spark.HBaseContext.$anonfun$bulkPut$1$adapted(HBaseContext.scala:221)
  8. 	at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1011)
  9. 	at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1011)
  10. 	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2278)
  11. 	at org.apache.spark.scheduler.Task.run(Task.scala:136)
  12. 	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  13. 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  14. 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  15. 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  16. 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  17. 	at java.lang.Thread.run(Thread.java:748)
  19. Driver stacktrace:
  20. 	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)
  21. 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)
  22. 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)
  23. 	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)
  24. 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)
  25. 	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)
  26. 	at scala.Option.foreach(Option.scala:407)
  27. 	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)
  28. 	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
  29. 	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2238)
  30. 	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  31. 	at org.apache.spark.rdd.RDD.withScope(RDD.scala:406)
  32. 	at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:1009)
  33. 	at org.apache.hadoop.hbase.spark.HBaseContext.bulkPut(HBaseContext.scala:221)
  34. 	at org.apache.hadoop.hbase.spark.HBaseRDDFunctions$GenericHBaseRDDFunctions.hbaseBulkPut(HBaseRDDFunctions.scala:60)
  35. 	at scala.collection.Iterator.foreach(Iterator.scala:943)
  36. 	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
  37. 	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
  39. 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  40. 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  41. 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

Could you tell me, what happened?

答案1

得分: 1

如果您使用 spark-submit 启动作业，请添加 --jars 参数，并提供您的 JAR 文件的路径。它将被添加到执行器的类路径中。

英文:

If you start your job with spark-submit, add --jars parameter with the path to your jar. It will be added to the executor's classpath.

huangapple
  • 本文由 发表于 2023年6月13日 13:03:13
  • 转载请务必保留本文链接：https://go.coder-hub.com/76461816.html
  • apache-spark
  • hbase
  • java
  • maven
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定