spark 3 错误 java.lang.UnsatisfiedLinkError: 在 java.library.path 中没有 zstd-jni

huangapple go评论89阅读模式
英文:

spark 3 error java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path

问题

After installing Spark3 on Red Hat 7, everything seems to be running fine.

os.environ['SPARK_HOME'] = "/users/spark/spark-3.0.0-bin-hadoop3.2"
os.environ['JAVA_HOME'] = "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64/jre"

However, when attempting a simple join with dat.join(ids_df, on=['symbol']).show(), an error occurred. Can someone assist? Many thanks.

Exception in thread "map-output-dispatcher-0" java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path
Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni.so or load zstd-jni from system libraries. Please try building from source the jar or providing libzstd-jni in your system.
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at com.github.luben.zstd.util.Native.load(Native.java:108)
    at com.github.luben.zstd.util.Native.load(Native.java:60)
    at com.github.luben.zstd.ZstdOutputStream.<clinit>(ZstdOutputStream.java:15)
    at org.apache.spark.io.ZStdCompressionCodec.compressedOutputStream(CompressionCodec.scala:224)
    at org.apache.spark.MapOutputTracker$.serializeMapStatuses(MapOutputTracker.scala:913)
    at org.apache.spark.ShuffleStatus.$anonfun$serializedMapStatus$2(MapOutputTracker.scala:210)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:72)
    at org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:207)
    at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:457)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
英文:

After install the Spark3 to redhat 7, everything seems to runs.

```
os.environ[&#39;SPARK_HOME&#39;] = &quot;/users/spark/spark-3.0.0-bin-hadoop3.2&quot;
os.environ[&#39;JAVA_HOME&#39;] =&quot;/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64/jre&quot;
```

a simple join dat.join(ids_df, on=[&#39;symbol&#39;]).show()

hit this error. Any one can help? Many thanks.


    Exception in thread &quot;map-output-dispatcher-0&quot; java.lang.UnsatisfiedLinkError: no zstd-jni in java.library.path
    Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni.so or load zstd-jni from system libraries. Please try building from source the jar or providing l                                         ibzstd-jni in your system.
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at com.github.luben.zstd.util.Native.load(Native.java:108)
        at com.github.luben.zstd.util.Native.load(Native.java:60)
        at com.github.luben.zstd.ZstdOutputStream.&lt;clinit&gt;(ZstdOutputStream.java:15)
        at org.apache.spark.io.ZStdCompressionCodec.compressedOutputStream(CompressionCodec.scala:224)
        at org.apache.spark.MapOutputTracker$.serializeMapStatuses(MapOutputTracker.scala:913)
        at org.apache.spark.ShuffleStatus.$anonfun$serializedMapStatus$2(MapOutputTracker.scala:210)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:72)
        at org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:207)
        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:457)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

答案1

得分: 2

需要在RedHat 7上以root用户执行yum install命令来安装libzstd:

  • 对于64位系统

     yum install libzstd.x86_64
    
  • 对于32位系统

     yum install libzstd.i686
    
英文:

You need to install the libzstd on RedHat 7 executing the yum install command with root user:

  • For 64 bits systems

     yum install libzstd.x86_64
    
  • For 32 bits systems

     yum install libzstd.i686
    

huangapple
  • 本文由 发表于 2020年7月29日 06:38:16
  • 转载请务必保留本文链接:https://go.coder-hub.com/63143795.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定