Problem installing hadoop-gremlin with janusgraph 安装hadoop-gremlin与janusgraph时出现问题

huangapple go评论49阅读模式
英文:

Problem installing hadoop-gremlin with janusgraph

问题

We are running into some issues while launching gremlin.sh with hadoop (ref: https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin) using janusgraph.

Any tips/help appreciated! Thanks!

Error with Janus 0.6.3 (https://github.com/JanusGraph/janusgraph/releases/download/v0.6.3/janusgraph-0.6.3.zip):
20:00:36 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: wasbs

Error with Janus 1.0.0-rc2 (https://github.com/JanusGraph/janusgraph/releases/download/v1.0.0-rc2/janusgraph-1.0.0-rc2.zip):
21:32:16 WARN org.apache.hadoop.util.NativeCodeLoader. - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found

The hadoop version we are using is 3.1.1

sshuser@hn0-spark:~/janusgraph$ hadoop version
Hadoop 3.1.1.4.1.14.2
Source code repository https://msdata.visualstudio.com/HDInsight/_git/bigtop -r 9482450b631bf06716ebd304e90e1ea0c806dd4d
Compiled by hdiuser on 2023-04-20T12:54Z
Compiled with protoc 2.5.0
From source with checksum 4224efc0ed85daa1281ef3d22ebb0c
This command was run using /usr/hdp/4.1.14.2/hadoop/hadoop-common-3.1.1.4.1.14.2.jar

The spark cluster is set up using Azure HDInsights.

英文:

We are running into some issues while launching gremlin.sh with hadoop (ref: https://tinkerpop.apache.org/docs/current/reference/#hadoop-gremlin) using janusgraph.

Any tips/help appreciated! Thanks!

Error with Janus 0.6.3 (https://github.com/JanusGraph/janusgraph/releases/download/v0.6.3/janusgraph-0.6.3.zip):
20:00:36 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: wasbs

sshuser@hn0-spark:~/janusgraph/janusgraph-0.6.3$ echo $HADOOP_CONF_DIR
/etc/hadoop/conf
sshuser@hn0-spark:~/janusgraph/janusgraph-0.6.3$ echo $CLASSPATH
/etc/hadoop/conf
sshuser@hn0-spark:~/janusgraph/janusgraph-0.6.3$ bin/gremlin.sh

         \,,,/
         (o o)
-----oOOo-(3)-oOOo-----
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/sshuser/janusgraph/janusgraph-0.6.3/lib/logback-classic-1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/sshuser/janusgraph/janusgraph-0.6.3/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
plugin activated: tinkerpop.server
plugin activated: tinkerpop.tinkergraph
22:04:12 WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: wasbs
        at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:125)
        at org.apache.tinkerpop.gremlin.jsr223.LazyBindingsCustomizer.getBindings(LazyBindingsCustomizer.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
        at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:298)
        at org.apache.tinkerpop.gremlin.console.PluggedIn$_activate_closure1.doCall(PluggedIn.groovy:67)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at groovy.lang.Closure.call(Closure.java:405)
        at groovy.lang.Closure.call(Closure.java:421)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2137)
        at org.codehaus.groovy.runtime.dgm$181.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at org.apache.tinkerpop.gremlin.console.PluggedIn.activate(PluggedIn.groovy:59)
        at org.apache.tinkerpop.gremlin.console.PluggedIn$activate.call(Unknown Source)
        at org.apache.tinkerpop.gremlin.console.Console$_closure18.doCall(Console.groovy:149)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
        at groovy.lang.Closure.call(Closure.java:405)
        at groovy.lang.Closure.call(Closure.java:421)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2331)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2316)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2357)
        at org.codehaus.groovy.runtime.dgm$186.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
        at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:147)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:265)
        at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:524)
Caused by: java.io.IOException: No FileSystem for scheme: wasbs
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181)
        at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:121)
        ... 59 more

Error with Janus 1.0.0-rc2 (https://github.com/JanusGraph/janusgraph/releases/download/v1.0.0-rc2/janusgraph-1.0.0-rc2.zip):
21:32:16 WARN org.apache.hadoop.util.NativeCodeLoader.<clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found

sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ echo $HADOOP_CONF_DIR
/etc/hadoop/conf
sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ echo $CLASSPATH
/etc/hadoop/conf
sshuser@hn0-spark:~/janusgraph/janusgraph-1.0.0-rc2$ bin/gremlin.sh
\,,,/
(o o)
-----oOOo-(3)-oOOo-----
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib/log4j-slf4j-impl-2.20.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib/logback-classic-1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
plugin activated: tinkerpop.server
plugin activated: tinkerpop.tinkergraph
22:05:05 INFO  org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.getCustomizers - HADOOP_GREMLIN_LIBS is set to: /home/sshuser/janusgraph/janusgraph-1.0.0-rc2/lib
22:05:06 WARN  org.apache.hadoop.util.NativeCodeLoader.&lt;clinit&gt; - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread &quot;main&quot; java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:125)
at org.apache.tinkerpop.gremlin.jsr223.LazyBindingsCustomizer.getBindings(LazyBindingsCustomizer.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:298)
at org.apache.tinkerpop.gremlin.console.PluggedIn$_activate_closure1.doCall(PluggedIn.groovy:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
at groovy.lang.Closure.call(Closure.java:405)
at groovy.lang.Closure.call(Closure.java:421)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2137)
at org.codehaus.groovy.runtime.dgm$181.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
at org.apache.tinkerpop.gremlin.console.PluggedIn.activate(PluggedIn.groovy:59)
at org.apache.tinkerpop.gremlin.console.PluggedIn$activate.call(Unknown Source)
at org.apache.tinkerpop.gremlin.console.Console$_closure18.doCall(Console.groovy:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
at groovy.lang.Closure.call(Closure.java:405)
at groovy.lang.Closure.call(Closure.java:421)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2331)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2316)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2357)
at org.codehaus.groovy.runtime.dgm$186.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
at org.apache.tinkerpop.gremlin.console.Console.&lt;init&gt;(Console.groovy:154)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:265)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:576)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2667)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
at org.apache.tinkerpop.gremlin.hadoop.jsr223.HadoopGremlinPlugin.lambda$static$0(HadoopGremlinPlugin.java:121)
... 59 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2571)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2665)
... 67 more

The hadoop version we are using is 3.1.1

sshuser@hn0-spark:~/janusgraph$ hadoop version
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.14.2/hadoop/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.14.2/hadoop-hdfs/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
Hadoop 3.1.1.4.1.14.2
Source code repository https://msdata.visualstudio.com/HDInsight/_git/bigtop -r 9482450b631bf06716ebd304e90e1ea0c806dd4d
Compiled by hdiuser on 2023-04-20T12:54Z
Compiled with protoc 2.5.0
From source with checksum 4224efc0ed85daa1281ef3d22ebb0c
This command was run using /usr/hdp/4.1.14.2/hadoop/hadoop-common-3.1.1.4.1.14.2.jar

The spark cluster is setup using Azure HDInsights.

答案1

得分: 3

JanusGraph-0.6.3随附hadoop-2.8客户端,因此不能与hadoop-3集群一起使用。因此,您应该依赖于使用hadoop-3.3客户端的JanusGraph-1.0.0.rc2。

关于java.lang.ClassNotFoundException异常:JanusGraph分发版不包含hadoop-aws和aws-java-sdk-s3 jars,建议您将hadoop-aws-3.3.4.jar和aws-java-s3-x.y.jar添加到JanusGraph分发版的libs文件夹中。

要使事情正常工作,您可以忽略警告"org.apache.hadoop.util.NativeCodeLoader"。

英文:

JanusGraph-0.6.3 ships with the hadoop-2.8 client, so you cannot expect it to work with a hadoop-3 cluster. So, indeed, you should depend on JanusGraph-1.0.0.rc2 which uses the hadoop-3.3 client.
Regarding the java.lang.ClassNotFoundException exception: the JanusGraph distribution does not include the hadoop-aws and aws-java-sdk-s3 jars, so advice you to add the hadoop-aws-3.3.4.jar and the aws-java-s3-x.y.jar to the libs folder of the JanusGraph distribution.

To get things working you can ignore the warning "org.apache.hadoop.util.NativeCodeLoader".

huangapple
  • 本文由 发表于 2023年6月1日 06:20:10
  • 转载请务必保留本文链接:https://go.coder-hub.com/76377637.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定