英文:
Hive with TEZ failed to start Hive CLI
问题
亲爱的，
Apache Hive 3.1.2与Hadoop 3.1.1在我配置了Tez后出现了问题，根据这个文档：[ https://github.com/NitinKumar94/Installing-Apache-Tez]
它始终报错，我尝试了许多解决方案都没有成功
主线程中的异常 "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/BatchListingOperations
...
有什么帮助吗？
我也尝试了以下方法，但没有成功
从 $HIVE_HOME/lib 复制 hive-exec jar 到 hdfs 目录 /user/tez/
英文:
Dears,
Apache Hive 3.1.2 with Hadoop 3.1.1 was working fine with until i configured Hive with Tez
based on this doc :[ https://github.com/NitinKumar94/Installing-Apache-Tez]
it always gives this error, i tried many solutions with no luck
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/BatchListingOperations
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3268)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3313)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.hadoop.hive.common.FileUtils.getJarFilesByPath(FileUtils.java:1006)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5198)
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Any help.
i tried the following also with no luck
Copy hive-exec jar from $HIVE_HOME/lib to hdfs dir /user/tez/
答案1
得分: 0
如果以下任何错误是你的问题，你可以尝试以下解决方案
错误: 无法找到或加载主类
org.apache.tez.dag.app.DAGAppMaster
你需要在Hive服务器实例中导出你的Tez配置文件（hive env sh）
export TEZ_CONF_DIR=/etc/tez/conf/
export TEZ_JARS=/your_path_2_tez/tez-0.10.2/
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}
然后你需要将你的Tez文件夹复制到HDFS，但不要压缩，你需要复制未压缩的Tez包
hdfs dfs -mkdir /tez/tez-0.10.2
然后复制你的文件
hdfs dfs -copyFromLocal /your_path_2_tez/tez-0.10.2/* /tez/tez-0.10.2/
然后在tez-site.xml中定义此路径
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/tez/tez-0.10.2,${fs.defaultFS}/tez/tez-0.10.2/lib</value>
</property>
此外，你需要复制你的Hadoop版本JAR文件（你的版本为3.1.1）到Tez的lib文件夹中：
hadoop-common-3.3.1.jar
hadoop-yarn-common-3.3.1.jar
hadoop-yarn-client-3.3.1.jar
......
.....
主要思路是，根据你的异常信息，找出哪个类不存在，然后你可以从你的Hadoop安装目录的共享JAR文件夹中复制该JAR文件。你还需要将这些JAR文件复制到HDFS中。
然后你可以重新启动你的Hive服务器
这样你就可以在没有错误的情况下使用Tez。
英文:
If any of below errors and yours you can try this solution
Error: Could not find or load main class
org.apache.tez.dag.app.DAGAppMaster
You need to export your tez confs in hive server instance ( hive env sh)
export TEZ_CONF_DIR=/etc/tez/conf/
export TEZ_JARS=/your_path_2_tez/tez-0.10.2/
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}
Then you need to copy your tez folder to hdfs but not zip , you need to copy uncompressed tez package
hdfs dfs -mkdir /tez/tez-0.10.2
Then copy your files
hdfs dfs -copyFromLocal /your_path_2_tez/tez-0.10.2/* /tez/tez-0.10.2/
Then define this path in tez-site xml
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/tez/tez-0.10.2,${fs.defaultFS}/tez/tez-0.10.2/lib</value>
</property>
Also you need to copy your hadoop version jars (yours 3.1.1) to tez lib folder :
hadoop-common-3.3.1.jar
hadoop-yarn-common-3.3.1.jar
hadoop-yarn-client-3.3.1.jar
......
.....
the main idea is which class is not exist looking your exception you can copy the jar from your hadoop share jars from your hadoop installation directory. You need to copy this jars to hdfs as well
Then you can restart your hive server
Here you can use tez without error
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论