Hadoop namenode 在 Windows 上的格式化 – java.lang.UnsupportedOperationException

huangapple go评论83阅读模式
英文:

Hadoop namenode formatting windows - java.lang.UnsupportedOperationException

问题

以下是翻译好的内容:

我正在学校的数据库课程中,我的教授让我们使用Hadoop v3.2.1。在按照YouTube上的教程在Windows上安装时,我卡在了格式化namenode的部分。以下是在命令提示符中显示的内容:

2020-03-15 15:38:05,819 INFO util.GSet: 正在计算map NameNodeRetryCache的容量
2020-03-15 15:38:05,819 INFO util.GSet: VM类型       = 64位
2020-03-15 15:38:05,820 INFO util.GSet: 0.029999999329447746% 最大内存 889 MB = 273.1 KB
2020-03-15 15:38:05,820 INFO util.GSet: 容量      = 2^15 = 32768 条目
2020-03-15 15:38:05,883 INFO namenode.FSImage: 分配新的BlockPoolId: BP-381120843-10.0.0.230-1584301085876
2020-03-15 15:38:05,884 ERROR namenode.NameNode: 启动namenode失败。
java.lang.UnsupportedOperationException
        at java.nio.file.Files.setPosixFilePermissions(Files.java:2044)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:452)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:591)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:613)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:188)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1206)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1649)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1759)
2020-03-15 15:38:05,887 INFO util.ExitUtil: 以状态1退出:java.lang.UnsupportedOperationException
2020-03-15 15:38:05,889 INFO namenode.NameNode: 关闭消息:
/************************************************************
SHUTDOWN_MSG:在DrStrange/10.0.0.230上关闭NameNode
************************************************************/

以下是我的配置:

core-site.xml:

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

mapred-site.xml:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
      <value>yarn</value>
  </property>
</configuration>

hdfs-site.xml:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>C:\hadoop-3.2.1\data\namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>C:\hadoop-3.2.1\data\datanode</value>
  </property>
</configuration>

yarn-site.xml:

<configuration>
<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

我当时在遵循这个教程:在Windows上安装Hadoop,但到一半的时候我意识到它太旧了,于是切换到了这个教程:在Windows 10上安装Hadoop 3.2.0

另外,我不确定这是否与我目前的问题有关,所以我会说明一下。当我跳到下一步并输入start-all时,资源管理器和节点管理器都出现错误。我把这些问题都放在一个问题里。

英文:

I am in a databasing class at school and my professor is having us work with hadoop v3.2.1. In following a youtube tutorial to install on windows, I am stuck on the formatting namenode part. this is what comes up in cmd:

2020-03-15 15:38:05,819 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2020-03-15 15:38:05,819 INFO util.GSet: VM type       = 64-bit
2020-03-15 15:38:05,820 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
2020-03-15 15:38:05,820 INFO util.GSet: capacity      = 2^15 = 32768 entries
2020-03-15 15:38:05,883 INFO namenode.FSImage: Allocated new BlockPoolId: BP-381120843-10.0.0.230-1584301085876
2020-03-15 15:38:05,884 ERROR namenode.NameNode: Failed to start namenode.
java.lang.UnsupportedOperationException
        at java.nio.file.Files.setPosixFilePermissions(Files.java:2044)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:452)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:591)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:613)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:188)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1206)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1649)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1759)
2020-03-15 15:38:05,887 INFO util.ExitUtil: Exiting with status 1: java.lang.UnsupportedOperationException
2020-03-15 15:38:05,889 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at DrStrange/10.0.0.230
************************************************************/

and here are my properties:

core-site.xml:

&lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;fs.defaultFS&lt;/name&gt;
    &lt;value&gt;hdfs://localhost:9000&lt;/value&gt;
  &lt;/property&gt;
&lt;/configuration&gt;

mapred-site.xml

&lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;mapreduce.framework.name&lt;/name&gt;
      &lt;value&gt;yarn&lt;/value&gt;
  &lt;/property&gt;
&lt;/configuration&gt;

hdfs-site.xml

&lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.replication&lt;/name&gt;
    &lt;value&gt;1&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.namenode.name.dir&lt;/name&gt;
    &lt;value&gt;C:\hadoop-3.2.1\data\namenode&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.datanode.data.dir&lt;/name&gt;
    &lt;value&gt;C:\hadoop-3.2.1\data\datanode&lt;/value&gt;
  &lt;/property&gt;
&lt;/configuration&gt;

yarn-site.xml:

&lt;configuration&gt;
&lt;!-- Site specific YARN configuration properties --&gt;
  &lt;property&gt;
    &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;
    &lt;value&gt;mapreduce_shuffle&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;yarn.nodemanager.aux-services.mapreduce.shuffle.class&lt;/name&gt;
    &lt;value&gt;org.apache.hadoop.mapred.ShuffleHandler&lt;/value&gt;
  &lt;/property&gt;
&lt;/configuration&gt;

and i was following this tutorial:
How to Install Hadoop on Windows until about halfway through when i realized it was too old and then switched to this one:
How to Install Hadoop 3.2.0 in Windows10

Also, I have no idea if this is related to my current problem, so I will say this. When I skip to the next step and type start-all, resource manager and node manager both error out. Figured I'd stuff it all into one question.

答案1

得分: 1

这是已经众所周知的,显然Hadoop 3.2.1在Windows 10上不能正常工作。我安装了3.2.0版本,现在正在取得进展。

英文:

It is well known apparently that hadoop 3.2.1 doesn't work on windows 10 properly. I installed 3.2.0 and am now making progress.

答案2

得分: 0

以下错误是Hadoop 3.2.1中的一个错误

此问题将在下一个版本中解决。目前,您可以通过以下步骤临时修复它:

  1. 此链接下载hadoop-hdfs-3.2.1.jar文件。
  2. 在文件夹%HADOOP_HOME%\share\hadoop\hdfs中将文件名hadoop-hdfs-3.2.1.jar重命名为hadoop-hdfs-3.2.1.bak
  3. 将下载的hadoop-hdfs-3.2.1.jar复制到文件夹%HADOOP_HOME%\share\hadoop\hdfs中。

参考资料

英文:

The following error is a Bug in Hadoop 3.2.1.

This issue will be solved within the next release. For now, you can fix it temporarily using the following steps:

  1. Download hadoop-hdfs-3.2.1.jar file from the following link.
  2. Rename the file name hadoop-hdfs-3.2.1.jar to hadoop-hdfs-3.2.1.bak in folder %HADOOP_HOME%\share\hadoop\hdfs
  3. Copy the downloaded hadoop-hdfs-3.2.1.jar to folder %HADOOP_HOME%\share\hadoop\hdfs

References

huangapple
  • 本文由 发表于 2020年3月16日 03:57:14
  • 转载请务必保留本文链接:https://go.coder-hub.com/60697054.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定