块hadoop hdfs格式获取错误块池 [英] hadoop hdfs formatting gets error failed for Block pool
问题描述
2015-05-28 21：41：57,544 WARN组织.apache.hadoop.hdfs.server.common.Storage：java.io.IOException：/ usr / local / hadoop / dfs / datanode中的不兼容clusterID：namenode clusterID = CID-e77ee39a-ab4a-4de1-b1a4-9d4da78b83e8; datanode clusterID = CID-6c250e90-658c-4363-9346-972330ff8bf9
2015-05-28 21：41：57,545 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode：块池初始化失败<登记> （Datanode Uuid unassigned）服务到localhost / 127.0.0.1：9000。退出。
java.io.IOException：所有指定的目录加载失败。
at .. org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead（DataStorage.java:477）
at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage（ DataNode.java:1387）
at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool（DataNode.java:1352）
at org.apache.hadoop.hdfs.server.datanode。 BPOfferService.verifyAndSetNamespaceInfo（BPOfferService.java:316）
位于org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake（BPServiceActor.java:228）
位于org.apache.hadoop.hdfs。 server.datanode.BPServiceActor.run（BPServiceActor.java:852）$ b $在java.lang.Thread.run（Thread.java:745）
...等等...
SHUTDOWN_MSG：
/ ********************************************* ***************
SHUTDOWN_MSG：在der-Inspiron-3521 / 127.0.1.1关闭DataNode
************ ******************************************** /
以下是我所做的步骤：
sbin / stop-dfs.sh
hdfs namenode -format
sbin / start-dfs.sh
您的信息：我的core-site.xml的临时目录如下所示：
<性>
<名称> hadoop.tmp.dir< / name>
<值> / usr / local / hadoop< /值>
< description>其他临时目录的基础。
< / description>
< / property>
以及我的hdfs-site.xml作为namenode和datanode，如下所示：
< property>
< name> dfs.namenode.name.dir< / name>
<值>文件：/ usr / local / hadoop / dfs / namenode< / value>
< / property>
<属性>
< name> dfs.datanode.data.dir< / name>
<值>文件：/ usr / local / hadoop / dfs / datanode< / value>
< / property>
更新：
我已经得到有点进一步与问题，但我仍然得到相同类型的错误。我可以运行 hdfs dfs -format 并根据建议更改版本 。之后，我使用 hdfs dfs -ls和hdfs dfs -mkdir来创建/ user / der 其中der是我的登录名。但是，当我运行我的
猪文件时，我在猪文件中使用了mkDirs和chmod错误。
这里是我datanode和namenode的权限：
drwx ------ 3 der der 4096 May 29 08:13 datanode
drwxrwxrwx 4 root root 4096 May 28 11:34 name
drwxrwxr-x 3 der der 4096 May 29 08:13 namenode
drwxrwxr-x 3 der der 4096 May 29 08:13 namesecondary
drwxr-xr-x 2 root root 4096 5月28日11:46 ww
看来datanode只对拥有者和用户组有权限，而不是用户。
这是我的猪脚本错误：
2015-05-29 08：37：27,152 [JobControl] INFO org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob - PigLatin：totalmiles.pig在提交时出错
ENOENT：没有这样的文件或目录
在org.apache.hadoop.io.nativeio.NativeIO $ POSIX.chmodImpl（本地方法）
在org.apache.hadoop.io.nativeio。 NativeIO $ POSIX.chmod（NativeIO.java:230）
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission（RawLocalFileSystem.java:724）
at org.apache.hadoop.fs.FilterFileSystem.setPermission（FilterFileSystem.java:502）
在org.apache.hadoop.fs.FileSystem.mkdirs（FileSystem.java:600）
在org.apache。 hadoop.mapreduce.JobResourceUploader.uploadFiles（JobResourceUploader.java:94）
在org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles（JobSubmitter.java:98）
在org.apache.hadoop.mapreduce。 JobSubmitter.submitJobInternal（JobSubmitter.java:193）
在org.apache.hadoop.mapreduce.Job $ 10.run（Job.java:1290）
在org.apache.hadoop.mapreduce.Job $ 10。运行（Job.java:1287）
at java.security.AccessController.doPrivileged（Native Method）
at javax.security.auth.Subject.doAs（Subject.java:415）
at org.apache.hadoop.mapreduce.Job.submit org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1657）
（org.apache）
（Job.java:1287） hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit（ControlledJob.java:335）
at sun.reflect.Nat iveMethodAccessorImpl.invoke0（Native Method）
at sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl
这是我的猪脚本：
$ $ p $ records = LOAD'1987.csv'使用PigStorage（'，'）AS
（Year，Month，DayofMonth，DayOfWeek，
DepTime，CRSDepTime，ArrTime，CRSArrTime，
UniqueCarrier，FlightNum，TailNum，ActualElapsedTime，
CRSElapsedTime，AirTime，ArrDelay，DepDelay，
Origin ，Dest，距离：int，TaxIn，
TaxiOut，取消，取消代码，转移，
CarrierDelay，WeatherDelay，NASDelay，SecurityDelay，
lateAircraftDelay）;
milage_recs = GROUP记录ALL;
tot_miles = FOREACH milage_recs GENERATE SUM（records.Distance）;
STORE tot_miles INTO'totalmiles4';
更新：
顺便说一下，我使用chmod go + rw在datanode上namenode服务器和datanode服务器）。
$ hr
5月30日更新：
更多详细信息。
我在猪脚本中将猪脚本的父目录更改为：
records = LOAD'/ user /der/1987.csv'USING PigStorage（'，'）AS
我有同样的错误。在客户端，这是错误。
无法从/读取数据， user / der / 1987.csv
输出：
在hdfs：// localhost：9000 / user / der / totalmiles4中产生结果失败
在服务器端，这里是namenode日志，就在我从我的猪脚本获得无效文件请求的那一刻。日志（使用尾巴-f）滚动。这表明服务器正在接受pig命令的请求。
2015-05-30 07:01:28,140 INFO BlockStateChange：BLOCK * addStoredBlock：blockMap updated：127.0.0.1:50010 is添加到
blk_1073741885_1061 {UCState = UNDER_CONSTRUCTION，
truncateBlock = null，
primaryNodeIndex = -1，replicas = [ReplicaUC [[DISK] DS-c84e0e37-2726-44da-af3e-67167c1010d1：NORMAL ：127.0.0.1：50010 | RBW]]}
size 0
2015-05-30 07:01:28,148 INFO org.apache.hadoop.hdfs.StateChange：DIR * completeFile：
/tmp/temp-11418443/tmp85697770/automaton-1.11-8.jar
由DFSClient_NONMAPREDUCE_-1939565577_1关闭
我只需要得到猪脚本的源代码，并检查它发出的extct hdfs命令。我认为我配置的hadoop hdfs服务存在问题。
2015-05 -28 21：41：57,544 WARN org.apache.hadoop.hdfs.server.common.Storage：java.io.IOException：/ usr / local / hadoop / dfs / datanode中的不兼容clusterID： namenode clusterID = CID- e77ee39a-ab4a-4de1-b1a4-9d4da78b83e8 ; datanode clusterID = CID-6c250e90-658c-4363-9346-972330ff8bf9
您的 namenode和datanode集群ID不匹配。
$ b $打开您的
usr / local / hadoop / dfs / datanode / current / VERSION file和change： clusterID = CID-6c250e90-658c-4363-9346-972330ff8bf9
到
clusterID = CID-e77ee39a-ab4a-4de1-b1a4-9d4da78b83e8
注意：每当您设置namenode的格式时，请检查 namenode 和 datanode 的 VERSION 文件。它们应该具有相同的
clusterID 和名称空间ID 。否则你的datanode，不会启动。
After formatting my hdfs, I get the following errors:
2015-05-28 21:41:57,544 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /usr/local/hadoop/dfs/datanode: namenode clusterID = CID-e77ee39a-ab4a-4de1-b1a4-9d4da78b83e8; datanode clusterID = CID-6c250e90-658c-4363-9346-972330ff8bf9
2015-05-28 21:41:57,545 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000. Exiting.
java.io.IOException: All specified directories are failed to load.
at.. org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:477)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1387)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1352)
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:316)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:228)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:852)
at java.lang.Thread.run(Thread.java:745)
...blah...
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at der-Inspiron-3521/127.0.1.1
************************************************************/
Here are the steps that I did:
sbin/stop-dfs.sh
hdfs namenode -format
sbin/start-dfs.sh
For your information: my core-site.xml has the temp dir as follows:
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop</value>
<description>A base for other temporary directories.
</description>
</property>
and my hdfs-site.xml as the namenode and datanode as follows:
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop/dfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/dfs/datanode</value>
</property>
Update:
I have gotten a bit further with the issue but still I am getting the same type of error. I was able to run the hdfs dfs -format and change the version as advised. After that I used the hdfs dfs -ls and hdfs dfs -mkdir to create /user/der wher der is my log in name. However when I ran the my
pig file, I take mkDirs and chmod errors in my pig file.
Here are the permissions for my datanode and namenode:
drwx------ 3 der der 4096 May 29 08:13 datanode
drwxrwxrwx 4 root root 4096 May 28 11:34 name
drwxrwxr-x 3 der der 4096 May 29 08:13 namenode
drwxrwxr-x 3 der der 4096 May 29 08:13 namesecondary
drwxr-xr-x 2 root root 4096 May 28 11:46 ww
It seems datanode only has permissions for owner and group but not user.
Here is my pig script error:
2015-05-29 08:37:27,152 [JobControl] INFO org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob - PigLatin:totalmiles.pig got an error while submitting
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:724)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:502)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:600)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:98)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:193)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
Here is my pig script:
records = LOAD '1987.csv' USING PigStorage(',') AS
(Year, Month, DayofMonth, DayOfWeek,
DepTime, CRSDepTime, ArrTime, CRSArrTime,
UniqueCarrier, FlightNum, TailNum,ActualElapsedTime,
CRSElapsedTime,AirTime,ArrDelay, DepDelay,
Origin, Dest, Distance:int, TaxIn,
TaxiOut, Cancelled,CancellationCode, Diverted,
CarrierDelay, WeatherDelay, NASDelay, SecurityDelay,
lateAircraftDelay);
milage_recs= GROUP records ALL;
tot_miles = FOREACH milage_recs GENERATE SUM(records.Distance);
STORE tot_miles INTO 'totalmiles4';
Update: By the way I used chmod go+rw on datanode (after I stopped the namenode server and datanode server). That did not work as well.
Update May 30: A bit more details. I changed the parent directory of the pig script in my pig script to this:
records = LOAD '/user/der/1987.csv' USING PigStorage(',') AS
I have the same error. On the client side, here's the error. The only difference is that there is not hdfs:// prefix to the failed input read.
Failed to read data from "/user/der/1987.csv"
Output(s):
Failed to produce result in "hdfs://localhost:9000/user/der/totalmiles4"
On the server side here is the namenode log just at the moment I get the invalid file request from my pig script. The log (using tail -f) scrolled. This indicates that the server is accepting request for the pig command.
2015-05-30 07:01:28,140 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:50010 is added to
blk_1073741885_1061{UCState=UNDER_CONSTRUCTION,
truncateBlock=null,
primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c84e0e37-2726-44da-af3e-67167c1010d1:NORMAL:127.0.0.1:50010|RBW]]}
size 0
2015-05-30 07:01:28,148 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile:
/tmp/temp-11418443/tmp85697770/automaton-1.11-8.jar
is closed by DFSClient_NONMAPREDUCE_-1939565577_1
I am just have to get the source code of the pig script and check the extact hdfs commands it issues. I think there is someone wrong with the hadoop hdfs services that I configured.
2015-05-28 21:41:57,544 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /usr/local/hadoop/dfs/datanode: namenode clusterID = CID-e77ee39a-ab4a-4de1-b1a4-9d4da78b83e8; datanode clusterID = CID-6c250e90-658c-4363-9346-972330ff8bf9
Your namenode and datanode cluster ID does not match.
Open your usr/local/hadoop/dfs/datanode/current/VERSION file and change:
clusterID=CID-6c250e90-658c-4363-9346-972330ff8bf9
to
clusterID=CID-e77ee39a-ab4a-4de1-b1a4-9d4da78b83e8
NOTE: Whenever you format your namenode, check the VERSION file of namenode and datanode. They both should have same clusterID and namespaceID. Otherwise your datanode, won't start.
这篇关于块hadoop hdfs格式获取错误块池的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
