超光速粒子的Dataproc主复制错误 [英] Tachyon on Dataproc Master Replication Error

问题描述

我在那里安装的Tachyon，星火，和Hadoop一个Dataproc主节点上运行一个简单的例子。

我有一个复制错误从星火写的Tachyon。有什么办法可以指定它不需要复制？

  15/10/17 8时45分21秒WARN org.apache.hadoop.hdfs.DFSClient：DataStreamer异常
org.apache.hadoop.ipc.RemoteException（java.io.IOException异常）：文件/ tmp /快子/工人/一万四千四百五十零亿七千百万零一/ 3/8只能被复制到0的节点，而不是minReplication（= 1）。有0数据节点（多个）运行和没有节点（多个）被排除在此操作。
    在org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550)
    在org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets（FSNamesystem.java:3110）
 

我打印日志的部分只是一个警告，但立即星火错误如下：

我查了的Tachyon配置文档，发现一些可能会导致这样的：

  tachyon.underfs.hdfs.implorg.apache.hadoop.hdfs.DistributedFileSystem
 

由于这是所有Dataproc主节点上，用Hadoop preinstalled和HDFS与星火的工作，我会认为这是一个问题，从内部的Tachyon解决的。

解决方案

您可以通过手动设置 dfs.replication 在 /等调整默认复制/hadoop/conf/hdfs-site.xml 比 2 的Dataproc默认其他一些价值。设置它只是在你的主人至少应该覆盖驱动程序调用， Hadoop的FS 调用，它似乎正确地传播到 Hadoop的DistCp使用呼叫，以及因此最有可能你不必担心，只要还设置它在每一个工人作为工人从工作范围的配置，得到他们的文件系统CONFIGS。

请注意 1 的复制已经意味着数据在总的单一副本，而不是意为除了主复制一个副本。因此，复制真的不能低于1的最小复制与 dfs.namenode.replication.min 在同一个 HDFS现场控制的.xml ;你可以看到它引用<一个href=\"https://github.com/apache/hadoop/blob/3b85bd7c3391719624a261f061a67cc4356be477/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L335\"相对=在 BlockManager.java 。在这里>

I have a simple example running on a Dataproc master node where Tachyon, Spark, and Hadoop are installed.

I have a replication error writing to Tachyon from Spark. Is there any way to specify it needs no replication?

15/10/17 08:45:21 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/tachyon/workers/1445071000001/3/8 could only be replicated to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and no node(s) are excluded in this operation.
    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1550)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3110)

The portion of the log I printed is just a warning, but a Spark error follows immediately.

I checked the Tachyon config docs, and found something that might be causing this:

tachyon.underfs.hdfs.impl   "org.apache.hadoop.hdfs.DistributedFileSystem"

Given that this is all on a Dataproc master node, with Hadoop preinstalled and HDFS working with Spark, I would think that this is a problem solvable from within Tachyon.

解决方案

You can adjust default replication by manually setting dfs.replication inside /etc/hadoop/conf/hdfs-site.xml to some value other than Dataproc's default of 2. Setting it just on your master should at least cover driver calls, hadoop fs calls, and it appears to correctly propagate into hadoop distcp calls as well so most likely you don't need to worry about also setting it on every worker as long as workers are getting their FileSystem configs from job-scoped configurations.

Note that replication of 1 already means a single copy of the data in total, rather than meaning "one replica in addition to the main copy". So, replication can't really go lower than 1. The minimum replication is controlled with dfs.namenode.replication.min in the same hdfs-site.xml; you can see it referenced here in BlockManager.java.

这篇关于超光速粒子的Dataproc主复制错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！