Spark 1.6中的HDFS单元测试 [英] HDFS Unit testing in Spark 1.6

问题描述

我正在Spark 1.6应用程序中编写单元测试，并且必须模拟HDFS.如此处所述，正在使用MiniDFSCluster https://wiki.apache.org/hadoop/HowToDevelopUnitTests .我在当前使用MiniDFSCluster的Spark 1.6代码中没有看到任何示例.

I am writing unit test in Spark 1.6 application and have to mock a HDFS. Is using the MiniDFSCluster https://wiki.apache.org/hadoop/HowToDevelopUnitTests as described here. I don't see any examples in the current Spark 1.6 code that use MiniDFSCluster.

推荐答案

下面是显示如何设置和使用集群的代码段.

Below is a snippet showing how to setup and use the Cluster.

    val conf = new HdfsConfiguration()
    val testDataCluster = new File("/var/path", "root")
    conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, testDataCluster.getAbsolutePath)
    conf.setInt("dfs.blocksize", 512)
    conf.setInt("dfs.namenode.fs-limits.min-block-size", 512)
    val dfs =  new MiniDFSCluster.Builder(conf).build()
    dfs.waitActive()
    val fileSystem: org.apache.hadoop.fs.FileSystem = dfs.getFileSystem
    fileSystem.copyFromLocalFile(false, new Path(sourceDir),
      new Path(hdfsDir))

您可能需要的一些依赖项(以下为sbt模块定义格式)

some of the dependencies that you might require are (the below are in sbt module definition format)

  "org.apache.hadoop" % "hadoop-client" % "2.7.2" % "provided",
  "org.apache.hadoop" % "hadoop-hdfs" % "2.7.2" % "test",
  "org.apache.hadoop" % "hadoop-common" % "2.7.2" % "test",
  "org.apache.hadoop" % "hadoop-hdfs" % "2.7.2" % "test" classifier "tests",
  "org.apache.hadoop" % "hadoop-common" % "2.7.2" % "test" classifier "tests"

这篇关于Spark 1.6中的HDFS单元测试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！