用gradle构建一个简单的MapReduce项目:Hadoop依赖项没有Mapper和Reducer [英] Building a simple MapReduce project with gradle: Hadoop dependencies don't have Mapper and Reducer

问题描述

我正在尝试构建一个简单的Hadoop mapreduce程序，为此我选择了Java.我检查了周围的示例代码并尝试构建自己.我创建了以下gradle脚本，当我查看已安装的依赖项时，没有一个具有Mapper或Reducer.甚至没有org.apache.hadoop.mapreduce包.

I'm trying to build a simple Hadoop mapreduce program and I chose Java for that job. I checked out the example codes around and tried to build myself. I created the following gradle script and when I looked at the installed dependencies, none had Mapper or Reducer. Not even org.apache.hadoop.mapreduce package.

group 'org.ardilgulez.demoprojects'
version '1.0-SNAPSHOT'

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.11'
    compile group: 'org.apache.hadoop', name: 'hadoop-common', version:'2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-hdfs', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-yarn-common', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-minicluster', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-core', version:'2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-jobclient', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-app', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-shuffle', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-common', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-client', version: '2.7.3'
}

我知道在这10个hadoop依赖项中我至少不需要7个，但是我不知道其中哪个依赖项具有org.apache.hadoop.mapreduce包(我知道这11个都没有但是).

I know I'm not going to be needing at least 7 out of those 10 hadoop dependencies, but I don't know which of these dependencies have org.apache.hadoop.mapreduce package (I know these 11 don't though).

我应该添加哪个依存关系/存储库，以便实际上可以构建mapreduce作业?

Which dependency/repository should I add so that I can actually build a mapreduce job?

我可以使用原始的org.apache.hadoop软件包而不是供应商软件包(例如Cloudera)来做到这一点吗?

Can I do that with raw org.apache.hadoop packages and not vendor packages (such as Cloudera)?

提前感谢所有帮助.

推荐答案

这应该是正确的依赖项:

This should be the correct dependency:

compile 'org.apache.hadoop:hadoop-mapreduce-client-core:2.7.3'

compile 'org.apache.hadoop:hadoop-mapreduce-client-core:2.7.3'

请确保刷新您的gradle项目.

这篇关于用gradle构建一个简单的MapReduce项目:Hadoop依赖项没有Mapper和Reducer的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！