2017年在OS X上从源代码构建Apache Hadoop [英] Build Apache Hadoop from source on OS X in 2017

问题描述

到目前为止，我已经分叉了Git仓库：

  https://github.com/apache/hadoop 
  

我一直在寻找有关如何从源代码进行构建的信息，我已经尝试过以下命令： / p>

  mvn软件包-Pdist -Dtar -DskipTests 
  

导致以下错误：

似乎没有关于此的好的文档，有人知道正确的过程吗？

---

更新：

也以这种方式尝试过：

根据

顺便说一句，官方文档完全无济于事-他们本质上说 使用Dock

解决方案

在OSX上使用Homebrew：

安装所需的依赖项：

  brew install protobuf@2.5 gcc autoconf automake libtool cmake snappy gzip bzip2 zlib openssl 
  

符号链接协议：

  ln -s /usr/local/Cellar/protobuf@2.5/2.5.0/bin/protoc / usr / local / bin / protoc 
  

验证版本（libprotoc 2.5.0）：

  protoc --version 
  

导出构建标志：

  export OPENSSL_ROOT_DIR = / usr / local / opt / openssl 
导出LDFLAGS =-L $ {OPENSSL_ROOT_DIR} / lib 
 export CPPFLAGS =-I $ {OPENSSL_ROOT_DIR} /包括 
 export PKG_CONFIG_PATH = $ {OPENSSL_ROOT_DIR} / lib / pkgconfig 
 export OPENSSL_INCLUDE_DIR = $ {OPENSSL_ROOT_DIR} / include 
  

检查Hadoop版本：

  hadoop版本
  

获取上面返回的版本的Hadoop源并进行构建：

  wget https ：//archive.apache.org/dist/hadoop/core/hadoop-3.1.0/hadoop-3.1.0-src.tar.gz 
 tar zxvf hadoop-3.1.0-src.tar.gz 
 cd hadoop-3.1.0-src 
 mvn软件包-Dmaven.javadoc.skip = true -Pdist，native -DskipTests -Dtar 
  

将本机库复制到Homebrew安装中：

  cp -R hadoop- dist / target / hadoop-3.1.0 / lib / /usr/local/Cellar/hadoop/3.1.0/lib/ 
  

更新hadoop-env.sh：

  vi /usr/local/Cellar/hadoop/3.1.0 /libexec/etc/hadoop/hadoop-env.sh 
  

修改HADOOP_OPTS（可能被注释掉）：

  export HADOOP_OPTS =-Djava.net.preferIPv4Stack = true -Djava.library.path = / usr / local / Cellar / hadoop / 3.1.0 / lib / native 
  

重新启动Hadoop并运行以下命令进行验证：

  hadoop checknative -a 
  

基于 https://medium.com/@faizanahemad/hadoop-native-libraries-installation-on-mac -osx-d8338a6923db

Thus far I've forked the Git repo:

https://github.com/apache/hadoop

I've been looking around for information on how to build from source, I've tried this command:

mvn package -Pdist -Dtar -DskipTests

which resulted in the following error:

It doesn't seem that there's good documentation about this, does anyone know the right procedure?

---

UPDATE:

Also tried in this way:

According to this file, you need to get get protoc to work for the package to build

brew link --force --overwrite protobuf250

The macOS dependencies are:

$ brew install cmake
$ brew install zlib
$ brew install protobuf
$ brew install snappy

but to no avail:

By the way, the official docs are totally unhelpful- they essentially say "use a Docker container"

解决方案

Using Homebrew on OSX:

Install the required dependencies:

brew install protobuf@2.5 gcc autoconf automake libtool cmake snappy gzip bzip2 zlib openssl

Symlink protoc:

ln -s /usr/local/Cellar/protobuf@2.5/2.5.0/bin/protoc /usr/local/bin/protoc

Verify the version (libprotoc 2.5.0):

protoc --version

Export build flags:

export OPENSSL_ROOT_DIR="/usr/local/opt/openssl"
export LDFLAGS="-L${OPENSSL_ROOT_DIR}/lib"
export CPPFLAGS="-I${OPENSSL_ROOT_DIR}/include"
export PKG_CONFIG_PATH="${OPENSSL_ROOT_DIR}/lib/pkgconfig"
export OPENSSL_INCLUDE_DIR="${OPENSSL_ROOT_DIR}/include"

Check Hadoop version:

hadoop version

Fetch the Hadoop source for the version returned above and build:

wget https://archive.apache.org/dist/hadoop/core/hadoop-3.1.0/hadoop-3.1.0-src.tar.gz
tar zxvf hadoop-3.1.0-src.tar.gz
cd hadoop-3.1.0-src
mvn package -Dmaven.javadoc.skip=true -Pdist,native -DskipTests -Dtar

Copy the native libs to your Homebrew installation:

cp -R hadoop-dist/target/hadoop-3.1.0/lib/ /usr/local/Cellar/hadoop/3.1.0/lib/

Update hadoop-env.sh:

vi /usr/local/Cellar/hadoop/3.1.0/libexec/etc/hadoop/hadoop-env.sh

Amend HADOOP_OPTS (may be commented out):

export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Djava.library.path=/usr/local/Cellar/hadoop/3.1.0/lib/native"

Restart Hadoop and run the following to verify:

hadoop checknative -a

Based on instructions from https://medium.com/@faizanahemad/hadoop-native-libraries-installation-on-mac-osx-d8338a6923db

这篇关于2017年在OS X上从源代码构建Apache Hadoop的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！