错误星火建筑使用Maven [英] Error building Spark with Maven

问题描述

我想建立星火1.2使用Maven。我的目标是在Hadoop 2.2使用PySpark纱线。

我看到了，这是唯一可能通过建立星火使用Maven。首先，这是真的吗？

如果这是真的，什么是下面的日志中的问题？如何纠正呢？

  C：\\星火\\火花1.2.0＆GT; MVN -Pyarn -Phadoop 2.2 -Dhadoop.version = 2.2.0 -DskipTests
清洁套装
拿起_JAVA_OPTIONS：-Djava.net preferIPv4Stack = TRUE
[INFO]扫描项目...
[INFO] ----------------------------------------------- -------------------------
[INFO] Reactor生成顺序：
[信息]
[INFO]星火项目父POM
[INFO]星火项目网络
[INFO]星火项目随机播放流媒体服务
[INFO]星火项目核心
[INFO]星火项目百吉
[INFO]星火项目GraphX
[INFO]星火项目流
[INFO]星火项目催化剂
[INFO]星火项目SQL
[INFO]星火项目ML库
[INFO]星火项目工具
[INFO]星火项目蜂房
[INFO]星火项目REPL
[INFO]星火项目YARN父POM
[INFO]星火项目YARN稳定的API
[INFO]星火项目大会
[INFO]星火项目外部微博
[INFO]星火项目外部水槽水槽
[INFO]星火项目外部水槽
[INFO]星火项目外部MQTT
[INFO]星火项目外部ZeroMQ
[INFO]星火项目外部卡夫卡
[INFO]星火工程实例
[INFO]星火项目YARN随机播放服务
[信息]
[INFO] ----------------------------------------------- -------------------------
[INFO]建设星火项目父POM 1.2.0
[INFO] ----------------------------------------------- -------------------------
[信息]
[信息] --- Maven的清理插件：2.5：清理（默认清洁）@火花父---
[信息]删除C：\\星火\\火花1.2.0 \\目标
[信息]
[信息] --- Maven的实施者 - 插件：1.3.1：强制（执行版本）@火花父
 ---
[信息]
[信息] ---集结帮手 -  Maven的插件：1.8：添加源（附加斯卡拉来源）@火花
母公司---
[INFO]源目录：C：\\星火\\火花1.2.0的\\ src \\主\\斯卡拉说。
[信息]
[信息] --- Maven的远程资源 - 插件：1.5：过程（默认）@火花父 - 
 - 
[信息]
[信息] ---斯卡拉，Maven的插件：3.2.0：编译（斯卡拉编译一）@火花父
 ---
[INFO]没有源编译
[信息]
[信息] ---集结帮手 -  Maven的插件：1.8：添加测试代码（附加斯卡拉 - 测试 - 来源
）@火花父---
[信息]测试源目录：C：\\星火\\火花1.2.0的\\ src \\测试\\斯卡拉说。
[信息]
[信息] ---斯卡拉，Maven的插件：3.2.0：testCompile（斯卡拉测试 - 编译 - 第）@水疗
RK-父---
[INFO]没有源编译
[信息]
[信息] --- Maven的依赖关系的插件：2.9：建设类路径（默认）@火花父
---
[INFO]写到classpath中的文件C：\\星火\\火花1.2.0 \\目标\\火花试验classpath.tx
T'。
[信息]
[信息] --- gmavenplus-插件：1.2：执行（默认）@火花父---
[INFO]使用Groovy 2.3.7进行执行。
[信息]
[信息] --- Maven的网站插件：3.3：附加描述符（附上描述符）@火花-P
---的arent
[信息]
[信息] --- Maven的遮阳插件：2.2：遮阳（默认）@火花父---
[信息]包括org.spark-project.spark：未使用：罐子：1.0.0在阴影罐子。
[INFO]与阴影的神器原更换神器。
[信息]
[信息] --- Maven的源插件：2.2.1：JAR-无叉（创建源-JAR）@火花面值
耳鼻喉科---
[信息]
[信息] --- scalastyle-Maven的插件：0.4.0：检查（缺省）@火花父---
【警告】sourceDirectory没有指定或不存在值= C：\\星火\\晶石
K-1.2.0的\\ src \\主\\斯卡拉
保存到OUTPUTFILE = C：\\星火\\火花1.2.0 \\ scalastyle-的Output.xml
处理0文件（S）
找到0个错误
找到0个警告
发现0的相关信息
完成了32毫秒
[信息]
[INFO] ----------------------------------------------- -------------------------
[INFO]建设星火项目网络1.2.0
[INFO] ----------------------------------------------- -------------------------
[信息]
[信息] --- Maven的清理插件：2.5：清理（默认清洁）@火花网络common_2
0.10 ---
[信息]删除C：\\星火\\火花1.2.0 \\网络\\ COMMON \\目标
[信息]
[信息] --- Maven的实施者 - 插件：1.3.1：强制（执行版本）@火花BP网络在
K-com​​mon_2.10 ---
[信息]
[信息] ---集结帮手 -  Maven的插件：1.8：添加源（附加斯卡拉来源）@火花
网络common_2.10 ---
[INFO]源目录：C：\\星火\\火花1.2.0 \\网络\\ COMMON \\ SRC \\主\\斯卡拉ADDE
ð。
[信息]
[信息] --- Maven的远程资源 - 插件：1.5：过程（默认）@火花网络-C
ommon_2.10 ---
[信息]
[信息] --- Maven的资源 - 插件：2.6：资源（缺省资源）@火花NETW
ORK-com​​mon_2.10 ---
[信息]使用UTF-8编码复制过滤资源。
[信息]跳到不存在的resourceDirectory C：\\星火\\火花1.2.0 \\网络\\共同\\ S
RC \\主\\资源
[INFO]复制3资源
[信息]
[信息] ---斯卡拉，Maven的插件：3.2.0：编译（斯卡拉编译一）@火花BP网络在
K-com​​mon_2.10 ---
【警告】锌服务器不可用的端口3030  - 恢复到正常INCREM
ental编译
[信息]使用渐进式编译
[INFO]编译器插件：BasicArtifact（org.scalamacros，paradise_2.10.4,2.0.1，空
）
[INFO] 42编译Java源代码到C：\\星火\\火花1.2.0 \\网络\\ COMMON \\目标\\ S
卡拉 -  2.10 \\类...
[INFO] ----------------------------------------------- -------------------------
[INFO]反应堆摘要：
[信息]
[INFO]星火项目父POM ...........................成功[5.267秒]
[INFO]星火项目网络...........................失败[1.922秒]
[INFO]星火项目随机播放流媒体服务............ SKIPPED
[INFO]星火项目核心................................. SKIPPED
[INFO]星火项目百吉................................ SKIPPED
[INFO]星火项目GraphX​​ ............................... SKIPPED
[INFO]星火项目流............................ SKIPPED
[INFO]星火项目催化剂............................. SKIPPED
[INFO]星火项目SQL .................................. SKIPPED
[INFO]星火项目ML图书馆........................... SKIPPED
[INFO]星火项目工具................................ SKIPPED
[INFO]星火项目蜂巢................................. SKIPPED
[INFO]星火项目REPL ................................. SKIPPED
[INFO]星火项目YARN父POM ...................... SKIPPED
[INFO]星火项目YARN稳定的API ...................... SKIPPED
[INFO]星火项目大会............................. SKIPPED
[INFO]星火项目外部微博..................... SKIPPED
[INFO]星火项目外部水槽水槽.................. SKIPPED
[INFO]星火项目外部水槽....................... SKIPPED
[INFO]星火项目外部MQTT ........................ SKIPPED
[INFO]星火项目外部ZeroMQ ...................... SKIPPED
[INFO]星火项目外部卡夫卡....................... SKIPPED
[INFO]星火工程实例............................. SKIPPED
[INFO]星火项目YARN洗牌服务................. SKIPPED
[INFO] ----------------------------------------------- -------------------------
[INFO]构建失败
[INFO] ----------------------------------------------- -------------------------
[INFO]总时间：8.048小号
[INFO]在表面处理：2015-02-09T10：17：47 + 08：00
[INFO]最后的内存：49M / 331M
[INFO] ----------------------------------------------- -------------------------
[**错误]未能执行目标net.alchim31.maven：斯卡拉，Maven的插件：3.2.0：compi
总结：项目火花网络common_2.10乐（斯卡拉编译一）java.io.IOE
xception：不能运行程序的javac：CreateProcess的错误= 2，系统无法˚F
的ind指定的文件 - ＆GT; [求助1] **
[错误]
[错误]要查看错误的完整堆栈跟踪，与-e SWIT重新运行Maven
CH。
[错误]使用-X开关启用完全调试日志记录重新运行Maven。
[错误]
[错误]有关错误和可能的解决方案，请REA更多信息
D中的以下文章：
[错误] [说明1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
xception
[错误]
[错误]纠正问题后，您可以使用以下命令恢复建设[错误] MVN＆LT;目标＆GT; -rf：火花网络common_2.10
 

解决方案

我第一次安装了JRE，而不是JDK。我的环境变量仍然引用的JRE文件夹，所以无法找到的javac.exe二进制文件。

I am trying to build Spark 1.2 with Maven. My goal is to use PySpark with YARN on Hadoop 2.2.

I saw that this was only possible by building Spark with Maven. First, is this true?

If it is true, what is the problem in the log below? How do I correct this?

C:\Spark\spark-1.2.0>mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -DskipTests
clean package
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Parent POM
[INFO] Spark Project Networking
[INFO] Spark Project Shuffle Streaming Service
[INFO] Spark Project Core
[INFO] Spark Project Bagel
[INFO] Spark Project GraphX
[INFO] Spark Project Streaming
[INFO] Spark Project Catalyst
[INFO] Spark Project SQL
[INFO] Spark Project ML Library
[INFO] Spark Project Tools
[INFO] Spark Project Hive
[INFO] Spark Project REPL
[INFO] Spark Project YARN Parent POM
[INFO] Spark Project YARN Stable API
[INFO] Spark Project Assembly
[INFO] Spark Project External Twitter
[INFO] Spark Project External Flume Sink
[INFO] Spark Project External Flume
[INFO] Spark Project External MQTT
[INFO] Spark Project External ZeroMQ
[INFO] Spark Project External Kafka
[INFO] Spark Project Examples
[INFO] Spark Project YARN Shuffle Service
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Parent POM 1.2.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ spark-parent ---
[INFO] Deleting C:\Spark\spark-1.2.0\target
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-versions) @ spark-parent
 ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-scala-sources) @ spark-
parent ---
[INFO] Source directory: C:\Spark\spark-1.2.0\src\main\scala added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-parent --
-
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ spark-parent
 ---
[INFO] No sources to compile
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-test-source (add-scala-test-sources
) @ spark-parent ---
[INFO] Test Source directory: C:\Spark\spark-1.2.0\src\test\scala added.
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile-first) @ spa
rk-parent ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-dependency-plugin:2.9:build-classpath (default) @ spark-parent
---
[INFO] Wrote classpath file 'C:\Spark\spark-1.2.0\target\spark-test-classpath.tx
t'.
[INFO]
[INFO] --- gmavenplus-plugin:1.2:execute (default) @ spark-parent ---
[INFO] Using Groovy 2.3.7 to perform execute.
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-p
arent ---
[INFO]
[INFO] --- maven-shade-plugin:2.2:shade (default) @ spark-parent ---
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO]
[INFO] --- maven-source-plugin:2.2.1:jar-no-fork (create-source-jar) @ spark-par
ent ---
[INFO]
[INFO] --- scalastyle-maven-plugin:0.4.0:check (default) @ spark-parent ---
[WARNING] sourceDirectory is not specified or does not exist value=C:\Spark\spar
k-1.2.0\src\main\scala
Saving to outputFile=C:\Spark\spark-1.2.0\scalastyle-output.xml
Processed 0 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 32 ms
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Networking 1.2.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ spark-network-common_2
.10 ---
[INFO] Deleting C:\Spark\spark-1.2.0\network\common\target
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-versions) @ spark-networ
k-common_2.10 ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-scala-sources) @ spark-
network-common_2.10 ---
[INFO] Source directory: C:\Spark\spark-1.2.0\network\common\src\main\scala adde
d.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-network-c
ommon_2.10 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-netw
ork-common_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory C:\Spark\spark-1.2.0\network\common\s
rc\main\resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @ spark-networ
k-common_2.10 ---
[WARNING] Zinc server is not available at port 3030 - reverting to normal increm
ental compile
[INFO] Using incremental compilation
[INFO] compiler plugin: BasicArtifact(org.scalamacros,paradise_2.10.4,2.0.1,null
)
[INFO] Compiling 42 Java sources to C:\Spark\spark-1.2.0\network\common\target\s
cala-2.10\classes...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  5.267 s]
[INFO] Spark Project Networking ........................... FAILURE [  1.922 s]
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project Bagel ................................ SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Parent POM ...................... SKIPPED
[INFO] Spark Project YARN Stable API ...................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.048 s
[INFO] Finished at: 2015-02-09T10:17:47+08:00
[INFO] Final Memory: 49M/331M
[INFO] ------------------------------------------------------------------------
[**ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compi
le (scala-compile-first) on project spark-network-common_2.10: wrap: java.io.IOE
xception: Cannot run program "javac": CreateProcess error=2, The system cannot f
ind the file specified -> [Help 1]**
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please rea
d the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
xception
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :spark-network-common_2.10

解决方案

I had first installed JRE instead of JDK. My environment variables still referenced the JRE folder, and so it couldn't find the javac.exe binary.

这篇关于错误星火建筑使用Maven的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！