winutils火花windows安装 [英] winutils spark windows installation

问题描述

我试图在Windows 10安装星火1.6.1，到目前为止，我也做了以下...

I am trying to install Spark 1.6.1 on windows 10 and so far I have done the following...

1. 下载火花1.6.1，解压到某个目录，然后将SPARK_HOME


2. 下载斯卡拉2.11.8，解压到某个目录，然后将SCALA_HOME


3. 将_JAVA_OPTION环境变量


4. 通过从网上下载 https://github.com/steveloughran/winutils.git 的winutils刚下载的zip目录，然后设置HADOOP_HOME环境变量。 （不知道这是不正确的，我不能克隆，因为拒绝的权限的目录）。

1. Downloaded spark 1.6.1, unpacked to some directory and then set SPARK_HOME
2. Downloaded scala 2.11.8, unpacked to some directory and then set SCALA_HOME
3. Set the _JAVA_OPTION env variable
4. Downloaded the winutils from https://github.com/steveloughran/winutils.git by just downloading the zip directory and then set HADOOP_HOME env variable. (Not sure if this was incorrect, I could not clone the directory because of permission denied).

当我去激发家庭和运行BIN \\火花壳我得到

When I go to spark home and run bin\spark-shell I get

'C:\Program' is not recognized as an internal or external command, operable program or batch file.

我必须失去了一些东西，我不明白我怎么可以从Windows环境下运行反正在bash脚本。但我希望我并不需要了解只是为了得到这个工作。我一直在关注这个家伙的教程 - 的https://hernandezpaul.word$p$pss.com/2016/01/24/apache-spark-installation-on-windows-10/ 。任何帮助将是AP preciated。

I must be missing something, I don't see how I could be running the bash scripts anyway from windows environment. But hopefully I don't need to understand just to get this working. I have been following this guy's tutorial - https://hernandezpaul.wordpress.com/2016/01/24/apache-spark-installation-on-windows-10/ . Any help would be appreciated.

推荐答案

您需要下载winutils可执行文件，而不是源$ C ​​$ C。

You need to download the winutils executable, not source code.

您可以在这里下载 ，或者你真的想整个Hadoop发行版，你可以找到2.6.0二进制这里。然后，你需要 HADOOP_HOME 设置为包含目录winutils.exe。

You can download it here, or if you really want the entire Hadoop distribution you can find the 2.6.0 binaries here. Then, you need to set HADOOP_HOME to the directory containing winutils.exe.

此外，确保您将Spark在该目录不包含空格的目录，这是否则将无法正常工作非常重要。

Also, make sure the directory you place Spark in is a directory that doesn't contain whitespaces, this is extremely important otherwise it won't work.

一旦你设置它，你不开始 spark-shell.sh ，启动火花shell.cmd ：

Once you've set it up, you don't start spark-shell.sh, you start spark-shell.cmd:

C:\Spark\bin>spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.1
      /_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_91)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
16/05/18 19:31:56 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-core-3.2.10.jar."
16/05/18 19:31:56 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
16/05/18 19:31:56 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-rdbms-3.2.9.jar."
16/05/18 19:31:56 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:31:56 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:32:01 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/05/18 19:32:01 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/05/18 19:32:07 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-core-3.2.10.jar."
16/05/18 19:32:07 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
16/05/18 19:32:07 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Spark/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/bin/../lib/datanucleus-rdbms-3.2.9.jar."
16/05/18 19:32:07 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:32:08 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/05/18 19:32:12 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/05/18 19:32:12 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
SQL context available as sqlContext.

scala>

这篇关于winutils火花windows安装的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！