./spark-shell无法正确启动(spark1.6.1-bin.hadoop2.6版本) [英] ./spark-shell doesn't start correctly (spark1.6.1-bin.hadoop2.6 version)

问题描述

我安装了以下spark版本:spark-1.6.1-bin-hadoop2.6.tgz.

I installed this spark version: spark-1.6.1-bin-hadoop2.6.tgz.

现在，当我使用./spark-shell命令启动spark时，我遇到了这个问题(它显示了很多错误行，所以我只写了一些看起来很重要的内容)

Now when I start spark with ./spark-shell command Im getting this issues (it shows a lot of error lines so I just put some that seems important)

     Cleanup action completed
        16/03/27 00:19:35 ERROR Schema: Failed initialising database.
        Failed to create database 'metastore_db', see the next exception for details.
        org.datanucleus.exceptions.NucleusDataStoreException: Failed to create database 'metastore_db', see the next exception for details.
            at org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:516)

        Caused by: java.sql.SQLException: Directory /usr/local/spark-1.6.1-bin-hadoop2.6/bin/metastore_db cannot be created.
            org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
            ... 128 more
        Caused by: ERROR XBM0H: Directory /usr/local/spark-1.6.1-bin-hadoop2.6/bin/metastore_db cannot be created.


        Nested Throwables StackTrace:
        java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.
  org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
            ... 128 more
        Caused by: ERROR XBM0H: Directory /usr/local/spark-1.6.1-bin-hadoop2.6/bin/metastore_db cannot be created.
            at org.apache.derby.iapi.error.StandardException.newException


        Caused by: java.sql.SQLException: Directory /usr/local/spark-1.6.1-bin-hadoop2.6/bin/metastore_db cannot be created.
            at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
            at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
            at 
            ... 128 more

        <console>:16: error: not found: value sqlContext
                 import sqlContext.implicits._
                        ^
        <console>:16: error: not found: value sqlContext
                 import sqlContext.sql
                        ^

        scala> 

我尝试了一些配置来解决此问题，我在其他有关值sqlContext not found的问题中进行了搜索，例如:

I tried some configurations to fix this issue that I search in other questions about the value sqlContext not found issue, like:

/etc/hosts文件:

/etc/hosts file:

127.0.0.1  hadoophost localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    10.2.0.15 hadoophost

echo $HOSTNAME返回:

hadoophost

hadoophost

.bashrc文件包含:

.bashrc file contains:

export SPARK_LOCAL_IP=127.0.0.1

但是不起作用，您能否提供一些帮助以尝试了解为什么火花无法正确启动?

But dont works, can you give some help to try understand why spark is not starting correctly?

hive-default.xml.template

hive-default.xml.template

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
--><configuration>
  <!-- WARNING!!! This file is auto generated for documentation purposes ONLY! -->
  <!-- WARNING!!! Any changes you make to this file will be ignored by Hive.   -->
  <!-- WARNING!!! You must make your changes in hive-site.xml instead.         -->

在主文件夹中，我遇到相同的问题:

In the home folder I get the same issues:

[hadoopadmin@hadoop home]$ pwd
/home
[hadoopadmin@hadoop home]$ 

文件夹权限:

[hadoopdadmin@hadoop spark-1.6.1-bin-hadoop2.6]$ ls -la
total 1416
drwxr-xr-x. 12 hadoop hadoop    4096 .
drwxr-xr-x. 16 root   root      4096  ..
drwxr-xr-x.  2 hadoop hadoop    4096  bin
-rw-r--r--.  1 hadoop hadoop 1343562  CHANGES.txt
drwxr-xr-x.  2 hadoop hadoop    4096  conf
drwxr-xr-x.  3 hadoop hadoop    4096  data
drwxr-xr-x.  3 hadoop hadoop    4096  ec2
drwxr-xr-x.  3 hadoop hadoop    4096  examples
drwxr-xr-x.  2 hadoop hadoop    4096  lib
-rw-r--r--.  1 hadoop hadoop   17352  LICENSE
drwxr-xr-x.  2 hadoop hadoop    4096  licenses
-rw-r--r--.  1 hadoop hadoop   23529  NOTICE
drwxr-xr-x.  6 hadoop hadoop    4096  python
drwxr-xr-x.  3 hadoop hadoop    4096  R
-rw-r--r--.  1 hadoop hadoop    3359  README.md
-rw-r--r--.  1 hadoop hadoop     120  RELEASE
drwxr-xr-x.  2 hadoop hadoop    4096  sbin

推荐答案

您正在使用具有蜂巢支持构建的spark.

You are using spark built with hive support.

有两种可能的解决方案，具体取决于您稍后要在Spark Shell或Spark工作中要执行的操作-

There are two possible solutions based on what you want to do later with your spark-shell or in your spark jobs -

1. 您要在hadoop + hive安装中访问hive表. 您应该将hive-site.xml放置在spark安装的conf子目录中.从现有的配置单元安装中找到hive-site.xml.例如，在我的cloudera VM中，hive-site.xml位于/usr/lib/hive/conf.完成此步骤后启动spark-shell应该成功连接到现有的配置单元metastore，并且不会尝试在当前工作目录中创建临时.metastore数据库.
2. 您不想在hadoop + hive安装中访问hive表. 如果您不关心连接到配置单元表，则可以遵循Alberto的解决方案.在您从中启动spark-shell的目录中解决权限问题.确保允许您在该目录中创建目录/文件.

1. You want to access hive tables in your hadoop+hive installation. You should place hive-site.xml in your spark installation's conf sub-directory. Find hive-site.xml from your existing hive installation. For example, in my cloudera VM the hive-site.xml is at /usr/lib/hive/conf. Launching the spark-shell after doing this step should successfully connect to existing hive metastore and will not try to create a temporary .metastore database in your current working directory.
2. You do NOT want to access hive tables in your hadoop+hive installation. If you do not care about connecting to hive tables, then you can follow Alberto's solution. Fix the permission issues in the directory from which you are launching spark-shell. Make sure you are allowed to create directories/files in that directory.

希望这会有所帮助.

这篇关于./spark-shell无法正确启动(spark1.6.1-bin.hadoop2.6版本)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！