Spark Twitter Streaming异常:(org.apache.spark.Logging)classnotfound [英] Spark Twitter Streaming exception : (org.apache.spark.Logging) classnotfound

问题描述

我正在尝试使用Maven在Scala中使用Spark Twitter Streaming示例，但运行它时出现以下错误:

I am trying Spark Twitter Streaming example with Scala using Maven but I am getting below error when I run it:

原因:java.lang.ClassNotFoundException:org.apache.spark.Logging

Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging

以下是我的依赖项:

<dependencies>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.10</artifactId>
    <version>2.0.0</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.10</artifactId>
    <version>2.0.0</version>
</dependency> 
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-twitter_2.11</artifactId>
    <version>1.6.2</version> 
</dependency> 

我知道Logging已移至org.apache.spark.internal.Logging，但是我不知道是否是原因，我已经尝试将依赖关系的版本更改为最新版本，但是没有运气.

I know that the Logging has been moved to org.apache.spark.internal.Logging but I don't know if it is the reason , I already tried to change the version of dependencies to the latest one but with no luck.

推荐答案

TLDR;

类org.apache.spark.Logging在Spark版本 1.5.2 中可用，或者较低的版本(尽管我并未在所有较低版本上进行测试)，但在高于相同版本的版本中不可用.

---

归结为使用 Apache Spark :

Class org.apache.spark.Logging is available in Spark version 1.5.2 or lower (though I didn't test on all lower versions) but is not available in versions higher than the same.

---

It all comes down to using incompatible version of Apache Spark:

user@ubuntu:~$ /opt/spark/bin/spark-shell
Welcome to
  ____              __
 / __/__  ___ _____/ /__
_\ \/ _ \/ _ `/ __/  '_/
/___/ .__/\_,_/_/ /_/\_\   version 2.0.0
   /_/      
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
scala> import org.apache.spark.Logging
<console>:23: error: object Logging is not a member of package org.apache.spark
import org.apache.spark.Logging
          ^

找不到类org.apache.spark.Logging.

---

2.让我们尝试在 Spark 1.6.2 :
上进行import org.apache.spark.Logging h3>

(与上述相同，即找不到org.apache.spark类.未找到日志.)

---

Class org.apache.spark.Logging is not found.

---

2. Let's try to import org.apache.spark.Logging on Spark 1.6.2:

(same as above i.e. Class org.apache.spark.Logging is not found.)

---

user@ubuntu:~$ /opt/spark-1.5.2-bin-hadoop2.6/bin/spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.5.2
      /_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
scala> import org.apache.spark.Logging
import org.apache.spark.Logging

是的！它已可用并且已成功导入

您会看到必需的org.apache.spark.Logging Spark-Streaming-Twitter ，可在Spark版本 1.5.2 或更低版本，所以我建议您使用1.5.2或更低版本的spark.

YES! It is available and successfully imported

As you can see that org.apache.spark.Logging which is required by the Spark-Streaming-Twitter, is available in Spark version 1.5.2 or lower, so I would recommend you to use 1.5.2 or a lower version of spark.

因此，您应该将maven依赖项替换为以下内容: (假设您正在使用Scala 2.11.x)

Hence, you should replace your maven dependencies with followings: (Assuming that you are using Scala 2.11.x)

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>1.5.2</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.11</artifactId>
    <version>1.5.2</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-twitter_2.11</artifactId>
    <version>1.6.2</version>
</dependency>

请注意，artifactId:2.11指的是scala版本，而version:1.5.2或1.6.2指的是库(火花核或spark-streaming-twitter)版本.

Note that the artifactId: 2.11 refers to scala version and version: 1.5.2 or 1.6.2 refers to the library (spark-core or spark-streaming-twitter) version.

这篇关于Spark Twitter Streaming异常:(org.apache.spark.Logging)classnotfound的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！