Spark Mongo DB 连接器 - 找不到类定义 [英] Spark Mongo DB connector - No Class Def Found

问题描述

我在尝试从 Mongo DB 读取数据时遇到问题.

I am facing and issue when trying to read data form Mongo DB.

我有 Spark 1.6.3，它有 Scala 2.10.5

I have Spark 1.6.3 which has Scala 2.10.5

我使用的是 Mongo DB 连接器版本 1.1 和包 2.10

I am using the Mongo DB Connector Version 1.1 and package 2.10

以下是我在 Mavan 中使用的依赖项

Following is the dependencies i had used in my Mavan

<dependency> 
<groupId>org.mongodb.spark</groupId> 
<artifactId>mongo-spark-connector_2.10</artifactId> 
<version>1.1.0</version> 
</dependency> 

出现类似

java.lang.NoClassDefFoundError: com/mongodb/spark/rdd/api/java/JavaMongoRDD
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:708)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.mongodb.spark.rdd.api.java.JavaMongoRDD
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 

但是能够找到引用中存在的类

But was able to find the class existing in the reference

提前致谢

推荐答案

将 mongo-spark-connector_2.10 的 jar 添加到您的运行时类路径

Add mongo-spark-connector_2.10's jar to your run time class path

可以通过以下方式实现，

It can be done it followting ways,

• 将 mongo-spark-connector jar 添加到 Spark lib 文件夹
• spark-submit 和 --jars=/path/mongo-spark-connector.jar
• spark.driver.extraClassPath 或者它的别名 --driver-class-path 在运行驱动程序的节点上设置额外的类路径.
• spark.executor.extraClassPath 在 Worker 节点上设置额外的类路径.

• Add mongo-spark-connector jar to Spark lib folder
• spark-submit with --jars=/path/mongo-spark-connector.jar
• spark.driver.extraClassPath or it's alias --driver-class-path to set extra classpaths on the node running the driver.
• spark.executor.extraClassPath to set extra class path on the Worker nodes.

如果您添加驱动程序类路径，那么您将需要添加执行程序类路径以使 jar 可用于工作节点上的执行程序

If you add Driver class path then you will have need to add executor class paths to make jar available to executor on worker nodes

尝试使用最新版本的 jar 2.2.8 应该有类mongo-spark-connector_2.10:2.2.8

Try to use the latest version jar 2.2.8 which shall have the class mongo-spark-connector_2.10:2.2.8

这篇关于Spark Mongo DB 连接器 - 找不到类定义的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！