原因：java.lang.ClassNotFoundException：kinesis.DefaultSource [英] Caused by: java.lang.ClassNotFoundException: kinesis.DefaultSource

查看：21
本文介绍了原因：java.lang.ClassNotFoundException：kinesis.DefaultSource的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

技术堆栈详细信息-

Scala - 2.11.8
Spark - 2.4.4
Delta - 0.7.0
Running On - AWS EMR

用法-

spark.readStream
      .format("kinesis")
      .option("streamName", kinesisConfs.streamName)
      .option("region", kinesisConfs.kinesisRegion)
      .option("initialPosition", "LATEST")
      .option("maxFetchDuration", kinesisConfs.maxFetchDuration)
      .option("shardsPerTask", kinesisConfs.shardsPerTask)
      .option("fetchBufferSize", kinesisConfs.fetchBufferSize)
      .option("minFetchPeriod", kinesisConfs.minFetchPeriod)
      .option("shardFetchInterval", kinesisConfs.shardFetchInterval)
      .option("maxFetchRate", kinesisConfs.maxFetchRate)
      .load()

POM.xml中的依赖项-

<dependencies>
        <!--  SCALA  -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
            <scope>provided</scope>
        </dependency>
        <!--  SPARK  -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.delta</groupId>
            <artifactId>delta-core_2.12</artifactId>
            <version>0.7.0</version>
            <scope>provided</scope>
        </dependency>
        <!--  JSON  -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <!--
        <dependency>
            <groupId>com.fasterxml.jackson.module</groupId>
            <artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        -->
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.5</version>
        </dependency>
        <dependency>
            <groupId>com.softwaremill.sttp</groupId>
            <artifactId>core_${scala.binary.version}</artifactId>
            <version>${sttp.version}</version>
        </dependency>
        <dependency>
            <groupId>com.softwaremill.sttp</groupId>
            <artifactId>json4s_${scala.binary.version}</artifactId>
            <version>${sttp.version}</version>
        </dependency>
        <dependency>
            <groupId>org.json4s</groupId>
            <artifactId>json4s-jackson_${scala.binary.version}</artifactId>
            <version>${json4s.version}</version>
        </dependency>
        <dependency>
            <groupId>org.json4s</groupId>
            <artifactId>json4s-ext_${scala.binary.version}</artifactId>
            <version>${json4s.version}</version>
        </dependency>

        <!--  TEST  -->
        <dependency>
            <groupId>org.scalactic</groupId>
            <artifactId>scalactic_${scala.binary.version}</artifactId>
            <version>3.0.5</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.scalatest</groupId>
            <artifactId>scalatest_${scala.binary.version}</artifactId>
            <version>3.0.5</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.scoverage</groupId>
            <artifactId>scalac-scoverage-plugin_${scala.binary.version}</artifactId>
            <version>${scoverage.plugin.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.storm-enroute</groupId>
            <artifactId>scalameter_${scala.binary.version}</artifactId>
            <version>0.17</version>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.mongodb</groupId>
                    <artifactId>casbah_${scala.binary.version}</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.mongodb</groupId>
            <artifactId>casbah_${scala.binary.version}</artifactId>
            <version>3.1.1</version>
            <type>pom</type>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-catalyst_${scala.binary.version}</artifactId>
            <version>${spark.version}</version>
            <type>test-jar</type>
            <scope>test</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.awaitility/awaitility-scala -->
        <dependency>
            <groupId>org.awaitility</groupId>
            <artifactId>awaitility-scala</artifactId>
            <version>3.1.6</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>....custom_dependency....</groupId>
            <artifactId>....custom_dependency....</artifactId>
            <version>2.0.0</version>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>com.google.code.gson</groupId>
                    <artifactId>gson</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>....custom_dependency....</groupId>
            <artifactId>....custom_dependency....</artifactId>
            <version>1.4.0</version>
            <scope>test</scope>
        </dependency>

        <!--  OTHERS  -->
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-text</artifactId>
            <version>1.6</version>
        </dependency>
        <dependency>
            <groupId>org.ocpsoft.prettytime</groupId>
            <artifactId>prettytime</artifactId>
            <version>4.0.1.Final</version>
        </dependency>
        <dependency>
            <groupId>com.github.scopt</groupId>
            <artifactId>scopt_${scala.binary.version}</artifactId>
            <version>3.7.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-kinesis -->
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-kinesis</artifactId>
            <version>1.11.313</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

错误-

Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: kinesis. Please find packages at http://spark.apache.org/third-party-projects.html
.
.
.
.
.
.
.
.
.
Caused by: java.lang.ClassNotFoundException: kinesis.DefaultSource
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:652)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:652)
    at scala.util.Try$.apply(Try.scala:192)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:652)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:652)
    at scala.util.Try.orElse(Try.scala:84)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:652)
    ... 40 more

使用-

运行Jar 
spark-submit --packages io.delta:delta-core_2.12:0.7.0 /home/hadoop/<jar_name> <some_more_arguments>

当前形式的上述代码成功地在Databricks上运行(我想这是因为Databricks Runtime提供了内部支持)

我到目前为止所尝试的-

在POM.xml依赖项中，我添加了-

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kinesis-asl_2.11</artifactId>
            <version>2.4.4</version>
        </dependency>

在电子病历上运行它-

spark-submit --packages io.delta:delta-core_2.12:0.7.0,org.apache.spark:spark-streaming-kinesis-asl_2.11:2.4.4 /home/hadoop/<Jar_name>.jar  <Some_more_arguments>

但即使这样也会导致相同的错误。我找不到如何解决此问题。

UPDATE-11/Feb/2021

也尝试了spark-streaming-kinesis-asl_2.12:3.0.1，但再次尝试了相同的结果。

提前谢谢。

推荐答案

添加以下依赖项并将其提供给类路径。我能够缓解此问题。

依赖关系-&>

        <dependency>
            <groupId>com.qubole.spark</groupId>
            <artifactId>spark-sql-kinesis_2.11</artifactId>
            <version>1.2.0_spark-2.4</version>
        </dependency>

如果有更好的解决方案，请让我知道。

这篇关于原因：java.lang.ClassNotFoundException：kinesis.DefaultSource的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆