如何修复“ ClassCastException：无法分配实例”-在本地工作，但不能在群集上独立运行 [英] How to fix 'ClassCastException: cannot assign instance of' - Works local but not in standalone on cluster
本文介绍了如何修复“ ClassCastException：无法分配实例”-在本地工作，但不能在群集上独立运行的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我有一个Spring Web应用程序（内置于Maven中），通过该应用程序我可以连接到我的spark集群（4个工人和1个master）和我的cassandra集群（4个节点）。应用程序启动，工作人员与主服务器通信，并且cassandra集群也正在运行。但是，当我通过Web应用程序的界面执行PCA（火花mllib）或任何其他计算（群集，peerson，spearman）时，出现以下错误：
I have a Spring web application(built in maven) with which I connect to my spark cluster(4 workers and 1 master) and to my cassandra cluster(4 nodes). The application starts, the workers communicate with the master and the cassandra cluster is also running. However when I do a PCA(spark mllib) or any other calculation(clustering, pearson, spearman) through the interface of my web-app I get the following error:
java.lang.ClassCastException：无法将scala.collection.immutable.List $ SerializationProxy实例分配给字段org.apache.spark.rdd.RDD.org $ apache $ spark $ rdd $ RDD $$ dependencies_在此命令上出现的org.apache.spark.rdd.MapPartitionsRDD实例中类型为scala.collection.Seq的
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
p>
which appears on this command:
df1=df2.as(Encoders.STRING()).collectAsList();
我已经复制了spark-2.4 .3-bin-hadoop2.7文件夹在master文件夹和每个工人文件夹上。另外，我在spark-jars文件夹中添加了3个jar，因为如果我不这样做（火花需要它们），则会出现错误：
I have copied spark-2.4.3-bin-hadoop2.7 folder on master and on every worker. Also I have added 3 jars in the spark-jars folder because I get an error if I do not(spark needs them):
- spark -cassandra-connector_2.11-2.4.0.jar
- guava-19.0.jar
- cassandra-driver-core-3.4.0.jar
看来这是一个很难追踪的错误，并且有很多不同的方法和解释（此处1 此处2 此处3 还有很多）。我尝试了最近几天发现的所有解决方案（以及更多），但似乎没有任何效果。我的应用程序的依赖关系和spark（可能是番石榴或scala）的依赖关系之间似乎存在冲突，但是我找不到确切的信息。我试过了：
It seems it is an error hard to trace and there is a considerable amount of different approaches and explanations (here1 here2 here3 and many more). I tried every solution I found (and many more) the last days but nothing seems to work. It seems to be a conflict between dependencies of my application and dependencies of spark(maybe guava or scala) but I cannot find what exactly. I have tried:
- 将我的application-jar-with-dependencies放入spark.jars类路径中。
- 在每个工作线程上复制具有依赖关系的application-jar。
- 将所有spark-jar替换为我的application-jar，分别位于master和集群的每个工作线程上。
- 在我的应用程序和spark-jars文件夹中没有相同的jar。
- 使用了不同版本的不同依赖项。
- 尝试使用提供阴影的插件（没有成功）或作用域。
- 玩过spark.jars和spark.executor.extraClassPath。
- 升级/降级节点上的scala版本（最终完全从worker和master卸载了scala，我仍然遇到相同的错误）
- 使用spark-version 2.4.0
- 和许多其他方法...
- put my application-jar-with-dependencies in spark.jars classpath.
- copy the application-jar-with-dependencies on every worker.
- replace all spark-jars with my applications-jars on master and on every worker of the cluster.
- not having the same jars in my application and spark-jars folder.
- played with different versions of different dependencies.
- tried to use shaded plugin(no success) or scope with provided.
- played with spark.jars and spark.executor.extraClassPath.
- Upgrade/downgrade scala version on nodes(ended up uninstalling scala completely from workers and master and I still get the same error)
- using spark-version 2.4.0
- and many other approaches...
火花配置为
public SparkSession sparksession() {
SparkSession sp = SparkSession
.builder()
.master("spark://(ip):7077")
.appName("biomet")
.config("spark.cassandra.connection.host","192.168.100.30,192.168.100.6,192.168.100.13,192.168.100.16")
.config("spark.executor.extraClassPath","/home/ubuntu/spark-2.4.3-bin-hadoop2.7/jars/guava-19.0.jar")
.config("spark.sql.shuffle.partitions",2)
.config("spark.driver.memory","4g")
.config("spark.driver.maxResultSize", "2g")
.config("spark.executor.memory","2g")
.config("spark.sql.pivotMaxValues","35000")
.config("spark.sql.caseSensitive",true)
.config("spark.hadoop.fs.s3a.access.key",".....")
.config("spark.hadoop.fs.s3a.secret.key",".......")
.getOrCreate();
return sp;
}
@Bean
public JavaSparkContext sc(){
JavaSparkContext sc = new JavaSparkContext(sparksession().sparkContext());
return sc;
}
和pom.xml是：
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>app</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>springapp</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.9.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>2.4.3</version>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-cassandra</artifactId>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-unshaded_2.11</artifactId>
<version>2.4.0</version>
</dependency>
</dependencies>
</project>
如果需要，这是mvn依赖项：tree
In case needed, this is the mvn dependency:tree
[INFO] com.example:app:jar:0.0.1-SNAPSHOT
[INFO] +- org.springframework.boot:spring-boot-starter-thymeleaf:jar:2.0.9.RELEASE:compile
[INFO] | +- org.springframework.boot:spring-boot-starter:jar:2.0.9.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot:jar:2.0.9.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.9.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.9.RELEASE:compile
[INFO] | | | +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] | | | | \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] | | | \- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] | | | \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] | | +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] | | \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] | +- org.thymeleaf:thymeleaf-spring5:jar:3.0.11.RELEASE:compile
[INFO] | | \- org.thymeleaf:thymeleaf:jar:3.0.11.RELEASE:compile
[INFO] | | +- org.attoparser:attoparser:jar:2.0.5.RELEASE:compile
[INFO] | | \- org.unbescape:unbescape:jar:1.1.6.RELEASE:compile
[INFO] | \- org.thymeleaf.extras:thymeleaf-extras-java8time:jar:3.0.4.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.9.RELEASE:compile
[INFO] | +- org.springframework.boot:spring-boot-starter-json:jar:2.0.9.RELEASE:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.8:compile
[INFO] | | | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] | | | \- com.fasterxml.jackson.core:jackson-core:jar:2.9.8:compile
[INFO] | | +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.8:compile
[INFO] | | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.8:compile
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.8:compile
[INFO] | +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.9.RELEASE:compile
[INFO] | | +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.39:compile
[INFO] | | +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.39:compile
[INFO] | | \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.39:compile
[INFO] | +- org.hibernate.validator:hibernate-validator:jar:6.0.16.Final:compile
[INFO] | | +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] | | +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] | | \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] | +- org.springframework:spring-web:jar:5.0.13.RELEASE:compile
[INFO] | \- org.springframework:spring-webmvc:jar:5.0.13.RELEASE:compile
[INFO] | \- org.springframework:spring-aop:jar:5.0.13.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:2.0.9.RELEASE:test
[INFO] | +- org.springframework.boot:spring-boot-test:jar:2.0.9.RELEASE:test
[INFO] | +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.0.9.RELEASE:test
[INFO] | +- com.jayway.jsonpath:json-path:jar:2.4.0:test
[INFO] | | \- net.minidev:json-smart:jar:2.3:test
[INFO] | | \- net.minidev:accessors-smart:jar:1.2:test
[INFO] | +- junit:junit:jar:4.12:test
[INFO] | +- org.assertj:assertj-core:jar:3.9.1:test
[INFO] | +- org.mockito:mockito-core:jar:2.15.0:test
[INFO] | | +- net.bytebuddy:byte-buddy:jar:1.7.11:test
[INFO] | | +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
[INFO] | | \- org.objenesis:objenesis:jar:2.6:compile
[INFO] | +- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] | +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] | +- org.skyscreamer:jsonassert:jar:1.5.0:test
[INFO] | | \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
[INFO] | +- org.springframework:spring-core:jar:5.0.13.RELEASE:compile
[INFO] | | \- org.springframework:spring-jcl:jar:5.0.13.RELEASE:compile
[INFO] | +- org.springframework:spring-test:jar:5.0.13.RELEASE:test
[INFO] | \- org.xmlunit:xmlunit-core:jar:2.5.1:test
[INFO] +- org.apache.spark:spark-mllib_2.11:jar:2.4.3:compile
[INFO] | +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.1.0:compile
[INFO] | +- org.apache.spark:spark-core_2.11:jar:2.4.3:compile
[INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
[INFO] | | +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] | | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | | | +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] | | | \- org.tukaani:xz:jar:1.5:compile
[INFO] | | +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:compile
[INFO] | | | +- org.apache.avro:avro-ipc:jar:1.8.2:compile
[INFO] | | | \- commons-codec:commons-codec:jar:1.11:compile
[INFO] | | +- com.twitter:chill_2.11:jar:0.9.3:compile
[INFO] | | | \- com.esotericsoftware:kryo-shaded:jar:4.0.2:compile
[INFO] | | | \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] | | +- com.twitter:chill-java:jar:0.9.3:compile
[INFO] | | +- org.apache.xbean:xbean-asm6-shaded:jar:4.8:compile
[INFO] | | +- org.apache.hadoop:hadoop-client:jar:2.6.5:compile
[INFO] | | | +- org.apache.hadoop:hadoop-common:jar:2.6.5:compile
[INFO] | | | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | | | +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | | | | +- commons-io:commons-io:jar:2.4:compile
[INFO] | | | | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | | | | \- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | | +- com.google.code.gson:gson:jar:2.8.5:compile
[INFO] | | | | +- org.apache.hadoop:hadoop-auth:jar:2.6.5:compile
[INFO] | | | | | +- org.apache.httpcomponents:httpclient:jar:4.5.8:compile
[INFO] | | | | | | \- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO] | | | | | \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] | | | | | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] | | | | | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] | | | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] | | | | +- org.apache.curator:curator-client:jar:2.6.0:compile
[INFO] | | | | \- org.htrace:htrace-core:jar:3.0.4:compile
[INFO] | | | +- org.apache.hadoop:hadoop-hdfs:jar:2.6.5:compile
[INFO] | | | | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | | | | \- xerces:xercesImpl:jar:2.9.1:compile
[INFO] | | | | \- xml-apis:xml-apis:jar:1.4.01:compile
[INFO] | | | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.5:compile
[INFO] | | | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.5:compile
[INFO] | | | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.5:compile
[INFO] | | | | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.5:compile
[INFO] | | | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.5:compile
[INFO] | | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.5:compile
[INFO] | | | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.5:compile
[INFO] | | | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.5:compile
[INFO] | | | | +- javax.xml.bind:jaxb-api:jar:2.3.1:compile
[INFO] | | | | | \- javax.activation:javax.activation-api:jar:1.2.0:compile
[INFO] | | | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
[INFO] | | | | \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
[INFO] | | | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.5:compile
[INFO] | | | \- org.apache.hadoop:hadoop-annotations:jar:2.6.5:compile
[INFO] | | +- org.apache.spark:spark-launcher_2.11:jar:2.4.3:compile
[INFO] | | +- org.apache.spark:spark-kvstore_2.11:jar:2.4.3:compile
[INFO] | | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] | | +- org.apache.spark:spark-network-common_2.11:jar:2.4.3:compile
[INFO] | | +- org.apache.spark:spark-network-shuffle_2.11:jar:2.4.3:compile
[INFO] | | +- org.apache.spark:spark-unsafe_2.11:jar:2.4.3:compile
[INFO] | | +- javax.activation:activation:jar:1.1.1:compile
[INFO] | | +- org.apache.curator:curator-recipes:jar:2.6.0:compile
[INFO] | | | \- org.apache.curator:curator-framework:jar:2.6.0:compile
[INFO] | | +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] | | +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] | | +- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] | | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | | +- org.slf4j:jul-to-slf4j:jar:1.7.26:compile
[INFO] | | +- org.slf4j:jcl-over-slf4j:jar:1.7.26:compile
[INFO] | | +- log4j:log4j:jar:1.2.17:compile
[INFO] | | +- org.slf4j:slf4j-log4j12:jar:1.7.26:compile
[INFO] | | +- com.ning:compress-lzf:jar:1.0.3:compile
[INFO] | | +- org.xerial.snappy:snappy-java:jar:1.1.7.3:compile
[INFO] | | +- org.lz4:lz4-java:jar:1.4.0:compile
[INFO] | | +- com.github.luben:zstd-jni:jar:1.3.2-2:compile
[INFO] | | +- org.roaringbitmap:RoaringBitmap:jar:0.7.45:compile
[INFO] | | | \- org.roaringbitmap:shims:jar:0.7.45:compile
[INFO] | | +- commons-net:commons-net:jar:3.1:compile
[INFO] | | +- org.json4s:json4s-jackson_2.11:jar:3.5.3:compile
[INFO] | | | \- org.json4s:json4s-core_2.11:jar:3.5.3:compile
[INFO] | | | +- org.json4s:json4s-ast_2.11:jar:3.5.3:compile
[INFO] | | | +- org.json4s:json4s-scalap_2.11:jar:3.5.3:compile
[INFO] | | | \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.6:compile
[INFO] | | +- org.glassfish.jersey.core:jersey-client:jar:2.26:compile
[INFO] | | +- org.glassfish.jersey.core:jersey-server:jar:2.26:compile
[INFO] | | | \- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.26:compile
[INFO] | | +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.26:compile
[INFO] | | +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.26:compile
[INFO] | | +- io.netty:netty:jar:3.9.9.Final:compile
[INFO] | | +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] | | +- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile
[INFO] | | +- io.dropwizard.metrics:metrics-jvm:jar:3.2.6:compile
[INFO] | | +- io.dropwizard.metrics:metrics-json:jar:3.2.6:compile
[INFO] | | +- io.dropwizard.metrics:metrics-graphite:jar:3.2.6:compile
[INFO] | | +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.9.8:compile
[INFO] | | | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.9.8:compile
[INFO] | | +- org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] | | +- oro:oro:jar:2.0.8:compile
[INFO] | | +- net.razorvine:pyrolite:jar:4.13:compile
[INFO] | | +- net.sf.py4j:py4j:jar:0.10.7:compile
[INFO] | | \- org.apache.commons:commons-crypto:jar:1.0.0:compile
[INFO] | +- org.apache.spark:spark-streaming_2.11:jar:2.4.3:compile
[INFO] | +- org.apache.spark:spark-sql_2.11:jar:2.4.3:compile
[INFO] | | +- com.univocity:univocity-parsers:jar:2.7.3:compile
[INFO] | | +- org.apache.spark:spark-sketch_2.11:jar:2.4.3:compile
[INFO] | | +- org.apache.spark:spark-catalyst_2.11:jar:2.4.3:compile
[INFO] | | | +- org.codehaus.janino:janino:jar:3.0.12:compile
[INFO] | | | +- org.codehaus.janino:commons-compiler:jar:3.0.9:compile
[INFO] | | | \- org.antlr:antlr4-runtime:jar:4.7:compile
[INFO] | | +- org.apache.orc:orc-core:jar:nohive:1.5.5:compile
[INFO] | | | +- org.apache.orc:orc-shims:jar:1.5.5:compile
[INFO] | | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | | | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | | | \- io.airlift:aircompressor:jar:0.10:compile
[INFO] | | +- org.apache.orc:orc-mapreduce:jar:nohive:1.5.5:compile
[INFO] | | +- org.apache.parquet:parquet-column:jar:1.10.1:compile
[INFO] | | | +- org.apache.parquet:parquet-common:jar:1.10.1:compile
[INFO] | | | \- org.apache.parquet:parquet-encoding:jar:1.10.1:compile
[INFO] | | +- org.apache.parquet:parquet-hadoop:jar:1.10.1:compile
[INFO] | | | +- org.apache.parquet:parquet-format:jar:2.4.0:compile
[INFO] | | | \- org.apache.parquet:parquet-jackson:jar:1.10.1:compile
[INFO] | | \- org.apache.arrow:arrow-vector:jar:0.10.0:compile
[INFO] | | +- org.apache.arrow:arrow-format:jar:0.10.0:compile
[INFO] | | +- org.apache.arrow:arrow-memory:jar:0.10.0:compile
[INFO] | | +- com.carrotsearch:hppc:jar:0.7.2:compile
[INFO] | | \- com.vlkan:flatbuffers:jar:1.2.0-3f79e055:compile
[INFO] | +- org.apache.spark:spark-graphx_2.11:jar:2.4.3:compile
[INFO] | | +- com.github.fommil.netlib:core:jar:1.1.2:compile
[INFO] | | \- net.sourceforge.f2j:arpack_combined_all:jar:0.1:compile
[INFO] | +- org.apache.spark:spark-mllib-local_2.11:jar:2.4.3:compile
[INFO] | +- org.scalanlp:breeze_2.11:jar:0.13.2:compile
[INFO] | | +- org.scalanlp:breeze-macros_2.11:jar:0.13.2:compile
[INFO] | | +- net.sf.opencsv:opencsv:jar:2.3:compile
[INFO] | | +- com.github.rwl:jtransforms:jar:2.4.0:compile
[INFO] | | +- org.spire-math:spire_2.11:jar:0.13.0:compile
[INFO] | | | +- org.spire-math:spire-macros_2.11:jar:0.13.0:compile
[INFO] | | | \- org.typelevel:machinist_2.11:jar:0.6.1:compile
[INFO] | | \- com.chuusai:shapeless_2.11:jar:2.3.2:compile
[INFO] | | \- org.typelevel:macro-compat_2.11:jar:1.1.1:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] | +- org.apache.spark:spark-tags_2.11:jar:2.4.3:compile
[INFO] | \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.glassfish.jersey.inject:jersey-hk2:jar:2.26:compile
[INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.26:compile
[INFO] | | +- javax.ws.rs:javax.ws.rs-api:jar:2.1:compile
[INFO] | | +- org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:compile
[INFO] | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
[INFO] | \- org.glassfish.hk2:hk2-locator:jar:2.5.0-b42:compile
[INFO] | +- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.5.0-b42:compile
[INFO] | +- org.glassfish.hk2:hk2-api:jar:2.5.0-b42:compile
[INFO] | | \- javax.inject:javax.inject:jar:1:compile
[INFO] | +- org.glassfish.hk2:hk2-utils:jar:2.5.0-b42:compile
[INFO] | \- org.javassist:javassist:jar:3.22.0-CR2:compile
[INFO] +- org.springframework.data:spring-data-cassandra:jar:2.0.14.RELEASE:compile
[INFO] | +- org.springframework:spring-context:jar:5.0.13.RELEASE:compile
[INFO] | +- org.springframework:spring-beans:jar:5.0.13.RELEASE:compile
[INFO] | +- org.springframework:spring-tx:jar:5.0.13.RELEASE:compile
[INFO] | +- org.springframework:spring-expression:jar:5.0.13.RELEASE:compile
[INFO] | +- org.springframework.data:spring-data-commons:jar:2.0.14.RELEASE:compile
[INFO] | +- com.datastax.cassandra:cassandra-driver-core:jar:3.4.0:compile
[INFO] | | +- io.netty:netty-handler:jar:4.1.34.Final:compile
[INFO] | | | +- io.netty:netty-common:jar:4.1.34.Final:compile
[INFO] | | | +- io.netty:netty-buffer:jar:4.1.34.Final:compile
[INFO] | | | +- io.netty:netty-transport:jar:4.1.34.Final:compile
[INFO] | | | | \- io.netty:netty-resolver:jar:4.1.34.Final:compile
[INFO] | | | \- io.netty:netty-codec:jar:4.1.34.Final:compile
[INFO] | | +- com.github.jnr:jnr-ffi:jar:2.0.7:compile
[INFO] | | | +- com.github.jnr:jffi:jar:1.2.10:compile
[INFO] | | | +- com.github.jnr:jffi:jar:native:1.2.10:runtime
[INFO] | | | +- org.ow2.asm:asm:jar:5.0.3:compile
[INFO] | | | +- org.ow2.asm:asm-commons:jar:5.0.3:compile
[INFO] | | | +- org.ow2.asm:asm-analysis:jar:5.0.3:compile
[INFO] | | | +- org.ow2.asm:asm-tree:jar:5.0.3:compile
[INFO] | | | +- org.ow2.asm:asm-util:jar:5.0.3:compile
[INFO] | | | \- com.github.jnr:jnr-x86asm:jar:1.0.2:compile
[INFO] | | \- com.github.jnr:jnr-posix:jar:3.0.27:compile
[INFO] | | \- com.github.jnr:jnr-constants:jar:0.9.0:compile
[INFO] | +- com.google.guava:guava:jar:19.0:compile
[INFO] | \- org.slf4j:slf4j-api:jar:1.7.26:compile
[INFO] \- com.datastax.spark:spark-cassandra-connector-unshaded_2.11:jar:2.4.0:compile
[INFO] +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] +- com.datastax.cassandra:cassandra-driver-mapping:jar:3.4.0:compile
[INFO] +- commons-beanutils:commons-beanutils:jar:1.9.3:compile
[INFO] | \- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] +- org.joda:joda-convert:jar:1.2:compile
[INFO] +- joda-time:joda-time:jar:2.9.9:compile
[INFO] +- io.netty:netty-all:jar:4.1.34.Final:compile
[INFO] +- com.twitter:jsr166e:jar:1.1.0:compile
[INFO] \- org.scala-lang:scala-reflect:jar:2.11.12:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
推荐答案
似乎依赖性之间存在冲突，但是不知道什么！这就是我所做的： Spring-Spark：罐子/依赖项之间的冲突
It seems there was a conflict between dependencies but do not know exaclty what! This is what I did: Spring - Spark: Conflicts between jars/dependencies
编辑：
正如上面链接中所述，我只需要更改此设置：
As it also explains in the link above, I just had to change this:
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.0.9.RELEASE</version>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
<configuration>
<requiresUnpack>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
</dependency>
</requiresUnpack>
</configuration>
</plugin>
其中：
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.1</version>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.0.9.RELEASE</version>
</dependency>
</dependencies>
<configuration>
<keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope>
<createDependencyReducedPom>true</createDependencyReducedPom>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer
implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
<resource>META-INF/spring.factories</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.example.app.Mtbapp1</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
这篇关于如何修复“ ClassCastException：无法分配实例”-在本地工作，但不能在群集上独立运行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
