无法从Apache Spark SQL 1.5.2在SQLContext中运行查询，获取java.lang.NoSuchMethodError [英] Cannot run queries in SQLContext from Apache Spark SQL 1.5.2, getting java.lang.NoSuchMethodError

问题描述

我有一个使用 Spark SQL (使用 local模式的 Spark 1.5.2 )的Java应用程序，但是如果没有该命令，我将无法执行任何SQL命令出错.

I have a Java application using Spark SQL (Spark 1.5.2 using local mode), but I cannot execute any SQL commands without getting errors.

这是我正在执行的代码:

This is the code I am executing:

//confs
SparkConf sparkConf = new SparkConf();  
sparkConf.set("spark.master","local");
sparkConf.set("spark.app.name","application01");
sparkConf.set("spark.driver.host","10.1.1.36");
sparkConf.set("spark.driver.port", "51810");
sparkConf.set("spark.executor.port", "51815");
sparkConf.set("spark.repl.class.uri","http://10.1.1.36:46146");
sparkConf.set("spark.executor.instances","2");
sparkConf.set("spark.jars","");
sparkConf.set("spark.executor.id","driver");
sparkConf.set("spark.submit.deployMode","client");
sparkConf.set("spark.fileserver.uri","http://10.1.1.36:47314");
sparkConf.set("spark.localProperties.clone","true");
sparkConf.set("spark.app.id","app-45631207172715-0002");

//Initialize contexts
JavaSparkContext sparkContext = new JavaSparkContext(sparkConf);
SQLContext sqlContext = new SQLContext(sparkContext);           

//execute command
sqlContext.sql("show tables").show();

pom.xml 中的火花依赖性如下:

Spark dependencies in pom.xml look like this:

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.10</artifactId>
  <version>1.5.2</version>
</dependency>

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql_2.10</artifactId>
  <version>1.5.2</version>
</dependency>

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-hive_2.10</artifactId>
  <version>1.5.2</version>
</dependency>

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-repl_2.10</artifactId>
  <version>1.5.2</version>
</dependency>

这是我遇到的错误:

java.lang.NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer$.handledType()Ljava/lang/Class;

堆栈跟踪为此处.

我的应用程序是在Tomcat 7上运行的Web应用程序.我没有任何其他配置文件.我可能做错了什么?既然我能够在一个干净的项目中运行相同的代码，这可能是一些依赖冲突吗?

My application is a web application running on Tomcat 7. I don't have any other configuration files. What could I be doing wrong? Could it be some dependency conflict, since I am able to run the same code in a clean project?

编辑:我发现了一个问题，提供了有关该问题的更多信息.

EDIT: I found an issue that gives some more information about the problem.

推荐答案

直到2.4之前，BigDecimalDeserializer才引入FasterXML/jackson-module-scala.确认以下内容:

BigDecimalDeserializer wasn't introduced to FasterXML/jackson-module-scala until 2.4. Confirm the following:

1. 在运行时，使用相同的jar进行编译的类路径上.
pom.xml文件中的
2. ${fasterxml.jackson.version} Spark SQL 为2.4.x或更高版本.

1. The same jars you compile with are on the classpath at runtime.
2. ${fasterxml.jackson.version} in the pom.xml file for Spark SQL is 2.4.x or greater.

<dependency>
  <groupId>com.fasterxml.jackson.core</groupId>
  <artifactId>jackson-databind</artifactId>
  <version>2.4.4</version>
</dependency>

这篇关于无法从Apache Spark SQL 1.5.2在SQLContext中运行查询，获取java.lang.NoSuchMethodError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！