Spark错误:NoSuchMethodError:scala.Predef $.$ conforms()Lscala/Predef $ less $ colon $ less [英] Error with Spark: NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less

问题描述

我有一个应用程序可以流式处理Twitter帖子. 它可以正常工作，但是在修改了构建路径之后，我没有设法将其设置为以前的样子.我不断收到此错误:

I've got an application which streams Twitter posts. It worked correctly, but after modifying the build path, I didn't manage to set it how it was before. I keep getting this error:

Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getSystemProperties(Utils.scala:1710)
at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:73)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:68)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:55)
at Analytics.init(Analytics.java:20)
at KafkaTwitterProducer.main(KafkaTwitterProducer.java:162)

这是我的pom.xml

This is my pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>TwitterStreamingAnalyzer</groupId>
  <artifactId>TwitterStreamingAnalyzer</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <build>
    <sourceDirectory>src</sourceDirectory>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.5.1</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <dependencies>
   <dependency>  
                <groupId>org.apache.kafka</groupId>  
                <artifactId>kafka_2.10</artifactId>  
                <version>0.10.2.1</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-core -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-core</artifactId>  
                <version>4.0.4</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-stream -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-stream</artifactId>  
                <version>4.0.4</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-async -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-async</artifactId>  
                <version>4.0.4</version>  
           </dependency> 
           
     </dependencies> 
</project>

我已经阅读了所有类似的问题，但是没有一个问题对我有所帮助，因为我尝试导入正确版本的jars，但仍然无法正常工作. 我该如何解决?

I already read all of the similar questions, but none of them helped me, because I tried to import the right versions of the jars, but it still doesn't work. How can I fix this?

推荐答案

看起来您正在尝试在其他版本(可能是2.11)编译的Spark集群中使用Scala 2.10中编译的kafka库.在您的pom.xml中尝试以下更改:

Looks like you are trying to use a kafka library compiled in Scala 2.10 in a Spark cluster compiled in other version (probably 2.11). Try this change in your pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>TwitterStreamingAnalyzer</groupId>
  <artifactId>TwitterStreamingAnalyzer</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <build>
    <sourceDirectory>src</sourceDirectory>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.5.1</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <dependencies>
   <dependency>  
                <groupId>org.apache.kafka</groupId>  
                <artifactId>kafka_2.11</artifactId>  
                <version>0.10.2.1</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-core -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-core</artifactId>  
                <version>4.0.4</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-stream -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-stream</artifactId>  
                <version>4.0.4</version>  
           </dependency>  
           <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-async -->  
           <dependency>  
                <groupId>org.twitter4j</groupId>  
                <artifactId>twitter4j-async</artifactId>  
                <version>4.0.4</version>  
           </dependency> 

     </dependencies> 
</project>

这篇关于Spark错误:NoSuchMethodError:scala.Predef $.$ conforms()Lscala/Predef $ less $ colon $ less的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！