星火的Java Appilcation：抛出java.lang.ClassNotFoundException [英] Spark Java Appilcation : java.lang.ClassNotFoundException

问题描述

我已经创建了使用Java的Apache一个应用星火。它所做的就是计算包含火花字1000倍线。

I've created a Apache Spark application using Java. All it does is counting the lines containing the "spark" word 1000 times.

下面是我的code：

public class Example1 {
    public static void main(String[] args) {
        String logfile = args[0];
        try{
            SparkConf conf = new SparkConf();
            conf.setAppName("Sample");
            conf.setMaster("spark://<master>:7077");
            conf.set("spark.executor.memory", "1g");
            JavaSparkContext sc = new JavaSparkContext(conf);
            JavaRDD<String> logData = sc.textFile(logfile).cache();
            long count = 0;
            for(int i=0; i<=1000; i++){
                count += logData.filter(new Function<String, Boolean>(){
                    public Boolean call(String s){
                        if (s.toLowerCase().contains("spark"))
                            return true;
                        else
                            return false;
                    }
                }).count();
            }
        }
        catch(Exception ex){
            System.out.println(ex.getMessage());
        }
    }
}

当我在Eclipse IDE进行调试，我遇到抛出java.lang.ClassNotFoundException ：

When I perform a debug in Eclipse IDE, I am encountering java.lang.ClassNotFoundException:

WARN scheduler.TaskSetManager: Loss was due to java.lang.ClassNotFoundException
java.lang.ClassNotFoundException: org.spark.java.examples.Example1$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)

我也试图使用群集内部署该火花提交，不过，遇到相同的异常。这里的堆栈跟踪的一部分：

I also tried to deploy this inside the cluster using spark-submit, but still, the same exception was encountered. Here's a portion of the stacktrace:

ERROR Executor: Exception in task ID 4
java.lang.ClassNotFoundException: org.spark.java.examples.Example1$1
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)

这是如何解决这个任何想法？在此先感谢！

Any ideas on how to resolve this? Thanks in advance!

推荐答案

您需要与你的工作给工人带来的罐子。要做到这一点，已经Maven构建一个罐子，罐子即添加到背景：

You need to deliver the jar with your job to the workers. To do that, have maven build a jar and add that jar to the context:

 conf.setJars(new String[]{"path/to/jar/Sample.jar"}); [*]

对于一个真正的工作，你将需要建立与依赖（Maven的检查灯罩插件）一罐，但对于没有外部的依赖一个简单的工作，一个简单的罐子就足够了。

For a 'real' job you would need to build a jar with dependencies (check Maven shade plugin), but for a simple job with no external dependencies, a simple jar is sufficient.

[*]我不是很熟悉星火的Java API，只是假设它应该是这样的。

[*] I'm not very familiar with the Spark java API, just assuming it should be something like this.

这篇关于星火的Java Appilcation：抛出java.lang.ClassNotFoundException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！