Spark异步作业因错误而失败 [英] Spark asynchronous job fails with error

问题描述

我正在用Java编写spark的代码.当我使用foreachAsync时，火花会失败，并给我java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.

I'm writing code for spark in java. When I use foreachAsync spark fails and gives me java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.

在此代码中:

JavaSparkContext sparkContext = new JavaSparkContext("local","MyAppName");
    JavaPairRDD<String, String> wholeTextFiles = sparkContext.wholeTextFiles("somePath");
    wholeTextFiles.foreach(new VoidFunction<Tuple2<String, String>>() {
        public void call(Tuple2<String, String> stringStringTuple2) throws Exception {
            //do something
        }
    });

工作正常.但是在这段代码中:

It works fine. But in this code:

JavaSparkContext sparkContext = new JavaSparkContext("local","MyAppName");
    JavaPairRDD<String, String> wholeTextFiles = sparkContext.wholeTextFiles("somePath");

    wholeTextFiles.foreachAsync(new VoidFunction<Tuple2<String, String>>() {
        public void call(Tuple2<String, String> stringStringTuple2) throws Exception {
            //do something
        }
    });

它返回错误.我哪里错了?

It returns error. Where I'm wrong?

推荐答案

这是因为foreachAsync返回一个Future对象，并且当您离开函数时，spark上下文已关闭(因为它是在本地创建的).

It's because foreachAsync returns a Future object and when you leave a function, the spark context is closed (because it's created locally).

如果在foreachAsync()上调用get()，则主线程将等待Future完成.

If you call get() on foreachAsync() then main thread will wait for the Future to complete.

这篇关于Spark异步作业因错误而失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！