在同一JVM中检测到多个SparkContext [英] Multiple SparkContext detected in the same JVM

问题描述

根据我的最后一个问题我必须为我定义Multiple SparkContext独特的JVM。

according my last question I have to define the Multiple SparkContext for my unique JVM.

我是用下一种方式做的（使用Java）：

I did it in the next way (using Java):

SparkConf conf = new SparkConf();
conf.setAppName("Spark MultipleContest Test");
conf.set("spark.driver.allowMultipleContexts", "true");
conf.setMaster("local");

之后我创建了下一个源代码：

After that I create the next source code:

SparkContext sc = new SparkContext(conf);
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);

以及稍后的代码：

JavaSparkContext ctx = new JavaSparkContext(conf);
JavaRDD<Row> testRDD = ctx.parallelize(AllList);

代码执行后我收到下一条错误消息：

After the code executing I got next error message:

16/01/19 15:21:08 WARN SparkContext: Multiple running SparkContexts detected in the same JVM!
org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:
org.apache.spark.SparkContext.<init>(SparkContext.scala:81)
test.MLlib.BinarryClassification.main(BinaryClassification.java:41)
    at org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$1.apply(SparkContext.scala:2083)
    at org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$1.apply(SparkContext.scala:2065)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:2065)
    at org.apache.spark.SparkContext$.setActiveContext(SparkContext.scala:2151)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:2023)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
    at test.MLlib.BinarryClassification.main(BinaryClassification.java:105)

数字 41 和 105 是这些行，其中两个对象都在Java代码中定义。我的问题是，如果我已经使用 set -method，是否可以在同一个JVM上执行多个SparkContext以及如何执行它？

The numbers 41 and 105 are the lines, where both objects are defined in Java code. My question is, is it possible to execute multiple SparkContext on the same JVM and how to do it, if I already use the set-method ?

推荐答案

您确定需要将JavaSparkContext作为单独的上下文吗？你提到的上一个问题并没有这么说。如果您已有Spark Context，则可以从中创建新的JavaSparkContext，而不是创建单独的上下文：

Are you sure you need the JavaSparkContext as a separate context? The previous question that you refer to doesn't say so. If you already have a Spark Context you can create a new JavaSparkContext from it, rather than create a separate context:

SparkConf conf = new SparkConf();
conf.setAppName("Spark MultipleContest Test");
conf.set("spark.driver.allowMultipleContexts", "true");
conf.setMaster("local");

SparkContext sc = new SparkContext(conf);
SQLContext sqlContext = new org.apache.spark.sql.SQLContext(sc);

//Create a Java Context which is the same as the scala one under the hood
JavaSparkContext.fromSparkContext(sc)

这篇关于在同一JVM中检测到多个SparkContext的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！