我可以在spark 2.2.*独立集群上运行spark 2.0.*工件吗? [英] Can I run spark 2.0.* artifact on a spark 2.2.* stand-alone cluster?

问题描述

我知道以下事实:随着spark主要版本的更改(即从1. *更改为2. *)，由于现有API的更改会导致编译时失败.

I am aware of the fact that with the change of major version of spark (i.e. from 1.* to 2.*) there will be compile time failures due to changes in existing APIs.

据我所知，火花保证使用次要版本更新(即2.0.*至2.2.*)，更改将向后兼容.

As per my knowledge spark guarantees that with minor version update (i.e. 2.0.* to 2.2.*), changes will be backward compatible.

尽管这将消除升级时编译时失败的可能性，但是可以安全地假设如果在Spark 2.2.*上的独立集群中使用工件提交作业，也不会出现任何运行时失败(jar)是使用2.0.*依赖项创建的?

Although this will eliminate the possibility of compile-time failures with upgrade, would it be safe to assume that there won't be any run time failure too if submit a job on spark 2.2.* stand alone cluster using an artifact(jar) created using 2.0.* dependencies?

推荐答案

如果使用通过2.0.*依赖关系创建的工件(jar)在2.2.*集群上提交作业，是否可以安全地假设运行时也不会出现故障?

would it be safe to assume that there won't be any run time failure too if submit a job on 2.2.* cluster using an artifact(jar) created using 2.0.* dependencies?

是的

我什至会说，除非我们谈论内置的Spark Standalone集群，否则没有Spark集群的概念.

I'd even say that there's no concept of a Spark cluster unless we talk about the built-in Spark Standalone cluster.

换句话说，您将Spark应用程序部署到集群中，例如Hadoop YARN或Apache Mesos，作为可能包含或不包含Spark jar的应用程序jar，因此无视环境中已经可用的内容.

In other words, you deploy a Spark application to a cluster, e.g. Hadoop YARN or Apache Mesos, as a application jar that may or may not contain Spark jars and so disregard what's already available in the environment.

但是，如果您确实想到了Spark Standalone，则可能会在2.0和2.2之间的版本之间发生问题，因为Spark应用程序中的jar必须与Spark worker的JVM上的jar兼容(它们已经预先加载了)).

If however you do think of Spark Standalone, things may have been broken between releases even between 2.0 and 2.2 as the jars in your Spark application have to be compatible with the ones on JVM of Spark workers (they are already pre-loaded).

我不声称Spark Standalone的发行版之间具有完全兼容性.

I would not claim full compatibility between releases of Spark Standalone.

这篇关于我可以在spark 2.2.*独立集群上运行spark 2.0.*工件吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！