如何分享2星火上下文之间星火RDD？ [英] How to share Spark RDD between 2 Spark contexts?

问题描述

我有一个RMI集群。每个RMI服务器有一个Spark上下文。
有没有什么办法不一样的火花上下文之间共享一个RDD？

I have an RMI cluster. Each RMI server has a Spark context. Is there any way to share an RDD between different Spark contexts?

推荐答案

正如丹尼尔Darabos已经指出这是不可能的。在星火每一个分布式对象为界，已应用于RDD的情况下创建它（ SparkContext 具体情况下， SQLContext 在数据帧数据集的情况下）。如果你想，你必须使用共享上下文应用程序之间共享对象（例如，见 火花jobserver ，<一个href=\"https://github.com/cloudera/hue/tree/master/apps/spark/java#welcome-to-livy-the-rest-spark-server\"相对=nofollow>李维，或阿帕奇齐柏林）。由于 RDD 或数据帧只是一个小地方的对象实在没有太多的交流。

As already stated by Daniel Darabos it is not possible. Every distributed object in Spark is bounded to specific context which has been used to create it (SparkContext in case of RDD, SQLContext in case of DataFrame dataset). If you want share objects between applications you have to use shared contexts (see for example spark-jobserver, Livy, or Apache Zeppelin). Since RDD or DataFrame is just a small local object there is really not much to share.

共享数据是一个完全不同的问题。您可以使用专门的内存缓存（阿帕奇点燃），或分布在内存中的文件系统（如 Alluxio - 前者的Tachyon）应用程序之间切换时，但你真的不能避免，尽量减少时延

Sharing data is a completely different problem. You can use specialized in memory cache (Apache Ignite) or distributed in memory file systems (like Alluxio - former Tachyon) to minimize the latency when switching between application but you cannot really avoid it.

这篇关于如何分享2星火上下文之间星火RDD？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！