火花createOrReplaceTempView与createGlobalTempView [英] spark createOrReplaceTempView vs createGlobalTempView

问题描述

Spark Dataset 2.0提供了两个功能createOrReplaceTempView和createGlobalTempView.我无法理解这两个功能之间的基本区别.

Spark Dataset 2.0 provides two functions createOrReplaceTempView and createGlobalTempView. I am not able to understand the basic difference between both functions.

根据 API文档:

createOrReplaceTempView :此操作的生命周期 临时视图与用于创建此数据集的[[SparkSession]]绑定在一起.
因此，当我调用sparkSession.close()时，定义将被销毁.是真的吗?

createOrReplaceTempView: The lifetime of this temporary view is tied to the [[SparkSession]] that was used to create this Dataset.
So, when I call sparkSession.close() the defined will be destroyed. is it true?

createGlobalTempView :此临时视图的生存期与此Spark应用程序相关.

createGlobalTempView: The lifetime of this temporary view is tied to this Spark application.

这种视图何时会被破坏?任何例子.像sparkSession.close()?

when this type of view will be destroyed? any example. like sparkSession.close()?

推荐答案

df.createOrReplaceTempView("tempViewName")
df.createGlobalTempView("tempViewName")

createOrReplaceTempView()使用此数据框df创建或替换本地临时视图.此视图的生命周期取决于SparkSession类，是否要删除此视图:

createOrReplaceTempView() creates or replaces a local temporary view with this dataframe df. Lifetime of this view is dependent to SparkSession class, is you want to drop this view :

spark.catalog.dropTempView("tempViewName")

或stop()将关闭会话

self.ss = SparkSession(sc)
...
self.ss.stop()

createGlobalTempView()使用此数据框df创建一个全局临时视图.该视图的寿命取决于spark应用程序本身. 如果您想放下:

createGlobalTempView() creates a global temporary view with this dataframe df. life time of this view is dependent to spark application itself. If you want to drop :

spark.catalog.dropGlobalTempView("tempViewName")

或stop()将关闭

ss =  SparkContext(conf=conf, ......)
...
ss.stop()

这篇关于火花createOrReplaceTempView与createGlobalTempView的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！