createOrReplaceTempView 在 Spark 中是如何工作的? [英] How does createOrReplaceTempView work in Spark?
问题描述
我是 Spark 和 Spark SQL 的新手.
I am new to Spark and Spark SQL.
createOrReplaceTempView
在 Spark 中是如何工作的?
How does createOrReplaceTempView
work in Spark?
如果我们将对象的 RDD
注册为表,是否会将所有数据保存在内存中?
If we register an RDD
of objects as a table will spark keep all the data in memory?
推荐答案
createOrReplaceTempView
创建(或替换,如果该视图名称已经存在)一个懒惰评估的视图",然后您可以像蜂巢一样使用它Spark SQL 中的表.除非您缓存支持视图的数据集,否则它不会 持久化到内存中.
createOrReplaceTempView
creates (or replaces if that view name already exists) a lazily evaluated "view" that you can then use like a hive table in Spark SQL. It does not persist to memory unless you cache the dataset that underpins the view.
scala> val s = Seq(1,2,3).toDF("num")
s: org.apache.spark.sql.DataFrame = [num: int]
scala> s.createOrReplaceTempView("nums")
scala> spark.table("nums")
res22: org.apache.spark.sql.DataFrame = [num: int]
scala> spark.table("nums").cache
res23: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [num: int]
scala> spark.table("nums").count
res24: Long = 3
数据仅在 .count
调用后才被完全缓存.这是它已被缓存的证据:
The data is cached fully only after the .count
call. Here's proof it's been cached:
相关 SO:spark createOrReplaceTempView vs createGlobalTempView
相关引用(与持久表相比):与 createOrReplaceTempView 命令不同,saveAsTable 将具体化 DataFrame 的内容并创建一个指向 Hive 元存储中数据的指针."来自 https://spark.apache.org/docs/latest/sql-programming-guide.html#saving-to-persistent-tables
Relevant quote (comparing to persistent table): "Unlike the createOrReplaceTempView command, saveAsTable will materialize the contents of the DataFrame and create a pointer to the data in the Hive metastore." from https://spark.apache.org/docs/latest/sql-programming-guide.html#saving-to-persistent-tables
注意:createOrReplaceTempView
以前是 registerTempTable
这篇关于createOrReplaceTempView 在 Spark 中是如何工作的?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!