如何检查DataFrame之前是否已被缓存/持久化? [英] How to check if a DataFrame was already cached/persisted before?

问题描述

对于spark的RDD对象，这很简单，因为它公开了getStorageLevel方法，但是DF似乎没有公开任何类似的东西.有人吗?

For spark's RDD object this is quite trivial as it exposes a getStorageLevel method, but DF does not seem to expose anything similar. anyone?

推荐答案

您可以使用 Catalog(org.apache.spark.sql.catalog.Catalog)检查数据帧是否已缓存.包含在Spark 2中.

You can check weather a DataFrame is cached or not using Catalog (org.apache.spark.sql.catalog.Catalog) which comes in Spark 2.

代码示例:

  val sparkSession = SparkSession.builder.
      master("local")
      .appName("example")
      .getOrCreate()

    val df = sparkSession.read.csv("src/main/resources/sales.csv")
    df.createTempView("sales")

    //interacting with catalog

    val catalog = sparkSession.catalog

    //print the databases

    catalog.listDatabases().select("name").show()

    // print all the tables

    catalog.listTables().select("name").show()

    // is cached
    println(catalog.isCached("sales"))
    df.cache()
    println(catalog.isCached("sales"))

使用上述代码，您可以列出所有表，并检查表是否被缓存.

Using the above code you can list all the tables and check weather a table is cached or not.

您可以检查工作代码示例

You can check the working code example here

这篇关于如何检查DataFrame之前是否已被缓存/持久化?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！