如何使用 Spark 对象获取 Hive 表的位置值? [英] How to get the value of the location for a Hive table using a Spark object?

问题描述

我希望能够检索给定 Spark 对象 (SparkSession) 的 Hive 表的 location 值.获取此值的一种方法是通过以下 SQL 查询解析位置的输出:

I am interested in being able to retrieve the location value of a Hive table given a Spark object (SparkSession). One way to obtain this value is by parsing the output of the location via the following SQL query:

describe formatted <table name>

我想知道是否有另一种方法无需解析输出即可获得 location 值.如果上述命令的输出在 Hive 版本之间发生变化，API 会很棒.如果需要外部依赖，它会是哪个?是否有一些可以获取位置值的示例火花代码?

I was wondering if there is another way to obtain the location value without having to parse the output. An API would be great in case the output of the above command changes between Hive versions. If an external dependency is needed, which would it be? Is there some sample spark code that can obtain the location value?

推荐答案

第一种方法

您可以将 input_file_name 与数据框一起使用.

You can use input_file_name with dataframe.

它将为您提供零件文件的绝对文件路径.

it will give you absolute file-path for a part file.

spark.read.table("zen.intent_master").select(input_file_name).take(1)

然后从中提取表路径.

第二种方法

你可以说它更像是黑客.

Its more of hack you can say.

package org.apache.spark.sql.hive

import java.net.URI

import org.apache.spark.sql.catalyst.catalog.{InMemoryCatalog, SessionCatalog}
import org.apache.spark.sql.catalyst.parser.ParserInterface
import org.apache.spark.sql.internal.{SessionState, SharedState}
import org.apache.spark.sql.SparkSession

class TableDetail {
  def getTableLocation(table: String, spark: SparkSession): URI = {
    val sessionState: SessionState = spark.sessionState
    val sharedState: SharedState = spark.sharedState
    val catalog: SessionCatalog = sessionState.catalog
    val sqlParser: ParserInterface = sessionState.sqlParser
    val client = sharedState.externalCatalog match {
      case catalog: HiveExternalCatalog => catalog.client
      case _: InMemoryCatalog => throw new IllegalArgumentException("In Memory catalog doesn't " +
        "support hive client API")
    }

    val idtfr = sqlParser.parseTableIdentifier(table)

    require(catalog.tableExists(idtfr), new IllegalArgumentException(idtfr + " done not exists"))
    val rawTable = client.getTable(idtfr.database.getOrElse("default"), idtfr.table)
    rawTable.location
  }
}

这篇关于如何使用 Spark 对象获取 Hive 表的位置值?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！