如何使用其架构从 Spark 数据框创建 hive 表? [英] How to create hive table from Spark data frame, using its schema?

问题描述

我想使用 Spark 数据框的架构创建一个 hive 表.我该怎么做?

I want to create a hive table using my Spark dataframe's schema. How can I do that?

对于固定列，我可以使用:

For fixed columns, I can use:

val CreateTable_query = "Create Table my table(a string, b string, c double)"
sparksession.sql(CreateTable_query) 

但是我的数据框中有很多列，那么有没有办法自动生成这样的查询?

But I have many columns in my dataframe, so is there a way to automatically generate such query?

推荐答案

假设您使用的是 Spark 2.1.0 或更高版本，并且 my_DF 是您的数据框，

Assuming, you are using Spark 2.1.0 or later and my_DF is your dataframe,

//get the schema split as string with comma-separated field-datatype pairs
StructType my_schema = my_DF.schema();
String columns = Arrays.stream(my_schema.fields())
                       .map(field -> field.name()+" "+field.dataType().typeName())
                       .collect(Collectors.joining(","));

//drop the table if already created
spark.sql("drop table if exists my_table");
//create the table using the dataframe schema
spark.sql("create table my_table(" + columns + ") 
    row format delimited fields terminated by '|' location '/my/hdfs/location'");
    //write the dataframe data to the hdfs location for the created Hive table
    my_DF.write()
    .format("com.databricks.spark.csv")
    .option("delimiter","|")
    .mode("overwrite")
    .save("/my/hdfs/location");

另一种使用临时表的方法

The other method using temp table

my_DF.createOrReplaceTempView("my_temp_table");
spark.sql("drop table if exists my_table");
spark.sql("create table my_table as select * from my_temp_table");

这篇关于如何使用其架构从 Spark 数据框创建 hive 表?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！