如何将 Spark DataFrame 插入 Hive 内部表? [英] How to insert Spark DataFrame to Hive Internal table?

问题描述

在追加模式下将 DF 插入 Hive 内部表的正确方法是什么.似乎我们可以使用saveAsTable"方法直接将 DF 写入 Hive 或将 DF 存储到临时表然后使用查询.

What's the right way to insert DF to Hive Internal table in Append Mode. It seems we can directly write the DF to Hive using "saveAsTable" method OR store the DF to temp table then use the query.

df.write().mode("append").saveAsTable("tableName")

或

df.registerTempTable("temptable") 
sqlContext.sql("CREATE TABLE IF NOT EXISTS mytable as select * from temptable")

第二种方法是追加记录还是覆盖记录?

Will the second approach append the records or overwrite it?

有没有其他方法可以有效地将 DF 写入 Hive 内部表?

Is there any other way to effectively write the DF to Hive Internal table?

推荐答案

df.saveAsTable("tableName", "append") 已弃用.相反，您应该采用第二种方法.

df.saveAsTable("tableName", "append") is deprecated. Instead you should the second approach.

sqlContext.sql("CREATE TABLE IF NOT EXISTS mytable as select * from temptable")

如果表不存在，它将创建表.当您第二次运行代码时，您需要删除现有表，否则您的代码将异常退出.

It will create table if the table doesnot exist. When you will run your code second time you need to drop the existing table otherwise your code will exit with exception.

另一种方法，如果您不想删除表.单独创建一个表，然后将您的数据插入该表中.

Another approach, If you don't want to drop table. Create a table separately, then insert your data into that table.

下面的代码将数据附加到现有表中

The below code will append data into existing table

sqlContext.sql("insert into table mytable select * from temptable")

下面的代码会将数据覆盖到现有表中

And the below code will overwrite the data into existing table

sqlContext.sql("insert overwrite table mytable select * from temptable")

此答案基于 Spark 1.6.2.如果您使用其他版本的 Spark，我建议您查看相应的文档.

This answer is based on Spark 1.6.2. In case you are using other version of Spark I would suggests to check the appropriate documentation.

这篇关于如何将 Spark DataFrame 插入 Hive 内部表?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！