如何使用 Scala 从 Spark 更新 ORC Hive 表 [英] How to Updata an ORC Hive table form Spark using Scala

问题描述

我想更新一个 orc 格式的 hive 表，我可以从我的 ambari hive 视图更新，但无法从 sacla (spark-shell) 运行相同的更新语句

I would like to update a hive table which is in orc format , I'm able to update from my ambari hive view, but unable to run same update statement from sacla (spark-shell)

objHiveContext.sql("select * from table_name ") 能够看到数据，但是当我运行时

objHiveContext.sql("select * from table_name ") able to see data but when I run

objHiveContext.sql("update table_name set column_name='testing'") 无法运行，发生了一些 Noviable 异常(更新附近的无效语法等)，因为我可以从 Ambari 视图更新(因为我设置了所有所需的配置，即 TBLPROPERTIES "orc.compress"="NONE" transactional true 等)

objHiveContext.sql("update table_name set column_name='testing' ") unable to run , some Noviable exception(Invalid syntax near update etc) is occurring where as I'm able to update from Ambari view(As I set all the required configurations i.e TBLPROPERTIES "orc.compress"="NONE" transactional true etc)

尝试将 Insert into using case 语句和所有但不能我们可以从 spark 更新 hive ORC 表吗?如果是，那么程序是什么?

Tried with Insert into using case statements and all but couldn't Can we UPDATE hive ORC tables from spark? If yes then what is the procedure ?

下方导入

import org.apache.spark.SparkConf
import org.apache.spark.SparkConf
import org.apache.spark._
import org.apache.spark.sql._
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.orc._

注意:我没有在那个表上应用任何分区或分桶如果我应用分桶，我什至无法在存储为 ORC 时查看数据蜂巢版本:1.2.1星火版本:1.4.1斯卡拉版本:2.10.6

Note: I didn't apply any partition or bucketing on that table If I apply bucketing I'm even unable to view data when stored as ORC Hive Version:1.2.1 Spark version:1.4.1 Scala Version :2.10.6

推荐答案

您是否尝试过使用 SaveMode.Append 根据以下链接使用 DataFrame.write API?

Have you tried the DataFrame.write API using SaveMode.Append per the link below?

http://spark.apache.org/docs/latest/sql-programming-guide.html#manually-specifying-options

使用orc"作为格式，使用append"作为保存模式.示例在上面的链接中.

use "orc" as the format and "append" as the save mode. examples are in that link above.

这篇关于如何使用 Scala 从 Spark 更新 ORC Hive 表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！