Spark 2.3.0 SQL无法将数据插入Hive HBase表 [英] Spark 2.3.0 SQL unable insert data into hive hbase table

问题描述

使用与hive 2.2.0集成的Spark 2.3 thriftserver.从火花beeline运行.尝试将数据插入Hive HBase表(以HBase为存储的Hive表)中.插入到蜂巢本机表就可以了.插入配置单元hbase表时，它将引发以下异常:

Use Spark 2.3 thriftserver integrated with hive 2.2.0. running from spark beeline. Try to insert data into hive hbase table (a hive table with hbase as storage). Insert into hive native table is ok. When inserting into hive hbase table, it throws the following exception:

ClassCastException:org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat无法转换为org.apache.hadoop.hive.ql.io.HiveOutputFormat

ClassCastException: org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.HiveOutputFormat

在org.apache.spark.scheduler.Task.run(Task.scala:109) 在org.apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:345) 在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 在java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:624) 在java.lang.Thread.run(Thread.java:748) 由以下原因引起:java.lang.ClassCastException:org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat无法转换为org.apache.hadoop.hive.ql.io.HiveOutputFormat 在org.apache.spark.sql.hive.execution.HiveFileFormat $$ anon $ 1.outputFormat $ lzycompute(HiveFileFormat.scala:93) k在写入行时失败. 在org.apache.spark.sql.execution.datasources.FileFormatWriter $ .org $ apache $ spark $ sql $ execution $ datasources $ FileFormatWriter $$ executeTask(FileFormatWriter.scala:285) 在org.apache.spark.sql.execution.datasources.FileFormatWriter $$ anonfun $ write $ 1.apply(FileFormatWriter.scala:197)处 在org.apache.spark.sql.execution.datasources.FileFormatWriter $$ anonfun $ write $ 1.apply(FileFormatWriter.scala:196) 在org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) 在org.apache.spark.scheduler.Task.run(Task.scala:109) 在org.apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:345) 在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 在java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:624) 在java.lang.Thread.run(Thread.java:748) 由以下原因引起:java.lang.ClassCastException:org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat无法转换为org.apache.hadoop.hive.ql.io.HiveOutputFormat 在org.apache.spark.sql.hive.execution.HiveFileFormat $$ anon $ 1.outputFormat $ lzycompute(HiveFileFormat.scala:93)

at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.HiveOutputFormat at org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.outputFormat$lzycompute(HiveFileFormat.scala:93) k failed while writing rows. at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:285) at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197) at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.HiveOutputFormat at org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.outputFormat$lzycompute(HiveFileFormat.scala:93)

推荐答案

似乎是一个已知的蜂巢问题.有几个请求请求可以解决它，但我还没有找到实际的解决方法.

Seems to be a known hive issue. There are several pull requests addressing it but I haven't found an actual fix yet.

https://issues.apache.org/jira/browse/SPARK-6628

这篇关于Spark 2.3.0 SQL无法将数据插入Hive HBase表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！