如何将多个语句传递到 Spark SQL HiveContext [英] How to pass multiple statements into Spark SQL HiveContext

问题描述

例如，我想传递给 Spark SQL 的 Hive HQL 语句很少:

For example I have few Hive HQL statements which I want to pass into Spark SQL:

set parquet.compression=SNAPPY;
create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE;
select * from MY_TABLE limit 5;

以下不起作用:

hiveContext.sql("set parquet.compression=SNAPPY; create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE; select * from MY_TABLE limit 5;")

如何将语句传递到 Spark SQL 中?

How to pass the statements into Spark SQL?

推荐答案

感谢@SamsonScharfrichter 的回答.

Thank you to @SamsonScharfrichter for the answer.

这会起作用:

hiveContext.sql("set spark.sql.parquet.compression.codec=SNAPPY")
hiveContext.sql("create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE")
val rs = hiveContext.sql("select * from MY_TABLE limit 5")

请注意，在这种特殊情况下，我们需要使用 spark.sql.parquet.compression.codec

Please note that in this particular case instead of parquet.compression key we need to use spark.sql.parquet.compression.codec

这篇关于如何将多个语句传递到 Spark SQL HiveContext的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！