如何在SparkSQL中以编程方式连接到Hive Metastore？ [英] How to connect to a Hive metastore programmatically in SparkSQL?

问题描述

我将HiveContext与SparkSQL结合使用，并试图连接到远程Hive Metastore，设置Hive Metastore的唯一方法是将hive-site.xml包含在类路径中（或将其复制到/ etc / spark / conf /）。

是否有一种方法可以在不包含hive-site.xml的情况下以编程方式设置此参数？如果是这样的话，Spark配置是如何使用的？

解决方案

对于Spark 1.x，您可以设置：

  System.setProperty（hive.metastore.uris，thrift：// METASTORE：9083）; 
 
 final SparkConf conf = new SparkConf（）; 
 SparkContext sc = new SparkContext（conf）; 
 HiveContext hiveContext = new HiveContext（sc）; 
  

或

  final SparkConf conf = new SparkConf（）; 
 SparkContext sc = new SparkContext（conf）; 
 HiveContext hiveContext = new HiveContext（sc）; 
 hiveContext.setConf（hive.metastore.uris，thrift：// METASTORE：9083）; 
  

更新如果您的Hive已被Kerberized ：

尝试在创建HiveContext之前设置它们：

  System.setProperty（hive.metastore.sasl .enabled，true）; 
 System.setProperty（hive.security.authorization.enabled，false）; 
 System.setProperty（hive.metastore.kerberos.principal，hivePrincipal）; 
 System.setProperty（hive.metastore.execute.setugi，true）; 
  

I'm using HiveContext with SparkSQL and I'm trying to connect to a remote Hive metastore, the only way to set the hive metastore is through including the hive-site.xml on the classpath (or copying it to /etc/spark/conf/).

Is there a way to set this parameter programmatically in a java code without including the hive-site.xml ? If so what is the Spark configuration to use ?

解决方案

For Spark 1.x, you can set with :

System.setProperty("hive.metastore.uris", "thrift://METASTORE:9083");

final SparkConf conf = new SparkConf();
SparkContext sc = new SparkContext(conf);
HiveContext hiveContext = new HiveContext(sc);

Or

final SparkConf conf = new SparkConf();
SparkContext sc = new SparkContext(conf);
HiveContext hiveContext = new HiveContext(sc);
hiveContext.setConf("hive.metastore.uris", "thrift://METASTORE:9083");

Update If your Hive is Kerberized :

Try setting these before creating the HiveContext :

System.setProperty("hive.metastore.sasl.enabled", "true");
System.setProperty("hive.security.authorization.enabled", "false");
System.setProperty("hive.metastore.kerberos.principal", hivePrincipal);
System.setProperty("hive.metastore.execute.setugi", "true");

这篇关于如何在SparkSQL中以编程方式连接到Hive Metastore？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！