将 apache spark 与 apache hive 远程连接. [英] connecting apache spark with apache hive remotely.

问题描述

我可以从安装了 apache spark 的集群中的 hive 服务器加载数据.但是如何将数据从远程配置单元服务器加载到数据帧中.hive jdbc 连接器是唯一的选择吗?

I can load data from hive server in the same cluster as where apache spark is installed. But how can i load data into dataframe from a remote hive server. Is the hive jdbc connector the only option to do so?

有什么建议我该怎么做?

any suggestion how can i do this?

推荐答案

您可以使用 org.apache.spark.sql.hive.HiveContext 对 Hive 表执行 SQL 查询.

You can use org.apache.spark.sql.hive.HiveContext to perform SQL query over Hive tables.

您也可以将 spark 连接到真正存储数据的底层 HDFS 目录.这将更加高效，因为 SQL 查询不需要解析或应用于文件的架构.

You can alternatively connect spark to the underlying HDFS directory where data is really stored. This will be more performant as the SQL query doesn't need parsed or the schema applied over the files.

如果集群是外部集群，你需要设置 hive.metastore.uris

If the cluster is an external one, you'll need to set hive.metastore.uris

这篇关于将 apache spark 与 apache hive 远程连接.的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！