让spark节俭服务器成为仓库目录的路径 [英] Giving spark thrift server the path to warehouse directory

问题描述

我已经设置了我的Spark集群，并且通过Spark SQL连接器成功连接了Tableau。

我使用（ saveAsTable ）从Spark shell创建了我的表，并从MySQL保存了数据框。

如何访问我从Tableau保存的表格？在启动Spark Thrift服务器时是否需要提供仓库目录的路径？如果是的话，如何做，如果没有，怎么做？

解决方案

确保您指向 spark- shell 和
thriftserver

Metastore共享可以通过两种方式实现，

1. 从同一位置启动shell和thrift
   
2. 为元数据设置远程数据库
   

您可以通过 - hiveconf 将配置单元配置文件传递给Spark Thrift服务器。和Spark confs - conf

  ./ sbin / start- thriftserver.sh \ 
 --conf spark.sql.warehouse.dir = path / to / warehouse / dir \ 
 --hiveconf hive.server2.thrift.port =< listening-port> \ 
 --hiveconf hive.server2.thrift.bind.host =< listening-host> \ 
 --master< master-uri> 
 ... 
  




I have setup my spark cluster and I am successful in connecting Tableau through Spark SQL connector.

I created my tables from spark shell and saved dataframes from MySQL using (saveAsTable).

How can I access the tables that I have saved from Tableau? Do I need to give a path of the warehouse directory when starting the spark thrift server? If yes, how it can be done and if no, how can this be done?

解决方案

Make sure you are pointing to the same metastore for spark-shell and thriftserver

Metastore sharing can be 2 ways, in simple

1. Starting both shell and thrift from same location
2. Setting up remote database for metastore

You can pass hive confs to Spark thrift server with --hiveconf and Spark confs with --conf

./sbin/start-thriftserver.sh \
  --conf spark.sql.warehouse.dir=path/to/warehouse/dir \
  --hiveconf hive.server2.thrift.port=<listening-port> \
  --hiveconf hive.server2.thrift.bind.host=<listening-host> \
  --master <master-uri>
  ...

这篇关于让spark节俭服务器成为仓库目录的路径的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！