如何使用Spark/Scala +代码+配置通过远程Hive存储(S3 ORC)在Sparksql中加载文件 [英] How to load files in sparksql through remote hive storage ( s3 orc) using spark/scala + code + configuration

问题描述

intellij(spark)---> Hive(Remote)---存储在S3上(orc格式) 无法通过spark/scala读取远程Hive表.

intellij(spark)--->Hive(Remote)---storage on S3(orc format) Not able to read remote Hive table through spark/scala.

能够读取表模式，但无法读取表.

was able to read table schema but not able to read table.

错误-线程主"中的异常java.lang.IllegalArgumentException: 必须将AWS Access Key ID和Secret Access Key指定为 s3 URL的用户名或密码(分别)，或通过设置 fs.s3.awsAccessKeyId或fs.s3.awsSecretAccessKey属性 (分别).

Error -Exception in thread "main" java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.sql.{Encoders, SparkSession}
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.orc._
import org.apache.spark.sql.types.StructType

object mainclas {

  def main(args: Array[String]): Unit = {

     val spark = SparkSession.builder
      .master("local[*]")
      .appName("hivetable")
      .config("hive.metastore.uris", "thrift://10.20.30.40:9083")
       .config("access-key","PQHFFDEGGDDVDVV")
       .config("secret-key","FFGSGHhjhhhdjhJHJHHJGJHGjHH")
       .config("format", "orc")
      .enableHiveSupport()
      .getOrCreate()

   val res = spark.sqlContext.sql("show tables").show()
   val res1 =spark.sql("select *from ace.visit limit 5").show() 
}
}`

推荐答案

尝试一下:

val spark = SparkSession.builder
  .master("local[*]")
  .appName("hivetable")
  .config("hive.metastore.uris", "thrift://10.20.30.40:9083")
  .config("fs.s3n.awsAccessKeyId","PQHFFDEGGDDVDVV")
  .config("fs.s3n.awsSecretAccessKey","FFGSGHhjhhhdjhJHJHHJGJHGjHH")
  .config("format", "orc")
  .enableHiveSupport()
  .getOrCreate()

这篇关于如何使用Spark/Scala +代码+配置通过远程Hive存储(S3 ORC)在Sparksql中加载文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！