Apache Spark 结构化流 (DataStreamWriter) 写入 Hive 表 [英] Apache Spark Structured Streaming (DataStreamWriter) write to Hive table

问题描述

我希望使用 Spark Structured 流从 Kafka 读取数据并对其进行处理并写入 Hive 表.

I am looking to use Spark Structured streaming to read data from Kafka and process it and write to Hive table.

 val spark = SparkSession
   .builder
   .appName("Kafka Test")
   .config("spark.sql.streaming.metricsEnabled", true)
   .config("spark.streaming.backpressure.enabled", "true")
   .enableHiveSupport()
   .getOrCreate()

val events = spark
  .readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "xxxxxxx")
  .option("startingOffsets", "latest")
  .option("subscribe", "yyyyyy")
  .load


val data = events.select(.....some columns...)

data.writeStream
  .format("parquet")
  .option("compression", "snappy")
  .outputMode("append")
  .partitionBy("ds")
  .option("path", "maprfs:/xxxxxxx")
  .start()
  .awaitTermination()

这确实创建了一个镶木地板文件，但是我如何更改它以模仿类似的东西，以便它写入可以使用 (select * from) 从 hive 或 spark-sql 读取的表格式

This does create a parquet files, however how do I change this to mimic something like, so that it writes into table format which can be read from hive or spark-sql using (select * from)

data.write.format("parquet").option("compression", "snappy").mode("append").partitionBy("ds").saveAsTable("xxxxxx")

推荐答案

我建议查看 Kafka Connect 用于将数据写入 HDFS.它是开源的，可以独立或作为Confluent 平台.

I would recommend looking at Kafka Connect for writing the data to HDFS. It is open source and available standalone or as part of Confluent Platform.

要过滤和转换数据，您可以使用 Kafka Streams 或 KSQL.KSQL 运行在 Kafka Streams 之上，为您提供了一种非常简单的方法来连接数据、过滤数据和构建聚合.

For filtering and transforming the data you could use Kafka Streams, or KSQL. KSQL runs on top of Kafka Streams, and gives you a very simple way to join data, filter it, and build aggregations.

这是在 KSQL 中进行数据流聚合的示例

Here's an example of doing aggregations of streams of data in KSQL

SELECT PAGE_ID,COUNT(*) FROM PAGE_CLICKS WINDOW TUMBLING (SIZE 1 HOUR) GROUP BY PAGE_ID

在 本博客.您可能还对 感兴趣这个演讲是关于用这些组件构建流数据管道

See KSQL in action in this blog. You might also be interested in this talk about building streaming data pipelines with these components

这篇关于Apache Spark 结构化流 (DataStreamWriter) 写入 Hive 表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！