Spark-写入Avro文件 [英] Spark - write Avro file

问题描述

在这样的流程中使用Spark(使用Scala API)写入Avro文件的常见做法是:

What are the common practices to write Avro files with Spark (using Scala API) in a flow like this:

1. 从HDFS解析一些日志文件
2. 为每个日志文件应用一些业务逻辑并生成Avro文件(或可能合并多个文件)
3. 将Avro文件写入HDFS

我尝试使用spark-avro，但这并没有太大帮助.

I tried to use spark-avro, but it doesn't help much.

val someLogs = sc.textFile(inputPath)

val rowRDD = someLogs.map { line =>
  createRow(...)
}

val sqlContext = new SQLContext(sc)
val dataFrame = sqlContext.createDataFrame(rowRDD, schema)
dataFrame.write.avro(outputPath)

此操作失败，并显示错误:

This fails with error:

org.apache.spark.sql.AnalysisException: 
      Reference 'StringField' is ambiguous, could be: StringField#0, StringField#1, StringField#2, StringField#3, ...

推荐答案

Databricks提供了spark-avro库，该库可帮助我们读取和写入Avro数据.

Databricks provided library spark-avro, which helps us in reading and writing Avro data.

dataframe.write.format("com.databricks.spark.avro").save(outputPath)

这篇关于Spark-写入Avro文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！