在Java中使用Spark读取Avro [英] Read Avro with Spark in java

问题描述

有人可以分享在Spark中使用Java读取Avro的示例吗? 找到了scala示例，但是java没有运气. 这是代码片段，它是代码的一部分，但是在使用ctx.newAPIHadoopFile方法时会遇到编译问题.

Can somebody share example of reading avro using java in spark? Found scala examples but no luck with java. Here is the code snippet which is part of code but running into compilation issues with the method ctx.newAPIHadoopFile.

JavaSparkContext ctx = new JavaSparkContext(sparkConf);
Configuration hadoopConf = new Configuration();
JavaRDD<SampleAvro> lines = ctx.newAPIHadoopFile(path, AvroInputFormat.class, AvroKey.class, NullWritable.class, new Configuration());

致谢

推荐答案

您可以使用Databricks的 spark-avro连接器库.
从Spark SQL读取或写入Avro数据的推荐方法是使用Spark的DataFrame API.

You can use the spark-avro connector library by Databricks.
The recommended way to read or write Avro data from Spark SQL is by using Spark's DataFrame APIs.

该连接器可从Spark SQL读取和写入Avro数据:

The connector enables both reading and writing Avro data from Spark SQL:

import org.apache.spark.sql.*;

SQLContext sqlContext = new SQLContext(sc);

// Creates a DataFrame from a specified file
DataFrame df = sqlContext.read().format("com.databricks.spark.avro")
    .load("src/test/resources/episodes.avro");

// Saves the subset of the Avro records read in
df.filter($"age > 5").write()
    .format("com.databricks.spark.avro")
    .save("/tmp/output");

请注意，此连接器针对Spark 1.2、1.3和1.4+具有不同版本:

Note that this connector has different versions for Spark 1.2, 1.3, and 1.4+:

Spark ver 连接器
1.2         0.2.0     
1.3 1.0.0     
1.4 +       2.0.1     

Spark verconnector
1.2          0.2.0      
1.3          1.0.0      
1.4+        2.0.1      

使用Maven:

<dependency>
    <groupId>com.databricks</groupId>
    <artifactId>spark-avro_2.10</artifactId>
    <version>{AVRO_CONNECTOR_VERSION}</version>
</dependency>

在以下位置查看更多信息: Spark SQL Avro库

See further info at: Spark SQL Avro Library

这篇关于在Java中使用Spark读取Avro的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！