将LZO文件作为数据集导入到Java Spark中 [英] Importing a lzo file into java spark as dataset
问题描述
我有一些使用lzo压缩的tsv格式的数据.现在,我想在Java Spark程序中使用这些数据.
I have some data in tsv format compressed using lzo. Now, I would like to use these data in a java spark program.
此刻,我能够解压缩文件,然后使用
At the moment, I am able to decompress the files and then import them in Java as text files using
SparkSession spark = SparkSession.builder()
.master("local[2]")
.appName("MyName")
.getOrCreate();
Dataset<Row> input = spark.read()
.option("sep", "\t")
.csv(args[0]);
input.show(5); // visually check if data were imported correctly
我在第一个参数中将路径传递到了解压缩的文件.如果我将lzo文件作为参数传递,则show的结果将是难以辨认的垃圾.
where I have passed the path to the decompressed file in the first argument. If I pass the lzo file as an argument, the result of show is illegible garbage.
有没有办法使它起作用?我将IntelliJ用作IDE,并且该项目是在Maven中设置的.
Is there a way to make it work? I use IntelliJ as an IDE and the project is set-up in Maven.
推荐答案
我找到了解决方案.它由两部分组成:安装hadoop-lzo软件包并进行配置.完成此操作后,如果将lzo文件导入到单个分区中就可以了,代码将与问题中的代码相同.
I found a solution. It consists of two parts: installing the hadoop-lzo package and configuring it; after doing this, the code will remain the same as in the question, provided one is OK with the lzo file being imported in a single partition.
在下面,我将解释如何在IntelliJ中设置的Maven项目中做到这一点.
In the following I will explain how to do it for a maven project set up in IntelliJ.
-
安装软件包hadoop-lzo:您需要修改maven项目文件夹中的
pom.xml
文件.它应包含以下摘录:
Installing the package hadoop-lzo: you need to modify the
pom.xml
file that is in your maven project folder. It should contain the following excerpt:
<repositories>
<repository>
<id>twitter-twttr</id>
<url>http://maven.twttr.com</url>
</repository>
</repositories>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<!-- Apache Spark main library -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<!-- Packages for datasets and dataframes -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.1.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.hadoop.gplcompression/hadoop-lzo -->
<dependency>
<groupId>com.hadoop.gplcompression</groupId>
<artifactId>hadoop-lzo</artifactId>
<version>0.4.20</version>
</dependency>
</dependencies>
这将激活包含软件包 hadoop-lzo 并使该项目可用hadoop-lzo.
This will activate the maven Twitter repository that contains the package hadoop-lzo and make hadoop-lzo available for the project.
-
第二步是创建一个
core-site.xml
文件,告诉hadoop您已经安装了新的编解码器.它应该放在程序文件夹中的某个位置.我将其放在src/main/resources/core-site.xml
下,并将该文件夹标记为资源(在IntelliJ Project面板上,右键单击该文件夹->将目录标记为->资源根).core-site.xml
文件应包含:
The second step is to create a
core-site.xml
file to tell hadoop that you have installed a new codec. It should be placed somewhere in the program folders. I put it undersrc/main/resources/core-site.xml
and marked the folder as a resource (right click on the folder from the IntelliJ Project panel -> Mark Directory as -> Resources root). Thecore-site.xml
file should contain:
<configuration>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.DefaultCodec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec,
org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.BZip2Codec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
</configuration>
就是这样!再次运行您的程序,它应该可以工作!
And that's it! Run your program again and it should work!
这篇关于将LZO文件作为数据集导入到Java Spark中的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!