如何通过pyspark读取gz压缩文件 [英] How to read gz compressed file by pyspark
问题描述
我有.gz压缩格式的行数据.我必须在pyspark中阅读 以下是代码段
I have line data in .gz compressed format. I have to read it in pyspark Following is the code snippet
rdd = sc.textFile("data/label.gz").map(func)
但是我无法成功读取以上文件.我如何读取gz压缩文件.我在此处但我当前的spark版本与该问题中的版本不同.我希望应该在hadoop中有一些内置函数.
But I could not read the above file successfully. How do I read gz compressed file. I have found a similar question here but my current version of spark is different that the version in that question. I expect there should be some built in function as in hadoop.
推荐答案
Spark文档明确指定您可以自动读取gz
文件:
Spark document clearly specify that you can read gz
file automatically:
Spark的所有基于文件的输入法,包括textFile,均受支持 也可以在目录,压缩文件和通配符上运行.为了 例如,您可以使用textFile("/my/directory"), textFile("/my/directory/ .txt")和textFile("/my/directory/ .gz").
All of Spark’s file-based input methods, including textFile, support running on directories, compressed files, and wildcards as well. For example, you can use textFile("/my/directory"), textFile("/my/directory/.txt"), and textFile("/my/directory/.gz").
我建议运行以下命令,然后查看结果:
I'd suggest running the following command, and see the result:
rdd = sc.textFile("data/label.gz")
print rdd.take(10)
假设spark找到文件data/label.gz
,它将打印文件中的10行.
Assuming that spark finds the the file data/label.gz
, it will print the 10 rows from the file.
请注意,像data/label.gz
这样的文件的默认位置将在spark-user的hdfs文件夹中.在那吗?
Note, that the default location for a file like data/label.gz
will be in the hdfs folder of the spark-user. Is it there?
这篇关于如何通过pyspark读取gz压缩文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!