如何通过 Spark 打开/流式传输 .zip 文件? [英] How to open/stream .zip files through Spark?

问题描述

我有要通过 Spark 打开的 zip 文件.由于 Hadoop 原生编解码器支持，我可以打开 .gzip 文件没问题，但无法使用 .zip 文件打开.

I have zip files that I would like to open 'through' Spark. I can open .gzip file no problem because of Hadoops native Codec support, but am unable to do so with .zip files.

是否有一种简单的方法可以读取 Spark 代码中的 zip 文件?我还搜索了要添加到 CompressionCodecFactory 的 zip 编解码器实现，但到目前为止都没有成功.

Is there an easy way to read a zip file in your Spark code? I've also searched for zip codec implementations to add to the CompressionCodecFactory, but am unsuccessful so far.

推荐答案

python 代码没有解决方案，我最近不得不阅读 pyspark 中的 zip.而且，在搜索如何做到这一点时，我遇到了这个问题.所以，希望这会帮助其他人.

There was no solution with python code and I recently had to read zips in pyspark. And, while searching how to do that I came across this question. So, hopefully this'll help others.

import zipfile
import io

def zip_extract(x):
    in_memory_data = io.BytesIO(x[1])
    file_obj = zipfile.ZipFile(in_memory_data, "r")
    files = [i for i in file_obj.namelist()]
    return dict(zip(files, [file_obj.open(file).read() for file in files]))


zips = sc.binaryFiles("hdfs:/Testing/*.zip")
files_data = zips.map(zip_extract).collect()

在上面的代码中，我返回了一个字典，以 zip 中的文件名作为键，以每个文件中的文本数据作为值.您可以根据自己的目的进行更改.

In the above code I returned a dictionary with filename in the zip as a key and the text data in each file as the value. you can change it however you want to suit your purposes.

这篇关于如何通过 Spark 打开/流式传输 .zip 文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！