Spark读取大文件作为输入流 [英] Spark to read a big file as inputstream

问题描述

我知道 spark 内置方法可以分区和读取大块文件，并使用文本文件作为 rdd 分发.但是，我正在自定义的加密文件系统中阅读此内容，而 Spark 本质上不支持该文件系统.我能想到的一种方法是读取输入流并加载多行并分发给执行程序.继续阅读，直到所有文件都加载完毕.所以没有执行器会因为内存不足错误而崩溃.可以在 spark 中做到这一点吗?

I know spark built in method can have partition and read huge chunk of file and distributed as rdd using textfile. However, i am reading this in a customized encrytped filessytem which spark does not support by nature. One way i can think of is to read an inputstream instead and loads multiple lines and distributed to executor. Keep reading until all file is loaded. So no executor will blow up due to out of memory error. Is that possible to do this in spark?

推荐答案

你可以针对不同的 n 尝试lines.take(n) 来找到你的集群的限制.
或

you can try lines.take(n) for different n to find the limit of your cluster.
or

spark.readStream.option("sep", ";").csv("filepath.csv")

这篇关于Spark读取大文件作为输入流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！