在 Spark 中写入和读取原始字节数组 - 使用 Sequence File SequenceFile [英] Write and read raw byte arrays in Spark - using Sequence File SequenceFile

问题描述

如何使用 Apache Spark 将 RDD[Array[Byte]] 写入文件并再次读取?

How do you write RDD[Array[Byte]] to a file using Apache Spark and read it back again?

推荐答案

常见问题似乎是奇怪的无法将异常从 BytesWritable 转换为 NullWritable.另一个常见的问题是 BytesWritable getBytes 是一堆完全没有意义的废话，根本没有得到字节.getBytes 所做的是获取您的字节，而不是在最后添加大量零！你必须使用 copyBytes

Common problems seem to be getting a weird cannot cast exception from BytesWritable to NullWritable. Other common problem is BytesWritable getBytes is a totally pointless pile of nonsense which doesn't get bytes at all. What getBytes does is get your bytes than adds a ton of zeros on the end! You have to use copyBytes

val rdd: RDD[Array[Byte]] = ???

// To write
rdd.map(bytesArray => (NullWritable.get(), new BytesWritable(bytesArray)))
  .saveAsSequenceFile("/output/path", codecOpt)

// To read
val rdd: RDD[Array[Byte]] = sc.sequenceFile[NullWritable, BytesWritable]("/input/path")
  .map(_._2.copyBytes())

这篇关于在 Spark 中写入和读取原始字节数组 - 使用 Sequence File SequenceFile的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！