Spark不会读取第一行中具有空值的列 [英] Spark doesn't read columns with null values in first row

问题描述

以下是我的csv文件中的内容:

Below is the content in my csv file :

A1,B1,C1
A2,B2,C2,D1
A3,B3,C3,D2,E1
A4,B4,C4,D3
A5,B5,C5,,E2

因此，有5列，但第一行中只有3个值.

So, there are 5 columns but only 3 values in the first row.

我使用以下命令阅读它:

I read it using the following command :

val csvDF : DataFrame = spark.read
.option("header", "false")
.option("delimiter", ",")
.option("inferSchema", "false")
.csv("file.csv") 

以下是我使用csvDF.show()

And following is what i get using csvDF.show()

+---+---+---+
|_c0|_c1|_c2|
+---+---+---+
| A1| B1| C1|
| A2| B2| C2|
| A3| B3| C3|
| A4| B4| C4|
| A5| B5| C5|
+---+---+---+

如何读取所有列中的所有数据?

How can i read all the data in all the columns?

推荐答案

基本上，您的csv文件格式不正确，因为每行中的列数不相等，如果您这样做，想要使用 spark.read.csv 进行阅读.但是，您可以改为使用 spark.read.textFile 读取它，然后解析每一行.

Basically your csv-file isn't properly formatted in the sense that it doesn't have a equal number of columns in each row, which is required if you want to read it with spark.read.csv. However, you can instead read it with spark.read.textFile and then parse each row.

据我了解，您事先不知道列数，因此您希望代码处理任意数量的列.为此，您需要确定数据集中的最大列数，因此需要对数据集进行两次传递.

As I understand it, you do not know the number of columns beforehand, so you want your code to handle an arbitrary number of columns. To do this you need to establish the maximum number of columns in your data set, so you need two passes over your data set.

对于这个特殊的问题，我实际上会使用RDD而不是DataFrames或Dataset，就像这样:

For this particular problem, I would actually go with RDDs instead of DataFrames or Datasets, like this:

val data  = spark.read.textFile("file.csv").rdd

val rdd = data.map(s => (s, s.split(",").length)).cache
val maxColumns = rdd.map(_._2).max()

val x = rdd
  .map(row => {
    val rowData = row._1.split(",")
    val extraColumns = Array.ofDim[String](maxColumns - rowData.length)
    Row((rowData ++ extraColumns).toList:_*)
  })

希望有帮助:)

这篇关于Spark不会读取第一行中具有空值的列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！