读取文件提示，将具有特定值的字段设置为null或“". [英] Read File spark, set field having specific value to null or ""

问题描述

我正在读取一个以|分隔的文本文件. .有些字段的值为\ N.当逐行读取文件到数据帧时，有什么方法可以使具有\ N值的字段为null或". 代码如下.

I'm Reading a text file delimited with | . There are some fields having value \N . When read the file row by row to a data-frame, is there any way to make the field having value \N to null or "" . Code is given below.

val inputDf = sqlContext.read.format("csv")
      .option("header", "true")
      .option("inferSchema", "false")
      .schema(myschema)
      .option("delimiter", "|")
      .option("nullValue", "")
      .load("My Input file Path")

推荐答案

"DataFrameNaFunctions"可用于在所有列中将"\ N"替换为":

"DataFrameNaFunctions" can be used for replace value "\N" in all columns with "":

df.na.replace(df.columns.toSeq, Map("\\N" -> ""))

这篇关于读取文件提示，将具有特定值的字段设置为null或“".的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！