在写入 Spark 数据帧时更改可空字段 [英] Nullable field is changed upon writing a Spark Dataframe

问题描述

以下代码从 parquet 文件中读取 Spark DataFrame 并写入另一个 parquet 文件.将 DataFrame 写入新 Parquet 文件后，ArrayType DataType 中的 Nullable 字段发生更改.

The following code reads a Spark DataFrame from parquet file and writes to another parquet file. Nullable filed in ArrayType DataType is changed after writing the DataFrame to a new Parquet file.

代码:

    SparkConf sparkConf = new SparkConf();
    String master = "local[2]";
    sparkConf.setMaster(master);
    sparkConf.setAppName("Local Spark Test");
    JavaSparkContext sparkContext = new JavaSparkContext(new SparkContext(sparkConf));
    SQLContext sqc = new SQLContext(sparkContext);
    DataFrame dataFrame = sqc.read().parquet("src/test/resources/users.parquet");
    StructField[] fields = dataFrame.schema().fields();
    System.out.println(fields[2].dataType());
    dataFrame.write().mode(SaveMode.Overwrite).parquet("src/test/resources/users1.parquet");


    DataFrame dataFrame1 = sqc.read().parquet("src/test/resources/users1.parquet");
    StructField [] fields1 = dataFrame1.schema().fields();
    System.out.println(fields1[2].dataType());

输出:

ArrayType(IntegerType,false)
ArrayType(IntegerType,true)

Spark 版本为:1.6.2

Spark version is: 1.6.2

推荐答案

对于 Spark 2.4 或之前版本，从 spark sql 写入的所有列都可以为 null.引用 官方指南

For Spark 2.4 or before, all the columns written from spark sql are nullable. Quoting the official guide

Parquet 是一种列格式，许多其他数据处理系统都支持该格式.Spark SQL 支持读取和写入 Parquet 文件，自动保留原始数据的模式.写入 Parquet 文件时，出于兼容性原因，所有列都会自动转换为可为空.

Parquet is a columnar format that is supported by many other data processing systems. Spark SQL provides support for both reading and writing Parquet files that automatically preserves the schema of the original data. When writing Parquet files, all columns are automatically converted to be nullable for compatibility reasons.

这篇关于在写入 Spark 数据帧时更改可空字段的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！