获取Apache Spark数据集中包含的列的列数据类型 [英] Get column data type of a column contained in a Apache spark data set

问题描述

我试图找到是否有一种方法可以使用Java获取Apache Spark数据集中包含的列的数据类型?我有一个数据集，其中包含一个名为 SSN 的列，并且我编写了此代码以修整该列中的数据:

I am trying to find if there is a way to get the datatype of a column contained in a Apache spark data set using java? I have a data set which contains a column called SSN and I wrote this code to trim the data in that column :

Dataset<Row> trimmedOutput = trimInput.select(trim(trimInput.col("SSN")).as("SSN")

我正在尝试获取 SSN 列的数据类型，以针对预期的类型对其进行验证.

I am trying to get the data type of the SSN column to validate it against the expected type.

有人可以帮我吗?

推荐答案

我来这里寻找相同的答案:)现在看一下API，这是我能想到的一种方法:

I came here looking for the same answer :) Now looking at the API, this is one way I can figure:

public static String dataTypeString(Dataset<Row> dataset, String colName) {
        StructField[] fields = dataset.schema().fields();
        String dataType = null;
        for(StructField field: fields) {
            if(field.name().equals(colName)) {
                dataType =  field.dataType().typeName();
                break;
            }
        }
        return dataType;
    }

要了解trimmedOutput数据集中的SSN列的数据类型，请按以下方式使用它:

To know the datatype of the SSN column in the trimmedOutput dataset, use it like below:

dataTypeString(trimmedOutput, "SSN") 

还有一个类似的方法simpleString()可以代替typeName()调用，API文档提到了两者之间的区别.

There is also a similar method simpleString() that you can invoke instead of typeName(), API docs mention the difference between these two.

如果您打算检查数据集中的某列是否属于某种数据类型，如果不是，则失败，以下代码将有所帮助:

If your intention is to check if a column in a dataset is of a certain datatype and fail if that's not the case, the below code will help:

SchemaUtils.checkColumnType(holdoutResults.schema(), 
                            "SSN", 
                            DataTypes.StrringType, 
                           "Datatype Mismatch for column SSN");

上面的调用将检查'SSN'列是否为String类型，如果不是，则失败，它将显示您作为最后一个参数传递的消息"SSN列的数据类型不匹配"，从而失败.此方法仅在ml库中的SchemUtils类上可用.

The above invocation will check if the 'SSN' column if of type String and if not so, it will fail by showing the message that you passed as the last argument - "Datatype Mismatch for column SSN". This method is available only on the SchemUtils class from the ml library.

这篇关于获取Apache Spark数据集中包含的列的列数据类型的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！