识别具有多种数据类型的列中的数据类型计数 [英] Identify count of datatypes in a column which has multiple datatypes

问题描述

当我读取一个excel文件时，它有一列像这样，

When I read an excel file, it has a column like this,

Col1
----
aaa
123
true
235
321
23.23
xxx

我需要确定此列中有多少数据类型.当数据很大时，处理时间也很长.pyspark 中的任何选项?

I need to identify how many datatypes we have in this column. When the data is big the processing time is also big. Any options in pyspark?

问候，灰

推荐答案

spark 没有内置 udfs 来返回值数据类型，所以实现 udf返回数据类型，您可以将此处定义的函数扩展为其他数据类型 long，使用 regexp 也是一个选项

spark doesn't have built in udfs to return the value data type, so implement udf to return data type, you can extend the function defined here for other data types long, using regexp also an option

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T

def get_data_type(val):
    data_type = None
    try:
        float(val)
        data_type = "float"
    except ValueError:
        if (data_type != None and val.isnumeric()):
            data_type = 'int'
        else:
            if (val.lower() in ("yes", "no", "true", "false")):
                data_type = 'boolean'
            else:
                data_type = "string"
    else:
        if float(val).is_integer():
            data_type = "int"
    return data_type

get_data_type_udf = F.udf(get_data_type, T.StringType())

df = spark.createDataFrame(['aaa','123','true','235','321','23.23'], T.StringType()).toDF("col1")
df = df.select(get_data_type_udf(F.col("col1")).alias("data_type")).groupBy("data_type").count()
df.show()

结果

+---------+-----+
|data_type|count|
+---------+-----+
|      int|    3|
|  boolean|    1|
|   string|    1|
|    float|    1|
+---------+-----+

这篇关于识别具有多种数据类型的列中的数据类型计数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！