Spark SQL - 自定义数据类型 UUID [英] Spark SQL - Custom Datatype UUID

问题描述

我正在尝试使用 Spark SQL 中的自定义数据类型将数据集中的列从 varchar 转换为 UUID.但我看到转换没有发生.如果我在这里遗漏了什么，请告诉我.

i am trying to convert the Column in the Dataset from varchar to UUID using the custom datatype in Spark SQL. But i see the conversion not happening. Please let me know if i am missing anything here.

val secdf = sc.parallelize( Array(("85d8b889-c793-4f23-93e9-ea18db640039","Revenue"), ("85d8b889-c793-4f23-93e9-ea18db640038","Income:123213"))).toDF("id", "report")
val metadataBuilder = new MetadataBuilder()
metadataBuilder.putString("database.column.type", "uuid")
metadataBuilder.putLong("jdbc.type", java.sql.Types.OTHER)
val metadata = metadataBuilder.build()
val secReportDF = secdf.withColumn("id", col("id").as("id", metadata))

推荐答案

由于我们无法在 Spark SQL 中强制转换为 UUID，我做了解决方案，我在 Postgres JDBC 客户端中添加了该属性作为 stringtype=unspecified 其中解决了我通过 Spark JDBC 插入 UUID 的问题

i did the work around as we are not able to cast to UUID in Spark SQL, i have added the property in the Postgres JDBC client as stringtype=unspecified which solved my issue in Inserting UUID through Spark JDBC

这篇关于Spark SQL - 自定义数据类型 UUID的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！