在 org.apache.spark.sql.types.DataTypes 中找不到 uuid [英] Cant find uuid in org.apache.spark.sql.types.DataTypes

问题描述

我们有一个 PostgreSQL 表，其列之一是 UUID.我们如何将 Spark 数据集中的 UUID 字段(使用 Java)发送到 PostgreSQL DB.我们无法在 org.apache.spark.sql.types.DataTypes 中找到 uuid 字段.

We have a PostgreSQL table which has UUID as one of the column. How do we send UUID field in Spark dataset(using Java) to PostgreSQL DB. We are not able to find uuid field in org.apache.spark.sql.types.DataTypes.

请指教.

推荐答案

正如已经指出的，尽管这些问题已经解决 (10186, 5753) 那里从 Spark 2.3.0 开始，仍然不支持 uuid Postgres 数据类型.

As already pointed out, despite these resolved issues (10186, 5753) there is still no supported uuid Postgres data type as of Spark 2.3.0.

但是，有一种解决方法，方法是使用 Spark 的 SaveMode.Append 并设置href="https://jdbc.postgresql.org/documentation/94/connect.html" rel="nofollow noreferrer">允许推断字符串类型的 Postgres JDBC 属性.简而言之，它的工作原理如下:

However, there's a workaround by using Spark's SaveMode.Append and setting the Postgres JDBC property to allow string types to be inferred. In short, it works like:

    val props = Map(
          JDBCOptions.JDBC_DRIVER_CLASS -> "org.postgresql.Driver",
          "url" -> url,
          "user" -> user,
          "stringtype" -> "unspecified"
        )
          
    yourData.write.mode(SaveMode.Append)
        .format("jdbc")
        .options(props)
        .option("dbtable", tableName)
        .save()

该表应使用已定义为 uuid 类型的 uuid 列创建.但是，如果您尝试让 Spark 2.3.0 创建此表，您将再次碰壁:

The table should be created with the uuid column already defined with type uuid. If you try to have Spark 2.3.0 create this table though, you will again hit a wall:

    yourData.write.mode(SaveMode.Overwrite)
        .format("jdbc")
        .options(props)
        .option("dbtable", tableName)
        .option("createTableColumnTypes", "some_uuid_column_name uuid")
        .save()

结果:

不支持数据类型 uuid.(第 1 行，位置 21)

DataType uuid is not supported.(line 1, pos 21)

这篇关于在 org.apache.spark.sql.types.DataTypes 中找不到 uuid的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！