如何在Spark DataFrame中将现有字段设置为主键? [英] How to set an existing field as a primary key in spark dataframe?

问题描述

当我使用JDBC连接器将spark数据帧的数据写入SQL DB时.它覆盖了表的属性.

When I am writing the data of a spark dataframe into SQL DB by using JDBC connector. It is overwritting the properties of the table.

因此，我想在写入数据之前在spark数据框中设置关键字段.

So, i want to set the keyfield in spark dataframe before writing the data.

url = "jdbc:sqlserver://{0}:{1};database={2};user={3};password={4};encrypt=true;trustServerCertificate=false; hostNameInCertificate=*.database.windows.net;loginTimeout=30;".format(jdbcHostname, jdbcPort, jdbcDatabase, JDBCusername, JDBCpassword)

newSchema_Product_names = [StructField('product__code',StringType(), False),                              
                               StructField('product__names__lang_code',StringType(),False),
                               StructField('product__names__name',StringType(),True),
                               StructField('product__country__code',StringType(),True),
                               StructField('product__country__name',StringType(),True)
                              ]

Product_names1 = sqlContext.createDataFrame(Product_names_new,StructType(newSchema_Product_names))
Product_names1.write.mode("overwrite").jdbc(url, "product_names")

之前:

之后:

推荐答案

@cronoik是的，它是正确的.我以为JDBC连接器不支持truncate选项.但是更改字段的大小后，它对我有用.

@cronoik yes its correct. I thought JDBC connector is not supporting truncate option. But after changing the size of the fields, it worked for me.

这篇关于如何在Spark DataFrame中将现有字段设置为主键?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！