将 Windows 身份验证用于带有 pyspark 的 SQL Server JDBC [英] Using windows authentication for SQL Server JDBC with pyspark

问题描述

我想问一下如何使用 Windows 身份验证和 pyspark 库连接 SQL Server?我可以与 Microsoft SQL Server Management Studio 连接，但当我尝试使用 spark 用 Python 编写代码时则无法连接.这是我目前尝试的方法.

I want to ask how can I connect the SQL Server using Windows Authentication, with pyspark library? I can connect with Microsoft SQL Server Management Studio but not when I try to code in Python with a spark.Here's what I tried so far.

from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.driver.extraClassPath","mssql-jdbc-6.4.0.jre8.jar") \
    .getOrCreate()

mssql_df = spark.read.format("jdbc") \
    .option("url", "jdbc:sqlserver://localhost:1433;databaseName=DATABASE-NAME") \
    .option("dbtable", "database-table-name") \
    .option("user", "Windows-Username") \
    .option("password", "Windows-Pass")\
    .option("driver", 'com.mysql.jdbc.Driver').load()

mssql_df.printSchema()
mssql_df.show()

推荐答案

如图 这里你可以设置integratedSecurity=true通过jdbc和Windows身份验证连接到SQL Server.

As shown here you can set the integratedSecurity=true to connect to SQL Server via jdbc and Windows Authentication.

然后 Spark 配置应该如下所示:

Then Spark configuration it should look as next:

mssql_df = spark.read.format("jdbc") \
    .option("url", "jdbc:sqlserver://localhost:1433;databaseName=DATABASE-NAME;integratedSecurity=true") \
    .option("dbtable", "database-table-name") \
    .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
    .load()

更新:

正如评论中所讨论的，用户应该将 sqljdbc_auth.dll 放在 mssql-jdbc-7.4.1.jre12.jar 所在的同一文件夹中，或者只设置 spark.driver.extraClassPath 用于由 : 分隔的两个 jar，如下所示:

As discussed in the comments user should place sqljdbc_auth.dll in the same folder where mssql-jdbc-7.4.1.jre12.jar lives or just set spark.driver.extraClassPath for both jars seperated by : as shown below:

.config("spark.driver.extraClassPath","/path.to/mssql-jdbc-6.4.0.jre8.jar:/path/to/sqljdbc_auth.dll")

sqljdbc_auth.dll 是 Microsoft JDBC Driver 6.0 for SQL Server 的一部分，您可以从 此处.或者，您可以在系统上安装 JDBC 驱动程序并指定 dll 的存储路径.

sqljdbc_auth.dll is part of the Microsoft JDBC Driver 6.0 for SQL Server and you can download it from here. Alternatively you can just install JDBC driver on your system and specify the path where the dll is stored.

这篇关于将 Windows 身份验证用于带有 pyspark 的 SQL Server JDBC的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！