使用Spark读取雪花表时遇到classnotfound异常 [英] Facing classnotfound exception while reading a snowflake table using spark

问题描述

我正在尝试从spark-shell读取雪花表.为此，我执行了以下操作.

I am trying to read a snowflake table from spark-shell. To do that, I did the following.

pyspark --jars spark-snowflake_2.11-2.8.0-spark_2.4.jar,jackson-dataformat-xml-2.10.3.jar
Using Python version 2.7.5 (default, Feb 20 2018 09:19:12)
SparkSession available as 'spark'.
>>> from pyspark import SparkConf, SparkContext
>>> from pyspark.sql import SQLContext
>>> from pyspark.sql.types import *
>>> from pyspark import SparkConf, SparkContext
>>> sc = SparkContext("local", "Simple App")
>>> spark = SQLContext(sc)
>>> spark_conf = SparkConf().setMaster('local').setAppName('CHECK')
>>> sfOptions = {
... "sfURL" : "url",
... "sfAccount" : "acntname",
... "sfUser" : 'username',
... "sfPassword" : 'pwd',
... "sfRole" :    'role',
... "sfDatabase" : 'dbname',
... "sfSchema" :  'schema',
... "sfWarehouse" : 'warehousename'
... }
>>> SNOWFLAKE_SOURCE = 'net.snowflake.spark.snowflake'
>>> df = spark.read.format(SNOWFLAKE_SOURCE).options(**sfOptions).option("query","select column from schema.table limit 1").load()

一旦运行load语句，我将面临以下classnotfound异常:

Once I run the load statement, I am facing the below classnotfound exception:

Caused by: java.lang.ClassNotFoundException: net.snowflake.client.jdbc.internal.fasterxml.jackson.databind.ObjectMapper

在上述操作中，除了读取雪花表和按照

In the above operation, there is nothing but reading a snowflake table and as per the documentation, I passed the required jar file I started spark-shell.

启动pyspark时加载的Spark版本为version 2.3.2.3.1.5.37-1 我尝试使用雪花连接器的多个版本2.3/2.4/2.8/3.0，并且还传递了jar文件jackson-dataformat-xml-2.10.3.jar，但仍然看到相同的异常.

The spark version that loads when I start pyspark is version 2.3.2.3.1.5.37-1 I tried with multiple versions of snowflake connector which are 2.3/2.4/2.8/3.0 and I also passed the jar file jackson-dataformat-xml-2.10.3.jar but I still see the same exception.

任何人都可以让我知道我在这里做的错误是什么以及如何纠正它吗?

Could anyone let me know what is the mistake I am doing here and how can I correct it ?

推荐答案

您应该运行

pyspark --jars spark-snowflake_2.11-2.8.0-spark_2.4.jar,snowflake-jdbc-3.12.5.jar

与代码相关的问题:

spark = SparkSession \
.builder \
.config("spark.jars", "<pathto>/snowflake-jdbc-3.12.5.jar,<pathto>/spark-snowflake_2.11-2.7.1-spark_2.4.jar") \
.config("spark.repl.local.jars",
        "<pathto>/snowflake-jdbc-3.12.5.jar,<pathto>/spark-snowflake_2.11-2.7.1-spark_2.4.jar") \
.config("spark.sql.catalogImplementation", "in-memory") \
.getOrCreate()

这篇关于使用Spark读取雪花表时遇到classnotfound异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！