如何使用雪花和python从 pandas 数据框创建Spark数据框？ [英] How to create a Spark data frame from Pandas data frame using snow flake and python?

问题描述

我有一个sql，它存储在python中的变量中，我们使用SnowFlake数据库。首先，我已经使用sql转换为Pandas Data框架，但是我需要转换为Spark数据框架，然后存储在CreateorReplaceTempView中。我尝试过：

I have a sql which is stored in a variable in python and we use SnowFlake database. First I have converted to Pandas Data frame using sql, but I need to convert to Spark Data frame and then store in a CreateorReplaceTempView. I tried:

    import pandas as pd
    import sf_connectivity (we have a code for establishing connection with Snowflake database)
    emp = 'Select * From Employee'
    snowflake_connection = sf_connectivity.collector() (It is a method to establish snowflake conenction)
    pd_df = pd.read_sql_query(emp, snowflake_connection)
    requirement 1: Create SnowFlake Dataframe (sf_df) from Pandas Dataframe (pd_df)
    requirement 2: sf_df.createOrReplaceTempView(Temp_Employee)

我该如何做？

推荐答案

根据我对上述问题的评论，最好只使用Snowflake Spark连接器将数据直接加载到Spark中的Dataframe中。以下是该文档的链接，详细说明了如何执行此操作：

Per my comment in the question above, you'd likely be better off just loading the data directly to a Dataframe in Spark using the Snowflake Spark connector. Here is a link to the document that details how to do this:

https://docs.snowflake.com/zh-CN/user-guide/spark-connector-use.html ＃moving-data-fromsnowflake-tospark

这篇关于如何使用雪花和python从 pandas 数据框创建Spark数据框？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！