将 Pandas 数据帧转换为 PySpark 数据帧 [英] Convert a pandas dataframe to a PySpark dataframe

问题描述

我有一个具有以下设置的脚本.

I have a script with the below setup.

我正在使用:

1) Spark dataframes 来拉取数据2)初始聚合后转换为pandas数据帧3) 想转换回 Spark 写入 HDFS

1) Spark dataframes to pull data in 2) Converting to pandas dataframes after initial aggregatioin 3) Want to convert back to Spark for writing to HDFS

从 Spark 转换 --> Pandas 很简单，但我在如何将 Pandas 数据帧转换回 Spark 方面苦苦挣扎.

The conversion from Spark --> Pandas was simple, but I am struggling with how to convert a Pandas dataframe back to spark.

你能提供建议吗?

from pyspark.sql import SparkSession
import pyspark.sql.functions as sqlfunc
from pyspark.sql.types import *
import argparse, sys
from pyspark.sql import *
import pyspark.sql.functions as sqlfunc
import pandas as pd

def create_session(appname):
    spark_session = SparkSession\
        .builder\
        .appName(appname)\
        .master('yarn')\
        .config("hive.metastore.uris", "thrift://uds-far-mn1.dab.02.net:9083")\
        .enableHiveSupport()\
        .getOrCreate()
    return spark_session
### START MAIN ###
if __name__ == '__main__':
    spark_session = create_session('testing_files')

我已经尝试了以下 - 没有错误，只是没有数据！确认一下，df6 确实有数据 &是一个熊猫数据框

I've tried the below - no errors, just no data! To confirm, df6 does have data & is a pandas dataframe

df6 = df5.sort_values(['sdsf'], ascending=["true"])
sdf = spark_session.createDataFrame(df6)
sdf.show()

推荐答案

我们开始:

# Spark to Pandas
df_pd = df.toPandas()

# Pandas to Spark
df_sp = spark_session.createDataFrame(df_pd)

这篇关于将 Pandas 数据帧转换为 PySpark 数据帧的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！