将 spark DataFrame 转换为 pandas DF [英] Convert a spark DataFrame to pandas DF

问题描述

有没有办法将 Spark Df(不是 RDD)转换为 Pandas DF

Is there a way to convert a Spark Df (not RDD) to pandas DF

我尝试了以下方法:

var some_df = Seq(
 ("A", "no"),
 ("B", "yes"),
 ("B", "yes"),
 ("B", "no")

 ).toDF(
"user_id", "phone_number")

代码:

%pyspark
pandas_df = some_df.toPandas()

错误:

 NameError: name 'some_df' is not defined

任何建议.

推荐答案

以下应该有效

some_df = sc.parallelize([
 ("A", "no"),
 ("B", "yes"),
 ("B", "yes"),
 ("B", "no")]
 ).toDF(["user_id", "phone_number"])
pandas_df = some_df.toPandas()

这篇关于将 spark DataFrame 转换为 pandas DF的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！