数据帧与 Apache Spark 中的 pyspark 转置 [英] Dataframe transpose with pyspark in Apache Spark

问题描述

我有一个具有以下结构的数据帧 df:

I have a dataframe df that have following structure:

+-----+-----+-----+-------+
|  s  |col_1|col_2|col_...|
+-----+-----+-----+-------+
| f1  |  0.0|  0.6|  ...  |
| f2  |  0.6|  0.7|  ...  |
| f3  |  0.5|  0.9|  ...  |
|  ...|  ...|  ...|  ...  |

我想计算这个数据帧的转置，所以它看起来像

And I want to calculate the transpose of this dataframe so it will be look like

+-------+-----+-----+-------+------+
|  s    | f1  | f2  | f3    |   ...|
+-------+-----+-----+-------+------+
|col_1  |  0.0|  0.6|  0.5  |   ...|
|col_2  |  0.6|  0.7|  0.9  |   ...|
|col_...|  ...|  ...|  ...  |   ...|

我将这两个解决方案捆绑在一起，但它返回数据帧没有指定的使用方法:

I tied this two solutions but it returns that dataframe has not the specified used method:

方法一:

 for x in df.columns:
    df = df.pivot(x)

方法二:

df = sc.parallelize([ (k,) + tuple(v[0:]) for k,v in df.items()]).toDF()

我该如何解决这个问题.

how can I fix this.

推荐答案

如果数据小到可以转置(未通过聚合进行透视)，您只需将其转换为 Pandas DataFrame:

If data is small enough to be transposed (not pivoted with aggregation) you can just convert it to Pandas DataFrame:

df = sc.parallelize([
    ("f1", 0.0, 0.6, 0.5),
    ("f2", 0.6, 0.7, 0.9)]).toDF(["s", "col_1", "col_2", "col_3"])

df.toPandas().set_index("s").transpose()
s       f1   f2
col_1  0.0  0.6
col_2  0.6  0.7
col_3  0.5  0.9

如果它太大，Spark 将无济于事.Spark DataFrame 按行分布数据(虽然本地使用列式存储)，因此单个行的大小仅限于本地内存.

If it is to large for this, Spark won't help. Spark DataFrame distributes data by row (although locally uses columnar storage), therefore size of a individual rows is limited to local memory.

这篇关于数据帧与 Apache Spark 中的 pyspark 转置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！