如何融化 Spark DataFrame? [英] How to melt Spark DataFrame?

问题描述

在 PySpark 或至少在 Scala 中的 Apache Spark 中是否有 Pandas Melt 函数的等价物?

Is there an equivalent of Pandas Melt Function in Apache Spark in PySpark or at least in Scala?

我一直在 python 中运行一个示例数据集，现在我想对整个数据集使用 Spark.

I was running a sample dataset till now in python and now I want to use Spark for the entire dataset.

提前致谢.

推荐答案

没有内置函数(如果您使用 SQL 和 Hive 支持，您可以使用 stack 函数，但在 Spark 中没有暴露，也没有原生实施)，但推出自己的产品是微不足道的.所需的导入:

There is no built-in function (if you work with SQL and Hive support enabled you can use stack function, but it is not exposed in Spark and has no native implementation) but it is trivial to roll your own. Required imports:

from pyspark.sql.functions import array, col, explode, lit, struct
from pyspark.sql import DataFrame
from typing import Iterable 

示例实现:

def melt(
        df: DataFrame, 
        id_vars: Iterable[str], value_vars: Iterable[str], 
        var_name: str="variable", value_name: str="value") -> DataFrame:
    """Convert :class:`DataFrame` from wide to long format."""

    # Create array<struct<variable: str, value: ...>>
    _vars_and_vals = array(*(
        struct(lit(c).alias(var_name), col(c).alias(value_name)) 
        for c in value_vars))

    # Add to the DataFrame and explode
    _tmp = df.withColumn("_vars_and_vals", explode(_vars_and_vals))

    cols = id_vars + [
            col("_vars_and_vals")[x].alias(x) for x in [var_name, value_name]]
    return _tmp.select(*cols)

以及一些测试(基于 Pandas doctests):

And some tests (based on Pandas doctests):

import pandas as pd

pdf = pd.DataFrame({'A': {0: 'a', 1: 'b', 2: 'c'},
                   'B': {0: 1, 1: 3, 2: 5},
                   'C': {0: 2, 1: 4, 2: 6}})

pd.melt(pdf, id_vars=['A'], value_vars=['B', 'C'])

   A variable  value
0  a        B      1
1  b        B      3
2  c        B      5
3  a        C      2
4  b        C      4
5  c        C      6

sdf = spark.createDataFrame(pdf)
melt(sdf, id_vars=['A'], value_vars=['B', 'C']).show()

+---+--------+-----+
|  A|variable|value|
+---+--------+-----+
|  a|       B|    1|
|  a|       C|    2|
|  b|       B|    3|
|  b|       C|    4|
|  c|       B|    5|
|  c|       C|    6|
+---+--------+-----+

注意:要与旧版 Python 一起使用，请删除类型注释.

Note: For use with legacy Python versions remove type annotations.

相关:

• r sparkR - 相当于melt函数
• 收集火花

这篇关于如何融化 Spark DataFrame?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！