如何在 PySpark 中过滤 MapType 中的键? [英] How to filter keys in MapType in PySpark?

问题描述

给定一个如下的数据帧，是否可以在保持架构完整的同时过滤掉 PySpark 中列 集合 (MapType(StringType, StringType, True)) 的一些键?

Given a DataFrame as below is it possible to filter out some keys of the Column collection (MapType(StringType, StringType, True)) in PySpark while keeping the schema intact?

root
 |-- id: string (nullable = true)
 |-- collection: map (nullable = true)
 |    |-- key: string
 |    |-- value: string

推荐答案

是的，这是可能的.您应该创建 udf 负责从地图中过滤键，并将其与 withColumn 转换一起使用以过滤 collection 字段中的键.

Yes it's possible. You should create udf responsible for filtering keys from map and use it with withColumn transformation to filter keys from collection field.

以下 Scala 实现示例:

Below example implementation in Scala:

// Start from implementing method in Scala responsible for filtering keys from Map
def filterKeys(collection: Map[String, String], keys: Iterable[String]): Map[String, String] =
    collection.filter{case (k,_) => !keys.exists(_ == k)}

// Create Spark UDF based on above function
val filterKeysUdf = udf((collection: Map[String, String], keys: Iterable[String]) => filterKeys(collection, keys))

// Use above udf to filter keys
val newDf = df.withColumn("collection", filterKeysUdf(df("collection"), lit(Array("k1"))))

Python 实现:

# Start from implementing method in Python responsible for filtering keys from dict
def filterKeys(collection, keys):
    return {k:collection[k] for k in collection if k not in keys}

# Create Spark UDF based on above function
filterKeysUdf = udf(filterKeys, MapType(StringType(), StringType()))

# Create array literal based on Python list
keywords_lit = array(*[lit(k) for k in ["k1","k2"]])

# Use above udf to filter keys
newDf = df.withColumn("collection", filterKeysUdf(df.collection, keywords_lit))

这篇关于如何在 PySpark 中过滤 MapType 中的键?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！