如何在PySpark的MapType中过滤键? [英] How to filter keys in MapType in PySpark?

问题描述

给出如下所示的DataFrame，是否可以在保持架构完好无损的情况下过滤出PySpark中的列集合(MapType(StringType，StringType，True))的某些键?

Given a DataFrame as below is it possible to filter out some keys of the Column collection (MapType(StringType, StringType, True)) in PySpark while keeping the schema intact?

root
 |-- id: string (nullable = true)
 |-- collection: map (nullable = true)
 |    |-- key: string
 |    |-- value: string

推荐答案

是的.您应该创建 udf 来负责从地图中过滤键，并将其与 withColumn 转换一起使用以从 collection 字段中过滤键.

Yes it's possible. You should create udf responsible for filtering keys from map and use it with withColumn transformation to filter keys from collection field.

下面在Scala中的示例实现:

Below example implementation in Scala:

// Start from implementing method in Scala responsible for filtering keys from Map
def filterKeys(collection: Map[String, String], keys: Iterable[String]): Map[String, String] =
    collection.filter{case (k,_) => !keys.exists(_ == k)}

// Create Spark UDF based on above function
val filterKeysUdf = udf((collection: Map[String, String], keys: Iterable[String]) => filterKeys(collection, keys))

// Use above udf to filter keys
val newDf = df.withColumn("collection", filterKeysUdf(df("collection"), lit(Array("k1"))))

Python的实现:

Implementation in Python:

# Start from implementing method in Python responsible for filtering keys from dict
def filterKeys(collection, keys):
    return {k:collection[k] for k in collection if k not in keys}

# Create Spark UDF based on above function
filterKeysUdf = udf(filterKeys, MapType(StringType(), StringType()))

# Create array literal based on Python list
keywords_lit = array(*[lit(k) for k in ["k1","k2"]])

# Use above udf to filter keys
newDf = df.withColumn("collection", filterKeysUdf(df.collection, keywords_lit))

这篇关于如何在PySpark的MapType中过滤键?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！