如何在pyspark中自动删除常量列? [英] How to automatically drop constant columns in pyspark?

问题描述

我在 pyspark 中有一个 spark 数据框，我需要从数据框中删除所有常量列.由于我不知道哪些列是常量，我无法手动取消选择常量列，即我需要一个自动程序.我很惊讶我无法在 stackoverflow 上找到一个简单的解决方案.

I have a spark dataframe in pyspark and I need to drop all constant columns from my dataframe. Since I don't know which columns are constant I cannot manually unselect the constant columns, i.e. I need an automatic procedure. I am surprised I was not able to find a simple solution on stackoverflow.

示例:

import pandas as pd
import pyspark
from pyspark.sql.session import SparkSession
spark = SparkSession.builder.appName("test").getOrCreate()

d = {'col1': [1, 2, 3, 4, 5], 
     'col2': [1, 2, 3, 4, 5],
     'col3': [0, 0, 0, 0, 0],
     'col4': [0, 0, 0, 0, 0]}
df_panda = pd.DataFrame(data=d)
df_spark = spark.createDataFrame(df_panda)
df_spark.show()

输出:

+----+----+----+----+
|col1|col2|col3|col4|
+----+----+----+----+
|   1|   1|   0|   0|
|   2|   2|   0|   0|
|   3|   3|   0|   0|
|   4|   4|   0|   0|
|   5|   5|   0|   0|
+----+----+----+----+

所需的输出:

+----+----+
|col1|col2|
+----+----+
|   1|   1|
|   2|   2|
|   3|   3|
|   4|   4|
|   5|   5|
+----+----+

在 pyspark 中自动删除常量列的最佳方法是什么?

What is the best way to automatically drop constant columns in pyspark?

推荐答案

首先计算每列中的不同值，然后删除仅包含一个不同值的列:

Count distinct values in each column first and then drop columns that contain only one distinct value:

import pyspark.sql.functions as f
cnt = df_spark.agg(*(f.countDistinct(c).alias(c) for c in df_spark.columns)).first()
cnt
# Row(col1=5, col2=5, col3=1, col4=1)
df_spark.drop(*[c for c in cnt.asDict() if cnt[c] == 1]).show()
+----+----+
|col1|col2|
+----+----+
|   1|   1|
|   2|   2|
|   3|   3|
|   4|   4|
|   5|   5|
+----+----+

这篇关于如何在pyspark中自动删除常量列?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！