如果spark数据框的特定列中的所有条目为null，则删除 [英] Drop if all entries in a spark dataframe's specific column is null

问题描述

使用Pyspark，如何选择/保留包含非空值的DataFrame的所有列；或等效地删除所有不包含数据的列.

Using Pyspark, how can I select/keep all columns of a DataFrame which contain a non-null value; or equivalently remove all columns which contain no data.

根据Suresh请求

for column in media.columns:
    if media.select(media[column]).distinct().count() == 1:
        media = media.drop(media[column])

在这里，我假设如果count为1，则应该为Nan.但是我想检查一下是否是楠.如果还有其他内置的spark函数，请告诉我.

Here I assumed that if count is one, then it should be Nan. But I wanted to check whether that is Nan. And if there's any other inbuilt spark function, let me know.

推荐答案

我尝试了.说，我有一个如下数据框，

I tried my way. Say, I have a dataframe as below,

>>> df.show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   1|   2|null|
|null|   3|null|
|   5|null|null|
+----+----+----+

>>> df1 = df.agg(*[F.count(c).alias(c) for c in df.columns])
>>> df1.show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   2|   2|   0|
+----+----+----+

>>> nonNull_cols = [c for c in df1.columns if df1[[c]].first()[c] > 0]
>>> df = df.select(*nonNull_cols)
>>> df.show()
+----+----+
|col1|col2|
+----+----+
|   1|   2|
|null|   3|
|   5|null|
+----+----+

这篇关于如果spark数据框的特定列中的所有条目为null，则删除的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！