如何在Sparklyr中查找缺少数据的列 [英] how to find colums having missing data in sparklyr

问题描述

示例数据示例

Si      K       Ca      Ba  Fe  Type
71.78   0.06    8.75    0   0   1
72.73   0.48    7.83    0   0   1
72.99   0.39    7.78    0   0   1
72.61   0.57    na  0   0   na
73.08   0.55    8.07    0   0   1
72.97   0.64    8.07    0   na  1
73.09   na  8.17    0   0   1
73.24   0.57    8.24    0   0   1
72.08   0.56    8.3 0   0   1
72.99   0.57    8.4 0   0.11    1
na  0.67    8.09    0   0.24    1

我们可以使用以下代码将数据加载到 sparklyr 中

we can load data into sparklyr with the following code

sdf_copy_to(sc,sampledata)

我正在寻找一个查询，该查询返回具有NA值的列例如

I am looking for a query that returns the columns having NA values for example like

si k ca fe
1  1  1 2

推荐答案

由于 tbl_spark 的实现以及Spark和R语义上的不兼容，此问题实际上有些棘手.即使可以应用 colSums ，Spark SQL也不允许在布尔值和数字之间进行隐式转换.这意味着您必须显式应用 as.numeric :

This problem is actually a bit tricky due to tbl_spark implementation and incompatibilities in Spark and R semantics. Even if could apply colSums, Spark SQL doesn't allow implicit conversions between booleans and numerics. This means you have to explicitly apply as.numeric:

library(dplyr)

sampledata <- copy_to(sc, data.frame(x=c(1, NA, 2), y=c(NA, 2, NA), z=42))

sampledata %>% 
  mutate_all(is.na) %>% 
  mutate_all(as.numeric) %>%
  summarize_all(sum)

# Source:   lazy query [?? x 3]
# Database: spark_connection
      x     y     z
  <dbl> <dbl> <dbl>
1     1     2     0

这篇关于如何在Sparklyr中查找缺少数据的列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！