如何与QUOT;负选取[在火花的数据框列 [英] How to "negative select" columns in spark's dataframe

问题描述

我不出来，但猜测这很简单。我有一个火花数据帧DF。该专用栏A，B和C。现在让我们说我有一个包含该DF的列名称的数组：

I can't figure it out, but guess it's simple. I have a spark dataframe df. This df has columns "A","B" and "C". Now let's say I have an Array containing the name of the columns of this df:

column_names = Array("A","B","C")

我想要做一个 df.select（）以这样的方式，我可以指定哪些列的不的选择。
例如：让我们说我不想选择列B。我试过

I'd like to do a df.select() in such a way, that I can specify which columns not to select. Example: let's say I do not want to select columns "B". I tried

df.select(column_names.filter(_!="B"))

但是，这并不工作，因为

but this does not work, as

org.apache.spark.sql.DataFrame
   不能被施加到（阵列​​[字符串]）

org.apache.spark.sql.DataFrame cannot be applied to (Array[String])

因此​​，<一个href=\"https://spark.apache.org/docs/1.3.0/api/java/org/apache/spark/sql/DataFrame.html#select(java.lang.String,%20scala.collection.Seq)\"相对=nofollow>这里它说，它应该用序列，而不是工作。然而，试图

So, here it says it should work with a Seq instead. However, trying

df.select(column_names.filter(_!="B").toSeq)

结果

org.apache.spark.sql.DataFrame
   不能被施加到（序号[字符串]）

org.apache.spark.sql.DataFrame cannot be applied to (Seq[String]).

我在做什么错了？

推荐答案

由于星火1.4 您可以使用<一个href=\"https://github.com/rakeshchalasani/spark/blob/ce2ec09ef702ad7191638299adf63e7af380ac18/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala#L865\"相对=nofollow> 降 方式：

Since Spark 1.4 you can use drop method:

斯卡拉

case class Point(x: Int, y: Int)
val df = sqlContext.createDataFrame(Point(0, 0) :: Point(1, 2) :: Nil)
df.drop("y")

的Python

df = sc.parallelize([(0, 0), (1, 2)]).toDF(["x", "y"])
df.drop("y")
## DataFrame[x: bigint]

这篇关于如何与QUOT;负选取[在火花的数据框列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！