在Scala中将数据帧作为可选功能参数传递 [英] Passing data frame as optional function parameter in Scala

问题描述

有没有一种方法可以将数据帧作为Scala中的可选输入函数参数传递? 例如:

Is there a way that I can pass a data frame as an optional input function parameter in Scala? Ex:

def test(sampleDF: DataFrame = df.sqlContext.emptyDataFrame): DataFrame = {


}


df.test(sampleDF)

尽管我在此处传递了一个有效的数据帧，但始终将其分配给一个空的数据帧，如何避免这种情况?

Though I am passing a valid data frame here , it is always assigned to an empty data frame, how can I avoid this?

推荐答案

是的，您可以将dataframe作为参数传递给函数

Yes you can pass dataframe as a parameter to a function

让我们说您有dataframe作为

import sqlContext.implicits._

val df = Seq(
  (1, 2, 3),
  (1, 2, 3)
).toDF("col1", "col2", "col3")

是

+----+----+----+
|col1|col2|col3|
+----+----+----+
|1   |2   |3   |
|1   |2   |3   |
+----+----+----+

您可以将其传递给以下函数

you can pass it to a function as below

import org.apache.spark.sql.DataFrame
def test(sampleDF: DataFrame): DataFrame = {
  sampleDF.select("col1", "col2") //doing some operation in dataframe
}

val testdf = test(df)

testdf为

+----+----+
|col1|col2|
+----+----+
|1   |2   |
|1   |2   |
+----+----+

已编辑

正如eliasah指出的那样，@ Garipaso希望使用可选参数.可以通过将函数定义为

As eliasah pointed out that @Garipaso wanted optional argument. This can be done by defining the function as

def test(sampleDF: DataFrame = sqlContext.emptyDataFrame): DataFrame = {
  if(sampleDF.count() > 0) sampleDF.select("col1", "col2") //doing some operation in dataframe
  else sqlContext.emptyDataFrame  
}

如果我们将有效数据框传递为

If we pass a valid dataframe as

test(df).show(false)

它将输出为

+----+----+
|col1|col2|
+----+----+
|1   |2   |
|1   |2   |
+----+----+

但是，如果我们不将参数传递为

But if we don't pass argument as

test().show(false)

我们将获得空的数据框为

we would get empty dataframe as

++
||
++
++

我希望答案会有所帮助

这篇关于在Scala中将数据帧作为可选功能参数传递的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！