如何基于Json Object Scala的调用方法产生火花? [英] How call method based on Json Object scala spark?

问题描述

我有以下两个功能

def method1(ip:String,r:Double,op:String)={
val data = spark.read.option("header", true).csv(ip).toDF()
val r3= data.select("c", "S").dropDuplicates("C", "S").withColumn("R", lit(r))
r3.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").save(op)
}

def method2(ip:String,op:String)={
val data = spark.read.option("header", true).csv(ip).toDF()
val r3= data.select("c", "S").dropDuplicates("C", "StockCode")
r3.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").save(op)
}

我想通过基于Json对象parameter调用此方法. 例如，如果我的输入json如下

I want to call this methods by based on Json object parameter . for example if my input json is like below

{"name":"method1","ip":"Or.csv","r":1.0,"op":"oppath"}

它必须调用method1和"Or.csv",1.0,oppath"作为参数，即json对象名称中的表示方法名称，而铰孔字段是参数.

It has to call method1 and "Or.csv",1.0,"oppath" as parameters I.e. in json object name indicate method name, and reaming fields are parameters.

请帮助我.

推荐答案

首先，我们需要通过spark将Json读取到数据帧中.

First we need to read Json through spark into a dataframe.

val df = sqlContext.read.json("path to the json file")

应该为您提供数据框

scala> df.show()
+------+-------+------+---+
|    ip|   name|    op|  r|
+------+-------+------+---+
|Or.csv|method1|oppath|1.0|
+------+-------+------+---+

下一个

   scala> def method1(ip:String,r:Double,op:String)={
         | val data = spark.read.option("header", true).csv(ip).toDF()
         | val r3= data.select("c", "S").dropDuplicates("C", "S").withColumn("R", lit(r))
         | r3.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").save(op)
         | }

method1: (ip: String, r: Double, op: String)Unit

下一个

   scala> def method2(ip:String,op:String)={
         | val data = spark.read.option("header", true).csv(ip).toDF()
         | val r3= data.select("c", "S").dropDuplicates("C", "StockCode")
         | r3.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").save(op)
         | }

method2: (ip: String, op: String)Unit

下一个

scala>df.withColumn("methodCalling",when($"name" === "method1",method1(df.first().getString(1),df.first().getDouble(2),df.first().getString(3))).otherwise(when($"name" === "method2", method2(df.first().getString(1),df.first().getString(2)))))

它将基于Json Object调用method1或method2.

it will call method1 or method2 based on Json Object.

这篇关于如何基于Json Object Scala的调用方法产生火花?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！