如何在数据框中引用广播变量 [英] How to refer broadcast variable in dataframes

问题描述

我使用spark1.6.我试图广播RDD，但不确定如何访问数据帧中的广播变量?

I use spark1.6. I tried to broadcast a RDD and am not sure how to access the broadcasted variable in the data frames?

我有两个数据框员工&部门.

I have two dataframes employee & department.

员工数据框

-------------------
Emp Id | Emp Name | Emp_Age
------------------
1 | john | 25

2 | David | 35

部门数据框

--------------------
Dept Id | Dept Name | Emp Id
-----------------------------
1 | Admin | 1

2 | HR | 2

import scala.collection.Map

val df_emp = hiveContext.sql("select * from emp")

val df_dept = hiveContext.sql("select * from dept")

val rdd = df_emp.rdd.map(row => (row.getInt(0),row.getString(1)))

val lkp = rdd.collectAsMap()

val bc = sc.broadcast(lkp)

print(bc.value.get(1).get)

--Below statement doesn't work

val combinedDF = df_dept.withColumn("emp_name",bc.value.get($"emp_id").get)

1. 如何在上面的CombinedDF语句中引用广播变量?
2. 如果lkp不返回任何值，该如何处理?
3. 是否有一种方法可以从lkp返回多个记录(假设在查找中如果emp_id = 1有2条记录，我想同时获得两条记录)
4. 如何从广播中返回多个值...(emp_name和emp_age)

推荐答案

如何在上面的CombinedDF语句中引用广播变量?

How do I refer the broadcast variable in the above combinedDF statement?

使用udf.如果emp_id是Int

val f = udf((emp_id: Int) =>  bc.value.get(emp_id))

df_dept.withColumn("emp_name", f($"emp_id"))

如果lkp不返回任何值，如何处理?

How to handle if the lkp doesn't return any value?

请勿使用如上所示的get

有没有办法从lkp返回多个记录

Is there a way to return multiple records from the lkp

使用groupByKey:

val lkp = rdd.groupByKey.collectAsMap()

和explode:

df_dept.withColumn("emp_name", f($"emp_id")).withColumn("emp_name", explode($"emp_name"))

，或者跳过所有步骤，然后按broadcast :

or just skip all the steps and broadcast:

import org.apache.spark.sql.functions._

df_emp.join(broadcast(df_dep), Seq("Emp Id"), "left")

这篇关于如何在数据框中引用广播变量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！