从另一个textFile读取基于键的文本文件作为列 [英] Read a Text File Based on key as a Column from another textFile

问题描述

我是Spark的新手，我正在尝试将Table作为textFIle加载到Spark中

I am new to Spark I am Trying to load Table into Spark as a textFIle

我想基于另一个文本文件列读取文本文件，例如:: Id作为A键 如果B.id与A.id相匹配，那么我必须将文件B读入Spark

I want to read the textfile based on another Text file Column eg:: Id as A key If B.id Matches A.id Then I have to read File B into Spark

val file2=sc.textFile("path")

推荐答案

一种方法是同时读取文件&然后根据id字段将它们加入，并仅从表b中选择那些列，如下所示

One way would be read both the files & then join them based on id field and select only those columns from table b, some thing like below

val df1 = Seq((1, "Anu"),(2, "Suresh"),(3, "Usha"), (4, "Nisha")).toDF("id","name")
val df2 = Seq((1, 23),(2, 24),(3, 24), (4, 25), (5, 30), (6, 32)).toDF("id","age")

df1.as("df1").join(df2.as("df2"), df1("id") === df2("id"), "inner").select("df2.*").show()

输出:

+---+---+
| id|age|
+---+---+
|  1| 23|
|  2| 24|
|  3| 24|
|  4| 25|
+---+---+

这篇关于从另一个textFile读取基于键的文本文件作为列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！