从另一个文本文件中读取基于键的文本文件作为列 [英] Read a Text File Based on key as a Column from another textFile

问题描述

我是 Spark 的新手，我正在尝试将 Table 作为文本文件加载到 Spark 中

I am new to Spark I am Trying to load Table into Spark as a textFIle

我想读取基于另一个文本文件列的文本文件，例如:: Id as A key如果 B.id 匹配 A.id 那么我必须将文件 B 读入 Spark

I want to read the textfile based on another Text file Column eg:: Id as A key If B.id Matches A.id Then I have to read File B into Spark

val file2=sc.textFile("path")

推荐答案

一种方法是同时读取文件和然后根据 id 字段加入它们并仅从表 b 中选择那些列，如下所示

One way would be read both the files & then join them based on id field and select only those columns from table b, some thing like below

val df1 = Seq((1, "Anu"),(2, "Suresh"),(3, "Usha"), (4, "Nisha")).toDF("id","name")
val df2 = Seq((1, 23),(2, 24),(3, 24), (4, 25), (5, 30), (6, 32)).toDF("id","age")

df1.as("df1").join(df2.as("df2"), df1("id") === df2("id"), "inner").select("df2.*").show()

输出:

+---+---+
| id|age|
+---+---+
|  1| 23|
|  2| 24|
|  3| 24|
|  4| 25|
+---+---+

这篇关于从另一个文本文件中读取基于键的文本文件作为列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！