从外部文件过滤数据帧 [英] filter dataframe from external file

问题描述

我想从外部文件中过滤数据框. 这是我的数据框的样子:

i want to filter my dataframe from an external file. this is how my dataframe look like:

val Insert=Append_Ot.filter(col("Name2").equalTo("brazil") || col("Name2").equalTo("france") || col("Name2").equalTo("algeria")|| col("Name2").equalTo("tunisia") || col("Name2").equalTo("egypte")  )

我要过滤它们的国家/地区的数量是可变的，因此在此文件的外部创建了一个文件:

The number of countries that i want to filter them is changeable, so created an external this file:

 1  brazil
 2  france
 3  algeria
 4  tunisia
 5  egypte

我想创建UDF来从此文件中过滤数据框.

i want to create UDF to filter my dataframe from this file.

谢谢

推荐答案

您需要从要过滤的文件中创建一个Seq. 看起来像这样:

You need to create a Seq from the file with which you want to filter. Something that looks like this:

val l = List("Brasil", "Algeria", "Tunisia", "Egypt")

您可以使用textFile方法.假设您的文件包含:

You can use textFile method. Suppose your file contains:

1 Algeria
2 Tunisia
3 Brasil
4 Egypt

您可以使用:

val countries = sc.textFile("hdfs://namenode/user/cloudera/file").map(_.split(" ")(1)).collect

这将为您提供:

countries : Array[String] = Array(Algeria, Tunisia, Brasil, Egypt)

然后，在列Name2

val Insert = Append_Ot.where($"Name2".isin( countries : _* ) )

这篇关于从外部文件过滤数据帧的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！