Spark有效地过滤小数据帧中存在的大数据帧中的条目 [英] Spark efficiently filtering entries from big dataframe that exist in a small dataframe

问题描述

我有一个 Spark 程序，它读取一个相对较大的数据帧(~3.2 TB)，其中包含 2 列:id、name 和另一个包含单列的相对较小的数据帧(~20k 个条目):id

I have a Spark program that reads a relatively big dataframe (~3.2 terabyte) that contains 2 columns: id, name and another relatively small dataframe (~20k entries) that contain a single column: id

如果它们出现在小数据框中，我想要做的是从大数据框中获取 ID 和名称

What I'm trying to do is take both the id and the name from the big dataframe if they appear in the small dataframe

我想知道什么是有效的解决方案，为什么?我想到的几个选项:

I was wondering what would be an efficient solution to get this working and why? Several options I had in mind:

1. 广播加入 2 个数据帧
2. 广播小数据帧并将其收集为字符串数组，然后对大数据帧进行过滤并将 isin 与字符串数组一起使用

还有其他我没有提到的选项吗?

Are there any other options that I didn't mention here?

如果有人也能解释为什么一个特定的解决方案比另一个更有效，我将不胜感激

I'll appreciate it if someone could also explain why a specific solution is more efficient than the other

提前致谢

推荐答案

AFAIK 这完全取决于您处理的数据大小和性能，

AFAIK its all depends on the size of data you are handling and performance ,

• 如果您使用 broadcast 函数，则默认大小为 10mb(对于通过 spark.sql.autobroadcastjointhreshhold 的小数据框，请参阅 我的答案 )您可以根据您的数据增加或减少大小.此外，广播数据将成为 sql 执行计划的一部分，并进一步指向催化剂优化器以进行进一步优化.另请参阅我的回答此处

• if you use broadcast function then default size is 10mb (for your small dataframe via spark.sql.autobroadcastjointhreshhold see my answer ) you can increase or decrease the size based on your data. Also, braodcasted data will be part of sql execution plan and further will be pointer to catalyst optimizer to do further optimization. Also see my answer here

广播共享变量(您想使用isin)没有上述优势.

where as broadcast shared variable (which you want to use isin) doesnt have above advantage.

请在我的评论中查看上面链接中的答案

pls see my answer in above link in my comment

这篇关于Spark有效地过滤小数据帧中存在的大数据帧中的条目的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！