评估火花广播变量的最大大小 [英] Evaluate the max size for a spark broadcast variable

问题描述

我试图测量我可以使用火花广播广播的变量的最大大小。

I'm trying to measure the max size of variable I can broadcast using spark broadcast.

我没有找到关于这个问题的任何解释。

I didn't find any explanation regarding this issue.

是否有人衡量它？请问spark有配置广播大小吗？

did someone measure it? does spark has configuration for broadcast size?

推荐答案

目前为〜2GB。在串行化过程中，您广播的任何内容都将转换为java字节数组，并且由于java数组的最大大小为Integer.MAX_VALUE，您将获得此限制。目前可能有一些努力增加了这个限制： SPARK-6235

It's currently ~2GB. Anything you broadcast is converted into java byte array during serialization and as java arrays have max size Integer.MAX_VALUE you get this limit. There may currently be some effort increasing this limit: SPARK-6235

这篇关于评估火花广播变量的最大大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！