火花广播变量的大小是否有限制? [英] Is there any limit on size of a spark broadcast variable?

问题描述

我想知道spark中广播变量的大小是否有限制，因为我需要广播一个大对象(6GB，将来可能会更大). 我已经在主页和该站点中进行了搜索，但没有找到任何答案. 播放大型物体会遇到问题吗?

I am wondering if there is a limit size of a broadcast variable in spark because I need to broadcast a large object (6GB and can be larger in the future). I have searched in the home page and this site but nothing was found to answer my question. Would I ever have problems with broadcasting a very large object?

推荐答案

Java Integer.MAX_VALUE有一个限制.也就是说，大约有2.3 GB. 请在Spark Jira中查看更多信息: https://issues.apache.org /jira/browse/SPARK-5739

There is a limit of Java Integer.MAX_VALUE. That is to say about 2,3 GB. Please have a look here in Spark Jira for more information: https://issues.apache.org/jira/browse/SPARK-5739

这篇关于火花广播变量的大小是否有限制?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！