AWS Glue执行程序内存限制 [英] AWS Glue executor memory limit
问题描述
我发现AWS Glue将执行器实例的内存限制设置为5 Gb --conf spark.executor.memory=5g
,有时,在大型数据集上,它会失败,并显示java.lang.OutOfMemoryError
.驱动程序实例--spark.driver.memory=5g
也是如此.
是否可以选择增加此值?
I found that AWS Glue set up executor's instance with memory limit to 5 Gb --conf spark.executor.memory=5g
and some times, on a big datasets it fails with java.lang.OutOfMemoryError
. The same is for driver instance --spark.driver.memory=5g
.
Is there any option to increase this value?
推荐答案
The official glue documentation suggests that glue doesn't support custom spark config.
AWS Glue内部还使用了几个参数名称, 您永远都不要设置:
There are also several argument names used by AWS Glue internally that you should never set:
-conf-AWS Glue内部.不要设置!
--conf — Internal to AWS Glue. Do not set!
-debug-AWS Glue的内部.不要设置!
--debug — Internal to AWS Glue. Do not set!
-mode-AWS Glue内部.不要设置!
--mode — Internal to AWS Glue. Do not set!
-JOB_NAME-AWS Glue的内部.不要设置!
--JOB_NAME — Internal to AWS Glue. Do not set!
关于解决此问题有更好的建议吗?
Any better suggestion on solving this problem?
这篇关于AWS Glue执行程序内存限制的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!