EMR Spark - TransportClient:无法发送 RPC [英] EMR Spark - TransportClient: Failed to send RPC
问题描述
我收到此错误,我尝试增加集群实例以及执行程序和驱动程序参数中的内存,但没有成功.
I'm getting this error, I tried to increase memory on cluster instances and in the executor and driver parameters without success.
17/05/07 23:17:07 ERROR TransportClient: Failed to send RPC 6465703946954088562 to ip-172-30-12-164.eu-central-1.compute.internal/172.30.12.164:34706: java.nio.channels.ClosedChannelException
有没有人有任何线索可以解决这个错误?
Does anyone have any clue to fix this error?
顺便说一句,我使用 YARN 作为集群管理器
BTW I'm using YARN as cluster manager
提前致谢
推荐答案
最后我解决了这个问题.这是由于磁盘空间不足.一列 hadoop 日志显示:
Finally I resolved the problem. It was due to insufficient disk space. One column of hadoop logs showed:
Hadoop YARN:1/1 本地目录不好:/var/lib/hadoop-yarn/cache/yarn/nm-local-dir;1/1 日志目录不好:/var/log/hadoop-yarn/containers
Hadoop YARN: 1/1 local-dirs are bad: /var/lib/hadoop-yarn/cache/yarn/nm-local-dir; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers
如果出现此错误,请腾出一些磁盘空间!"
"If you are getting this error, make some disk space!"
要查看此错误,我必须激活 EMR 中的纱线日志.见
To see this error I have to activate the yarn logs in EMR. See
http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
为了访问集群 ec2 实例中的日志端口,我为它更改了安全组
To have access to the logs port in the cluster ec2 instances I changed security groups for it
即:
主实例在这里监听:172.30.12.84:8088 和核心这里的实例:172.30.12.21:8042
master instance was listening here: 172.30.12.84:8088 And core instance here: 172.30.12.21:8042
最后,我解决了在 etl.py 中更改其他具有更大磁盘的实例类型的问题:
Finally I fixed the problem changing in etl.py the type of instances by other ones with bigger disks:
大师:m3.2xlarge
核心:c3.4xlarge
master: m3.2xlarge
core: c3.4xlarge
这篇关于EMR Spark - TransportClient:无法发送 RPC的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!