老一代堆内存逐渐增加 [英] Gradual Increase in old generation heap memory
问题描述
我在Spark Streaming中面临着一个非常奇怪的问题.我正在使用spark 2.0.2，节点数3，执行程序数3 {1接收器和2处理器}，每个执行程序的内存2 GB，每个执行程序1的内核.批处理间隔为10秒.我的批量大小约为. 1000条记录(约150 KB).
I am facing a very strange issue in spark streaming. I am using spark 2.0.2, number of nodes 3, number of executors 3 {1 receiver and 2 processor}, memory per executor 2 GB, cores per executor 1. The batch interval is 10 sec. My batch size is approx. 1000 records (approx 150 KB).
我的批处理时间从最初的2秒逐渐增加到几分钟，但在最初的40-50小时中，它运行良好.之后，调度延迟和处理时间开始激增.
The processing time of my batch is increasing gradually from 2sec initially to few mins but for the first 40-50 hours it runs quite well. After that, scheduling delay and processing time start shooting up.
我尝试查看GC，并且驱动程序的旧版本堆内存容量不断增加.这可能是原因吗?我使用 jstat 监视了堆内存.在六个小时的时间内，容量从1161216字节增加到1397760字节.
I had tried taking a look on the GC and there is a continuous increase in the old generation heap memory capacity of driver. Could this be the reason? I monitored heap memory using jstat. The increase in capacity is from 1161216 Bytes to 1397760 Bytes over a period of six hours.
运行驱动程序的计算机具有8个物理核心，并且经过40-50小时的流传输后，该计算机上的CPU使用率在所有8个核心上均为100％，并且老一代堆的使用已满，因此 FullGC的很常见.
The machine on which the driver is running has 8 physical cores and after 40-50 hours of streaming, the CPU usage on the machine is 100% on all the 8 cores and the old generation heap usage is full and the FullGC's are very frequent.
我还看到了 jira问题，上面说有记忆火花流中出现泄漏，但它也表示在火花1.5之后已解决.这相关吗?
I had also seen a jira issue which says there is a memory leak in spark streaming but it also says that it was resolved after spark 1.5. Is this relevant ?
修改:
在应用程序启动后大约50小时，我也进行了堆转储.
I have also taken the heap dump approximately 50 hr after application start.
为什么scala.collection.immutable.$colon$colon的实例如此之多?
Why are there so many instances of scala.collection.immutable.$colon$colon?
推荐答案
问题已得到解决现在.请看一下并升级您的火花.
The issue has been fixed now. Kindly take a look at this and upgrade your spark.
这篇关于老一代堆内存逐渐增加的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
