星火VS弗林克可用内存不足 [英] Spark vs Flink low memory available

问题描述

我有建立一个Spark和弗林克的k-means应用程序。
我的测试案例是一个3节点群集上1万点集群。

I have build a Spark and Flink k-means application. My test case is a clustering on 1 million points on a 3 node cluster.

当内存瓶颈开始，弗林克开始外包给磁盘和工作缓慢，但工程。
然而，星火失去executers如果内存已满并再次开始（无限循环？）。

When in-memory bottlenecks begin, Flink starts to outsource to disk and work slowly but works. However, Spark lose executers if the memory is full and starts again (infinite loop?).

我尝试定制存储器从邮件列表在这里，感谢帮助设置。但是星火确实还无法正常工作。

I try to customize the memory setting with the help from the mailing list here, thanks. But Spark does still not work.

是否有必要，必须设置任何配置？我的意思是弗林克适用于低内存，星火也必须能;还是不？

Is it necessary to have any configurations to be set? I mean Flink works with low memory, Spark must also be able to; or not?

推荐答案

我不是专家星火（和我是一个弗林克贡献者）。据我知道，如果没有足够的主内存星火不能溢出到磁盘。这是对火花弗林克的一个优点。然而，星火宣布了一项新的项目叫钨都，使类似弗林克托管内存。我不知道这功能已经可用：<一href=\"https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html\">https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html

I am not a Spark expert (and I am an Flink contributor). As far as I know, Spark is not able to spill to disk if there is not enough main memory. This is one advantage of Flink over Spark. However, Spark announced a new project call "Tungsten" to enable managed memory similar to Flink. I don't know if this feature is already available: https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html

有一对夫妇约星火出现内存不足的问题，所以这个问题（互联网搜索与火花内存不足产量许多成果，太）：

There are a couple of SO question about Spark out of memory problems (an Internet search with "spark out of memory" yield many results, too):

<一个href=\"https://stackoverflow.com/questions/21138751/spark-java-lang-outofmemoryerror-java-heap-space\">spark java.lang.OutOfMemoryError：Java堆空间
<一href=\"https://stackoverflow.com/questions/22637518/spark-runs-out-of-memory-when-grouping-by-key\">Spark通过关键分组时耗尽内存
星火出内存

这些帮助可能的。

这篇关于星火VS弗林克可用内存不足的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！