增加的可用内存来星火 [英] Increasing available memory to Spark

问题描述

要增加最大可用内存使用：

To increase the MAX available memory I use :

export SPARK_MEM=1 g

另外，我可以使用

Alternatively I can use

val conf = new SparkConf()
             .setMaster("local")
             .setAppName("My application")
             .set("spark.executor.memory", "1g")
val sc = new SparkContext(conf)

我跑的过程需要超过1g多。我想用20克但我只是有一个可用的RAM8克。可以磁盘存储器RAM内存来扩充为星火工作的一部分，如果是这样，这是怎么实现的？

The process I'm running requires much more than 1g. I would like to use 20g but I just have 8g of RAM available. Can disk memory be augmented with RAM memory as part of a Spark job, if so how is this achieved ?

有介绍如何将作业分配到多个星火安装？火花文档

Is there a Spark doc which describes how to distribute jobs to multiple Spark installations ?

有关我使用所有默认设置的火花配置（在指定的http：// spark.apache.org/docs/0.9.0/configuration.html ），除了上面我所指定。
我有一台机器实例如下：

For spark configuration I'm using all defaults (specified at http://spark.apache.org/docs/0.9.0/configuration.html) except for what I have specified above. I have a single machine instance with following :

CPU : 4 cores
RAM : 8GB
HD : 40GB

更新：

我觉得这是我在寻找的DOC： HTTP： //spark.apache.org/docs/0.9.1/spark-standalone.html

I think this is the doc I'm looking for : http://spark.apache.org/docs/0.9.1/spark-standalone.html

推荐答案

如果你正在试图解决一台计算机上的一个问题，我不认为这是可行的使用星火。火花的一点是，它提供了一种分配在多台计算机的计算，特别是在数据不适合在一台机器上的情况。

If you are trying to solve a problem on a single computer, I do not think it is practical to use Spark. The point of Spark is that it provides a way to distribute computation across multiple machines, especially in cases where the data does not fit on a single machine.

这是说，只要设置 spark.executor.memory 到20克拿到20 GB的虚拟内存。一旦物理内存耗尽，交换将被用来代替。如果您配置足够的交换，你将能够利用20 GB的。但它开始交换你的进程很可能会放慢到爬行。

That said, just set spark.executor.memory to 20g to get 20 GB of virtual memory. Once the physical memory is exhausted, swap will be used instead. If you have enough swap configured, you will be able to make use of 20 GB. But your process will most likely slow down to a crawl when it starts swapping.

这篇关于增加的可用内存来星火的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！