在IntelliJ Scala控制台中运行时如何设置Spark MemoryStore大小? [英] How to set Spark MemoryStore size when running in IntelliJ Scala Console?

问题描述

我正在Linux 64(Fedora 25)上的Intellij(CE 2017.1)Scala控制台中以脚本形式运行Spark代码.我在一开始就设置了SparkContext:

I am running Spark code as script in Intellij (CE 2017.1) Scala Console on Linux 64 (Fedora 25). I set SparkContext at the start:

import org.apache.spark.{SparkConf, SparkContext}
val conf = new SparkConf().
  setAppName("RandomForest").
  setMaster("local[*]").
  set("spark.local.dir", "/spark-tmp").
  set("spark.driver.memory", "4g").
  set("spark.executor.memory", "4g")

val sc = new SparkContext(conf)

但是正在运行的SparkContext始终以同一行开头:

But the running SparkContext always starts with the same line:

17/03/27 20:12:21 INFO SparkContext:运行Spark版本2.1.0

17/03/27 20:12:21 INFO SparkContext: Running Spark version 2.1.0

17/03/27 20:12:21信息MemoryStore:MemoryStore开始时的容量为871.6 MB

17/03/27 20:12:21 INFO MemoryStore: MemoryStore started with capacity 871.6 MB

17/03/27 20:12:21信息BlockManagerMasterEndpoint:使用871.8 MB RAM，BlockManagerId(驱动程序，192.168.1.65、38119，无)注册块管理器192.168.1.65:38119

17/03/27 20:12:21 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.65:38119 with 871.8 MB RAM, BlockManagerId(driver, 192.168.1.65, 38119, None)

Spark Web UI中的Executors选项卡显示相同的金额. 在启动前从终端导出_JAVA_OPTIONS =-Xms2g -Xmx4g"对此也无效.

And the Executors tab in the Spark web UI shows the same amount. Exporting _JAVA_OPTIONS="-Xms2g -Xmx4g" from the terminal before start also had no effect here.

推荐答案

增加Web UI的Spark MemoryStore并最终增加Storage Memory Executors选项卡的唯一方法是直接在Intellij Scala控制台设置中的VM选项中添加-Xms2g -Xmx4g.开始.

The only way to increase Spark MemoryStore and eventually Storage memory Executors tab of web UI was to add -Xms2g -Xmx4g in VM options directly in Intellij Scala Console settings before start.

现在信息行将打印:

17/03/27 20:12:21信息MemoryStore:MemoryStore开始时的容量为2004.6 MB

17/03/27 20:12:21 INFO MemoryStore: MemoryStore started with capacity 2004.6 MB

17/03/27 20:12:21信息BlockManagerMasterEndpoint:使用2004.6 MB RAM，BlockManagerId(驱动程序，192.168.1.65、41997，无)注册块管理器192.168.1.65:41997

17/03/27 20:12:21 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.65:41997 with 2004.6 MB RAM, BlockManagerId(driver, 192.168.1.65, 41997, None)

，Spark Web UI Executors选项卡的存储内存"显示为2.1 GB.

and the Spark web UI Executors tab Storage Memory shows 2.1 GB.

这篇关于在IntelliJ Scala控制台中运行时如何设置Spark MemoryStore大小?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！