当为spark master指定local [n1，n2，n3]时，三个参数是什么? [英] When specifying local[n1,n2,n3] for spark master, what are the three parameters?

问题描述

对于发动星火，我已经看到:

For launching spark, I have seen:

--master local[n1,n2,n3] 

其中n1，n2和n3是整数.

where n1, n2, and n3 are integers.

这些指的是什么?

推荐答案

主规范在

The master specification is parsed in SparkContext.createTaskScheduler. (See the link for the implementation.) The possibilities with local are:

• local使用1个线程.
• local[N]使用N个线程.
• local[*]使用的线程数与内核数一样.
• local[N, M]和local[*, M]与上面类似，但是将最大任务失败数设置为M.这允许您在本地运行时启用重试. (通常会禁用本地重试.启用它们对于测试很有用.)
• local-cluster[numSlaves, coresPerSlave, memoryPerSlave]在已配置的单独进程中启动执行程序，但不需要运行的工作程序和主服务器.这是在单元测试中模拟集群的一种轻量级方法. (另请参见 SPARK-595 .)

• local uses 1 thread.
• local[N] uses N threads.
• local[*] uses as many threads as there are cores.
• local[N, M] and local[*, M] are like above, but set the maximal task failures to M. This allows you to enable retries when running locally. (Normally local retries are disabled. Enabling them is useful for testing.)
• local-cluster[numSlaves, coresPerSlave, memoryPerSlave] starts executors in separate processes as configured, but it does not require running workers and masters. It's a lightweight way to simulate a cluster in unit tests. (See also SPARK-595.)

这篇关于当为spark master指定local [n1，n2，n3]时，三个参数是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！