SPARK可以正确使用多核吗? [英] Can SPARK use multicore properly?

问题描述

我已经读过关于spark的内容，发现spark是用scala编写的.由于scala是一种功能语言，例如erlang，因此可以正确使用多核.正确吗?

I've read about spark and I found out that spark is written in scala. Since scala is a functional language, like erlang, it can use multi core properly. Is that correct?

我想知道我是否可以在具有多核处理器的分布式系统中使用spark.一个任务可以同时使用所有内核吗?我读过YARN会为每个不同的任务分配不同的内核，但是在这种情况下，它是一个任务

I'm wondering if I can use spark in distributed system which have multicore processors. Can a single task use all cores at the same time? I've read YARN will assign different cores on each different tasks, but in this case, it is a single task

而且，使用JAVA(hadoop)中的多线程编程来使用每台计算机中的所有内核是否足够?既然Linux Scheduler是基于线程的?

And, is it just enough to use multi thread programming in JAVA (hadoop) to use all cores in each computer? Since linux scheduler is based on threads?

推荐答案

是的，因为这是其既定的目的，所以可以拆分和并行化可并行化的内容.您甚至可以指定每个执行者要使用的内存量.

Yes, it can, as this is its stated purpose — to split and parallelize what is parallelizeable. You can even specify amount of memory to be used by each executor.

但是，某些任务无法并行执行，这就是为什么Spark有时只占据一个核心的原因.

However, some tasks cannot be parallelized, which is why sometimes Spark only occupies one core.

如果您使用Spark外壳，请确保设置要使用的内核数，如对此问题的回答中所述为什么Spark不使用本地计算机上的所有内核

If you use the Spark shell, make sure you set the number of cores to use, as it is said in the answer to this question Why is Spark not using all cores on local machine

来源:Spark官方文档 https://spark.apache.org/docs/Latest/configuration.html

Source: official Spark docs https://spark.apache.org/docs/latest/configuration.html

这篇关于SPARK可以正确使用多核吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！