将Apache Spark中的所有资源与Yarn一起使用 [英] Using all resources in Apache Spark with Yarn

问题描述

我正在将Apache Spark与Yarn客户端一起使用.我的Spark集群中有4台工作PC，每台PC都有8个vcpus和30 GB的内存.我将执行程序的内存设置为2G，将实例数设置为33.我的工作需要10个小时才能运行，所有机器都闲置了80％.

I am using Apache Spark with Yarn client. I have 4 worker PCs with 8 vcpus each and 30 GB of ram in my spark cluster. Im set my executor memory to 2G and number of instances to 33. My job is taking 10 hours to run and all machines are about 80% idle.

我不了解执行程序内存和执行程序实例之间的相关性.每个Vcpu是否应该有一个实例?我应该将执行程序的内存设置为每台机器上的机器/#executor的内存吗?

I dont understand the correlation between executor memory and executor instances. Should I have an instance per Vcpu? Should I set the executor memory to be memory of machine/#executors per machine?

推荐答案

我们使用cassandra作为Spark的数据源.问题是没有足够的分区.我们需要进一步拆分数据.我们将#cassandra分区映射到spark分区的过程还不够小，我们只能生成10或20个任务，而不是100个任务.

We were using cassandra as our data source for spark. The problem was there were not enough partitions. We needed to split up the data more. Our mapping for # of cassandra partitions to spark partitions was not small enough and we would only generate 10 or 20 tasks instead of 100s of tasks.

这篇关于将Apache Spark中的所有资源与Yarn一起使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！