并行高效地运行多个作业 [英] Run several jobs parallelly and Efficiently

问题描述

OS:Cent-OS

OS: Cent-OS

我有一些30,000作业(或脚本)要运行.每个工作需要3-5分钟.我有48 CPUs(nproc = 48).我可以并行使用40 CPUs to run 40 Jobs.请建议一些脚本或工具通过并行运行40个Jobs来处理30,000个Jobs.

I have some 30,000 jobs(or Scripts) to run. Each job takes 3-5 Min. I have 48 CPUs(nproc = 48). I can use 40 CPUs to run 40 Jobs parallelly. please suggest some script or tools can handle 30,000 Jobs by running each 40 Jobs parallely.

我做了什么:

• 我创建了40个不同的文件夹，并通过为每个目录创建一个shell脚本来并行执行作业.

• I created 40 Different folders and executed the jobs parallely by creating a shell script for each directory.

我想知道下一次处理此类工作的更好方法.

I want to know better ways to handle this kind of jobs next time.

推荐答案

正如Mark Setchell所说:GNU并行.

As Mark Setchell says: GNU Parallel.

find scripts/ -type f | parallel

如果您坚持要保持8个CPU空闲:

If you insists on keeping 8 CPUs free:

find scripts/ -type f | parallel -j-8

但是通常简单地使用nice效率更高，因为当没有其他人需要它们时，它将为您提供全部48个内核:

But usually it is more efficient simply to use nice as that will give you all 48 cores when no one else needs them:

find scripts/ -type f | nice -n 15 parallel

要了解更多信息:

• 观看介绍视频以获取快速介绍: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
• 遍历该教程(man parallel_tutorial).您命令行 爱着你.

• Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
• Walk through the tutorial (man parallel_tutorial). You command line with love you for it.

这篇关于并行高效地运行多个作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！