当我的qsub作业的服务器上完成如何自动运行bash脚本？ [英] how to automatically run a bash script when my qsub jobs are finished on a server?

问题描述

我想运行一个脚本时，我一切所发送到服务器的工作就完成了。

I would like to run a script when all of the jobs that I have sent to a server are done.

例如，我送

ssh server "for i in config*; do qsub ./run 1 $i; done"

和我回去启动的作业列表。我想自动在服务器上启动另一个脚本一次处理这些作业的输出都已经完成。

And I get back a list of the jobs that were started. I would like to automatically start another script on the server to process the output from these jobs once all are completed.

我想AP preciate任何意见，这将有助于我避免以下不雅的解决方案：

I would appreciate any advice that would help me avoid the following inelegant solution:

如果我保存每个1000作业标识的来自一个单独的文件上面的电话，我可以检查每个文件的内容与正在运行的作业当前列表，从调用输出，即：

If I save each of the 1000 job id's from the above call in a separate file, I could check the contents of each file against the current list of running jobs, i.e. output from a call to:

ssh qstat

我只需要检查每一个半小时，但我想有一个更好的办法。

I would only need to check every half hour, but I would imagine that there is a better way.

推荐答案

这取决于你所使用的作业调度和什么版本，但有能太多，如果也可以做你的结果的处理采取另一种方法有点在同一个队列的工作。

It depends a bit on what job scheduler you are using and what version, but there's another approach that can be taken too if your results-processing can also be done on the same queue as the job.

在较新版本的扭矩管理大量相关工作的一个非常简便的方法（和网格引擎，和其他人）是启动任何个人的工作作为作业阵列（参见的http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm#-t).这就需要以某种方式号码单独运行，这可能是也可能不是方便映射;但如果你能为你的工作做到这一点，它大大简化了管理工作;您可以qsub命令他们都在一条线，你可以QDEL或QHOLD他们都在一次（同时还具有单独处理工作的能力）。

One very handy way of managing lots of related job in more recent versions of torque (and with grid engine, and others) is to launch the any individual jobs as a job array (cf. http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm#-t). This requires mapping the individual runs to numbers somehow, which may or may not be convenient; but if you can do it for your jobs, it does greatly simplify managing the jobs; you can qsub them all in one line, you can qdel or qhold them all at once (while still having the capability to deal with jobs individually).

如果你这样做，那么你可以提交其中有作业的数组这将只运行一次阵列中所有的作业都是有依赖性的分析工作完成：（参见<一个href=\"http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm#dependencyExamples\" rel=\"nofollow\">http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm#dependencyExamples).提交作业看起来像：

If you do this, then you could submit an analysis job which had a dependency on the array of jobs which would only run once all of the jobs in the array were complete: (cf. http://docs.adaptivecomputing.com/torque/4-1-4/Content/topics/commands/qsub.htm#dependencyExamples). Submitting the job would look like:

qsub analyze.sh -W depend=afterokarray:427[]

在这里analyze.sh有剧本做了分析，427是作业的阵列您启动的作业ID。 （按[]表示只有所有人都完成后运行）。的语法的不同为其他调度（例如，SGE / OGE），但思路是一样的。

where analyze.sh had the script to do the analysis, and 427 would be the job id of the array of jobs you launched. (The [] means only run after all are completed). The syntax differs for other schedulers (eg, SGE/OGE) but the ideas are the same.

获得这一权利可以采取一些做的，当然特里斯坦的方法具有简单，并与任何调度工作的优势;但学习使用工作数组在这种情况下，如果你会做很多的，这可能是值得你的时间。

Getting this right can take some doing, and certainly Tristan's approach has the advantage of being simple, and working with any scheduler; but learning to use job arrays in this situation if you'll be doing alot of this may be worth your time.

这篇关于当我的qsub作业的服务器上完成如何自动运行bash脚本？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！