在调用slurm sbatch之前为日志文件创建目录 [英] Create directory for log file before calling slurm sbatch

问题描述

Slurm sbatch 将stdout和stderr定向到 -o 和 -e 标志指定的文件，但是如果文件路径包含不存在的目录.有什么方法可以自动为我的日志文件创建目录?

Slurm sbatch directs stdout and stderr to the files specified by the -o and -e flags, but fails to do so if the filepath contains directories that don't exist. Is there some way to automatically make the directories for my log files?

• 每次手动创建这些目录的效率很低，因为我要运行每个批处理提交数十次.
• 让作业名称的变化形式存在于文件名而不是目录中，这会导致大量混乱且组织混乱的日志，当我需要检查作业的执行情况时必须对其进行排序.

我发现执行此操作的唯一方法是将对 sbatch 的调用包装在bash脚本中，该bash脚本的长度比这种小事情所需的时间长很多倍.我在下面提供了一个简短的示例.

The only way I've found to do this is to wrap my calls to sbatch inside bash scripts that are many times longer than seems necessary for such a small thing. I've included a shortened example below.

#!/bin/bash
# Set up and run job array for my_script.py, which takes as positional
# arguments a config file (passed via $1) and an array index.

#SBATCH --array=1-100
#SBATCH -n 1
#SBATCH -t 12:00:00
#SBATCH -p short
#SBATCH -J sim_sumstats
#SBATCH --mem=1600

# Initialize variables used for script control flow
sub_or_main='sub'

# Parse options
while getopts ":A" opt; do
    case $opt in
        A)
            sub_or_main='main'
            ;;
        \?)
            # Capture invalid options
            echo "Invalid option: -$OPTARG" >&2
            exit 1
            ;;
    esac
done

shift $((OPTIND - 1))

# Either run the submit script or the main array
if [ $sub_or_main == 'sub' ]; then
    # Submit script creates folders for log files, then calls sbatch on this
    # script in main mode.
    now=$(date +"%y%m%d-%H%M")
    name=$(basename $1 .json)
    logpath="log/my_script_name/$name/$now"
    mkdir -p $logpath
    sbatch \
        -o $logpath/%a.out \
        -e $logpath/%a.out \
        $0 -A $1
else
    # Main loop. Just calls my_script.py with the array ID.
    python ./my_script.py $1 ${SLURM_ARRAY_TASK_ID}
fi

拥有一个像这样的脚本可以工作，但看起来非常浪费:我已经将sbatch提交脚本的长度增加了一倍多，仅仅是为了整理我的日志文件.此外，其中大多数是添加的代码，这些代码在其他作业的批处理提交脚本之间将是相似的，例如调用 my_script2.py 等，因此会导致很多代码重复.不禁认为必须有更好的方法.

Having a script like this works, but seems awfully wasteful: I've more than doubled the length of my sbatch submit script just to organize my log files. Moreover, most of that is added code that's going to be similar between batch submit scripts for other jobs, e.g. calling my_script2.py etc, so it makes for a lot of code duplication. Can't help but think there has to be a better way.

推荐答案

您可以在提交脚本中自行重定向Python脚本的输出，然后选择放弃Slurm日志，或将有趣的信息写入Slurm日志关于出处跟踪和可重复性目的的工作.

You can redirect the output of your Python script by yourself in your submission script, and either choose to discard the Slurm log, or write to the Slurm log interesting information about the job for provenance tracking and reproducibility purposes.

您可以使提交脚本如下:

You could have a submission script go like this:

#!/bin/bash
# Set up and run job array for my_script.py, which takes as positional
# arguments a config file (passed via $1) and an array index.

#SBATCH --array=1-100
#SBATCH -n 1
#SBATCH -t 12:00:00
#SBATCH -p short
#SBATCH -J sim_sumstats
#SBATCH --mem=1600

now=$(date +"%y%m%d-%H%M")
name=$(basename $1 .json)
logpath="log/my_script_name/$name/$now"
mkdir -p $logpath
logfile="$logpath/${SLURM_ARRAY_TASK_ID}.out"

echo "Writing to ${logfile}"
scontrol show -dd job $SLURM_JOB_ID
printenv

python ./my_script.py $1 ${SLURM_ARRAY_TASK_ID} > ${logfile}

这样，Python脚本的输出将在您需要的位置，并且在创建日志文件之前将创建父目录.

This way, the output from the Python script will be there where you want it, and the parent directory will be created before the log file is created.

此外，您将拥有Slurm创建的标准输出文件，具有默认的命名方案，其中包含有关作业的信息(来自 scontrol )和来自环境的信息(具有 printenv )代码>).

Additionally, you will have the standard output file created by Slurm, with the default naming scheme, holding information about the job (from scontrol) and from the environment (with printenv).

但是，如果要防止Slurm尝试创建输出文件，请设置-output =/dev/null .

But if you want to prevent Slurm from attempting to create the output file, set --output=/dev/null.

这篇关于在调用slurm sbatch之前为日志文件创建目录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！