在SGE群集问题上的SnakeMake中使用Conda环境 [英] Using Conda enviroment in SnakeMake on SGE cluster problem

问题描述

相关:包含Python脚本，conda和群集的SnakeMake规则

我一直在尝试设置SnakeMake管道以在SGE群集(qsub)上运行.使用直接安装到计算节点的简单命令或工具，就没有问题. 但是，当我尝试设置SnakeMake来通过SGE节点上的Conda下载工具时出现问题.

I have been trying to set up my SnakeMake pipelines to run on SGE clusters (qsub). Using simple commands or tools that are installed directly to computational nodes, there is no problem. However, there is a problem when I try to set up SnakeMake to download tools through Conda on SGE nodes.

我测试的Snakefile是:

My testing Snakefile is:

rule bwa_sge_c_test:
    conda:
        "bwa.yaml"
    shell:
        "bwa > snaketest.txt"

"bwa.yaml"文件为:

"bwa.yaml" file is:

channels:
    - bioconda
dependencies:
    - bwa=0.7.17

我使用以下命令运行SnakeMake:

I run SnakeMake with command:

snakemake -d "/home/<username>" --use-conda --cluster "qsub -cwd -q testing-nod08" --jobs 1

结果，我在SGE计算节点中收到此错误:

As a result, I get this error in SGE computational node:

/usr/bin/python3: No module named snakemake.__main__; 'snakemake' is a package and cannot be directly executed
touch: cannot touch '/home/krampl/.snakemake/tmp.7le8izvw/0.jobfailed': No such file or directory

我尝试将"snakemake = 5.2.2"附加到"bwa.yaml"(如同事建议)，但错误仍然存​​在.

I have tried to append "snakemake=5.2.2" to "bwa.yaml" (as a colleague suggested), but the error remains.

我的问题是:是什么导致此错误，以及如何解决此错误，以便我可以在SGE群集中从SnakeMake运行Conda环境?

My questions are: What causes this error and how to fix this so I can run Conda enviroments from SnakeMake in SGE clusters?

推荐答案

您可能需要将环境变量发送到qsub.

You probably need to send your environment variables to qsub.

snakemake -d "/home/<username>" --use-conda --cluster "qsub -V -cwd -q testing-nod08" --jobs 1

-V将把所有环境变量(包括PATH)发送到qsub作业. 我通常通过SGE发送工作的方法是构建一个脚本，该脚本将封装我的工作:

The -V will send all your environment variables to qsub job including your PATH. What I usually do to send job over SGE is build a script that will encapsulate my jobs:

sge.sh

#$ -cwd
#$ -V
#$ -e ./logs/
#$ -o ./logs/

{exec_job}

您当然可以使用其他选项，例如-q，然后按照以下方式使用snakemake:

You can of course use other options like -q and then use snakemake as follow:

snakemake --cluster "qsub" --jobscript sge.sh ....

(不要忘了在调用snakemake之前创建日志文件夹)

(do not forget to create the logs folder before calling snakemake)

这篇关于在SGE群集问题上的SnakeMake中使用Conda环境的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！