Modularization

Snakemake provides several means for modularization of your workflows. These features allow you to:

  • distribute large workflows over multiple smaller files,

  • split workflows into different steps/sub workflows that
    • make things more clear by introducing structure and
    • allow for reuseable sub workflows, and
  • use reuseable wrapper scripts for certain tools instead of copy-and-paste code.

Includes

Another Snakefile with all its rules can be included into the current:

include: "path/to/other/snakefile"

The default target rule (often called the all-rule), won’t be affected by the include. I.e. it will always be the first rule in your Snakefile, no matter how many includes you have above your first rule. From version 3.2 on, includes are relative to the directory of the Snakefile in which they occur. For example, if above Snakefile resides in the directory my/dir, then Snakemake will search for the include at my/dir/path/to/other/snakefile, regardless of the working directory.

Sub-Workflows

In addition to including rules of another workflow, Snakemake allows to depend on the output of other workflows as sub-workflows. A sub-workflow is executed independently before the current workflow is executed. Thereby, Snakemake ensures that all files the current workflow depends on are created or updated if necessary. This allows to build nice hierarchies of separated workflows and even to separate the directories on disk.

subworkflow otherworkflow:
    workdir: "../path/to/otherworkflow"
    snakefile: "../path/to/otherworkflow/Snakefile"

rule a:
    input:  otherworkflow("test.txt")
    output: ...
    shell:  ...

Here, the subworkflow is named “otherworkflow” and it is located in the working directory ../path/to/otherworkflow. The snakefile is in the same directory and called Snakefile. If snakefile is not defined for the subworkflow, it is assumed be located in the workdir location and called Snakefile, hence, above we could have left the snakefile keyword out as well. If workdir is not specified, it is assumed to be the same as the current one. Files that are output from the subworkflow that we depend on are marked with the otherworkflow function (see the input of rule a). This function automatically determines the absolute path to the file (here ../path/to/otherworkflow/test.txt).

When executing, snakemake first tries to create (or update, if necessary) test.txt (and all other possibly mentioned dependencies) by executing the subworkflow. Then the current workflow is executed. This can also happen recursively, since the subworkflow may have its own subworkflows as well.

Wrappers

With Snakemake 3.5.5, the wrapper directive is introduced (experimental). This directive allows to have re-usable wrapper scripts around e.g. command line tools. In contrast to modularization strategies like include or subworkflows, the wrapper directive allows to re-wire the DAG of jobs. For example

rule samtools_sort:
    input:
        "mapped/{sample}.bam"
    output:
        "mapped/{sample}.sorted.bam"
    params:
        "-m 4G"
    threads: 8
    wrapper:
        "0.0.8/bio/samtools_sort"

Refers to the wrapper "0.0.8/bio/samtools_sort" to create the output from the input. Snakemake will automatically download the wrapper from the Snakemake Wrapper Repository. Thereby, 0.0.8 can be replaced with the git version tag you want to use, or a commit id (see here). This ensures reproducibility since changes in the wrapper implementation won’t be propagated automatically to your workflow. Alternatively, e.g., for development, the wrapper directive can also point to full URLs, including URLs to local files with file://. Examples for each wrapper can be found in the READMEs located in the wrapper subdirectories at the Snakemake Wrapper Repository.

The Snakemake Wrapper Repository is meant as a collaborative project and pull requests are very welcome.