Changelog

8.14.0 (2024-06-11)

Features

  • add flag to mark files where path should not be modified (#2888) (d142b46)

  • support per rule shell exec setting via resources (#2862) (ab8d2dd)

Documentation

  • update FAQ - recommend ensure function for failing on empty output (#2910) (b035071)

8.13.0 (2024-06-05)

Features

  • support for default value specification when using lookup helper function (#2907) (08e88e2)

Documentation

8.12.0 (2024-05-27)

Features

  • Include parameters in extended benchmarks (#2887) (31a9c9b)

Bug Fixes

  • fix corner case bug in input function exception handling (#2895) (fc24292)

  • fix quoting issues when passing complex apptainer args to spawned jobs (#2898) (b07e2e0)

  • properly restrict scheduler if –jobs/-j is used (in contrast to –cores) in local execution (#2897) (6a276bb)

  • typo in workflow specific profile default location (#2878) (74627d3)

8.11.6 (2024-05-17)

Bug Fixes

  • fix opening of multiple checkout output files in the same input function when using remote storage (2f8e719)

8.11.5 (2024-05-16)

Bug Fixes

  • avoid premature deletion of local copies of remote storage input files used by multiple jobs (#2874) (21ec649)

  • fix opening of checkpoint output files from remote storage (#2873) (e7cb7fb)

Documentation

  • add link to code of conduct (889a3bc)

8.11.4 (2024-05-11)

Bug Fixes

  • fix missing await when opening checkpoint output (#2868) (25a361b)

  • make checkpoint updates synchronous (#2871) (b0e7ebd)

Documentation

  • update code of conduct email address (3047683)

8.11.3 (2024-05-03)

Bug Fixes

  • ignore errors when cleaning up runtime cache (#2859) (6df7046)

  • show queries of remote storage files instead of local paths in summary (#2860) (ba1db8e)

8.11.2 (2024-05-02)

Bug Fixes

  • bug when requesting extended benchmark with slurm (#2855) (0e039ff)

8.11.1 (2024-04-29)

Bug Fixes

  • check template rendering output for leaked input file paths (#2850) (433302e)

  • do not distinguish between local and remote rules in dryrun (74b99ec)

  • omit norun jobs when determining remote storage input file retrieval (#2854) (37a7c7f)

  • Prevent binary log files to crash snakemake execution with show-failed-logs (#2827) (8a80bda)

  • replace pkg_resources for python 3.12 (#2831) (ac144fc)

  • return set instead of list when just –quiet (#2829) (eeb57e2)

  • small typo in error (#2853) (325a715)

  • use keyword arguments in _IOFile.open (#2847) (50c84dc)

Documentation

8.11.0 (2024-04-25)

Features

  • allow for more extensive benchmark file in jsonl format (#2691) (de12463)

Bug Fixes

  • only download input for local jobs in the main process, not within remote groups (#2842) (97f428b)

  • remove non-empty local copies of remote storage dirs (#2845) (71b2b87)

  • retrieve files from storage if necessary when calling their open method (e.g. when accessing output files from a checkpoint) (#2839) (5448208)

8.10.8 (2024-04-19)

Bug Fixes

  • extend workflow test suite by wildcards with slashes (in order to detect bugs that can occur in executors) (#2810) (fc9971b)

  • invalid extensions with multixt and cache (#1808) (#2823) (a845b1c)

  • module prefix added twice on expand (#2814) (27416f4)

  • retrieve inputs of local rules from storage even if they are intermediate results (#2811) (d158aa8)

Documentation

  • add snakemake and reportplugins to req.txt to show all cli options (#2817) (aaa9065)

8.10.7 (2024-04-12)

Bug Fixes

Documentation

  • add Morten Lund as another maintainer. (293bc05)

8.10.6 (2024-04-04)

Bug Fixes

  • only constrain by –max-threads if threads of job are already known (#2790) (5f28fcd)

8.10.5 (2024-04-04)

Bug Fixes

  • properly delete local copies of storage files after remote jobs (#2793) (e3362b0)

  • respect APPTAINER_CACHEDIR and allow env variables in –apptainer-prefix and –conda-prefix (#2795) (b1694cd)

8.10.4 (2024-03-27)

Documentation

8.10.3 (2024-03-27)

Documentation

8.10.2 (2024-03-26)

Bug Fixes

  • remove default packages from conda envs (#2749) (027906c)

  • use base64 encoding for passing default resources args to jobs (#2780) (4735bc3)

8.10.1 (2024-03-26)

Bug Fixes

  • passing of –set-threads values to remote jobs (#2775) (4fd767a)

  • use base64 encoding when passing resources and threads to remote jobs (this solves issues with complex quoted resources) (#2778) (a8ee4d8)

8.10.0 (2024-03-22)

Features

  • expose ResourceSettings in TestWorkflowsBase (#2770) (e7c323b)

8.9.0 (2024-03-18)

Features

  • add function ‘exists’ for checking the prior existence of files or dirs before workflow execution while considering any remote storage settings. In addition: some bug fixes for error handling and the update/before_update functionality. (ee96393)

8.8.0 (2024-03-15)

Features

Bug Fixes

  • various error handling improvements, fixed logging/error behavior (stdout from dryrun, stderr otherwise) (#2759) (d0d1f48)

8.7.0 (2024-03-13)

Features

  • add flag for marking output as being updated instead of rewritten (update(“test.txt”)) (#2754) (9ba5d95)

  • allow default storage provider to be explicitly set to none (#2746) (ce519d7)

8.6.0 (2024-03-11)

Features

  • add setting for defining separate local storage prefix for remote jobs; improved ergonomics for semantic helper functions (#2743) (5007e5c)

  • allow passing of lists of functions or single functions to expand (#2741) (32e65df)

Bug Fixes

  • fix error message for invalid storage provider queries (977951e)

  • fix premature deletion of temp files in combination with checkpoints (#2737) (b22ba5f)

8.5.5 (2024-03-07)

Bug Fixes

  • less frightering message when telling about missing output files as reason for running a job (7b6c9d4)

8.5.4 (2024-03-06)

Bug Fixes

  • fix bugs in –summary and –list-input-changes. Removed outdated statement in tutorial. (#2735) (55c06d8)

Documentation

  • fix explanation on how to use all cores in tutorial (#2733) (6420428)

  • fix syntax highlighting in tutorial (2887604)

8.5.3 (2024-02-26)

Bug Fixes

  • error when detecting mime type during report creation (#2721) (42dad42)

8.5.2 (2024-02-24)

Bug Fixes

  • when using remote storage: only wait for files if job did not error (8c7ee91)

Documentation

8.5.1 (2024-02-24)

Documentation

8.5.0 (2024-02-24)

Features

  • add ability to return input functions from input functions. Such nesting is evaluated 10 times at most. Beyond that, an error is thrown. (#2717) (7a47924)

  • support for report plugins (#2700) (2f7d4b5)

Bug Fixes

  • fix wait for files in case of using remote storage and remote execution (#2718) (eec3a5f)

  • proper interpretation of standard resources given as strings (e.g. runtime as ‘5m’). Avoid the need to set additional quotes around size or timespan resources. Improved error messages for resource handling. (#2716) (b6636e9)

8.4.12 (2024-02-20)

Bug Fixes

  • constrain dependencies to match conda experience (#2710) (d9a7a13)

  • various bug fixes for resource parsing (#2711) (d1daf0b)

8.4.11 (2024-02-19)

Documentation

8.4.10 (2024-02-19)

Bug Fixes

  • properly handle –touch when using a storage provider (#2705) (fca138d)

8.4.9 (2024-02-15)

Bug Fixes

Documentation

  • added Mastodon follow label with just ‘Follow’ similar to X (#2692) (7e36496)

8.4.8 (2024-02-09)

Bug Fixes

  • fix bug causing FileNotFoundError when accessing checkpoint output. (c81954d)

  • Fix collect-lookup attribute error (#2687) (e39c74c)

  • Fixed plot axis label on report (#2683) (a4c2a03)

8.4.7 (2024-02-07)

Documentation

  • improve branch function docs (e9d1a11)

8.4.6 (2024-02-06)

Bug Fixes

  • fix missing storage information when handling already completed checkpoints. This solves a bug causing failure to retrieve storage files in workflows with checkpoints. (5791c60)

8.4.5 (2024-02-06)

Bug Fixes

  • for local execution, always unrestrictedly assume shared FS (#2679) (0bee50b)

  • support list of queries for storage provider (#2674) (d53ef92)

  • use default container image if nothing is provided (#2677) (109c991)

8.4.4 (2024-02-05)

Bug Fixes

  • fixed bug in handling of resource overrides for remote job submission (5c06dd6)

  • output of rulegraph, closes #2656 (#2671) (f9b9110)

  • SyntaxWarning due to invalid escape sequences in non-raw regex pattern string (#2670) (3748d9d)

8.4.3 (2024-02-02)

Bug Fixes

  • Do not scheduler execution message if no jobs are ready (b1c4f47)

  • fix string resource definition in CLI and profile (#2627) (bbd76ae)

  • if report files are within storage, retrieve them from storage before loading into report (60041bd)

8.4.2 (2024-01-30)

Bug Fixes

  • allow lookup dpath or query to be a callable (33f1637)

  • fix error when passing callable as dpath or query of lookup function (0e5b878)

8.4.1 (2024-01-30)

Fixes

  • fixed resource handling in profiles

8.4.0 (2024-01-29)

Features

  • add cols argument to lookup function; fix various minor bugs on cluster systems (#2651) (ca7a602)

Bug Fixes

8.3.2 (2024-01-25)

Bug Fixes

  • do not require cores to be set for non-executing modes (#2646) (30cf026)

8.3.1 (2024-01-23)

Documentation

8.3.0 (2024-01-23)

Features

  • implement semantic helper functions for input and param function handling (#2344) (b4b5e51)

  • support for continuously updated input (using Python queues) (#2594) (db1c0ed)

8.2.4 (2024-01-23)

Bug Fixes

  • fix exception when handling syntax error during parsing (d5a7a56)

8.2.3 (2024-01-19)

Documentation

  • handle overflow of content div (b23e277)

8.2.2 (2024-01-19)

Documentation

8.2.1 (2024-01-17)

Bug Fixes

  • do not require cores to be set for rule-level methods of the workflow API or the corresponding CLI commands (e.g. –lint). (#2629) (2040468)

  • fix false complaints about rules with multiple output files (#2628) (b1b4f5b)

  • migration guide typo and wrong link (#2625) (645f3d1)

8.2.0 (2024-01-16)

Features

  • add method to obtain group args for spawned jobs (bd1b450)

Bug Fixes

  • properly resolve wildcards in group components (#2620) (c788a46)

  • return set of rules when obtaining allowed rules for remote job (2c44cf6)

8.1.3 (2024-01-15)

Bug Fixes

  • bug with preemptible rules (#2616) (c6d7141)

  • do not pass snakefile as metadata when wms monitor flag is used (#2573) (13b3205)

  • use default group settings if not execution workflow (fixes attribute error occurring with –report) (#2617) (21e9964)

8.1.2 (2024-01-12)

Bug Fixes

  • local mtime handling in case of storage plugins and cleaner error message for parallel storage retrieval (#2611) (880b264)

  • Migrate away from deprecated pulp API (#2610) (fb26640)

8.1.1 (2024-01-11)

Bug Fixes

  • deduplicate input files before retrieval from storage (#2600) (37cf475)

8.1.0 (2024-01-08)

Features

Bug Fixes

  • add mamba to docker image (eb0c884)

  • correctly report lineno (#2584) (967a0d7)

  • move apptainer into separate env in docker image (94e9e2c)

  • single line f-string format error in py3.12 (#2588) (87c06c0)

Documentation

8.0.1 (2023-12-21)

Bug Fixes

  • remove bash completion entrypoint (no longer supported, was too slow to be usable anyway) (922b53a)

Documentation

  • fix cli options rendering (264c1a9)

  • fixes in migration guide (f8adefa)

8.0.0 (2023-12-20)

⚠ BREAKING CHANGES

Snakemake 8 marks the beginning of decomposing Snakemake into a framework of plugins. This enables the democratization of method development within the Snakemake ecosystem. We start with plugins for storage and execution backends. In the future, there will be plugins for the scheduling, metadata, software deployment, reporting, and many more. This way, it will be possible to easily launch and explore new developments in workflow management and reproducible data analysis without the need to get your work merged into the main codebase of Snakemake and also without the need to develop a new workflow management system as a proof of concept.

In detail, Snakemake 8 introduces the following changes. Unfortunately it was unavoidable to break some usages (we apologize). Nevertheless, we tried to ensure that every removed or modified feature has been replaced with an equivalent reimplementation, as outlined in our migration docs. While Snakemake 8 has an even more thorough testing framework than any release before, and while it has been quite heavily tested in practice by us, you might initially experience bugs and glitches for which we want to apologize beforehand. We think that the massive codebase improvements are worth it in the long run, and hope that everything goes well. As always, any pull requests with test cases and pointers to bugs are more than welcome.

Detailed breaking changes

  • removed the long time ago deprecated support for dynamic, version, and subworkflow (see the migration docs)

  • migrated old remote providers into storage plugins (see the migration docs)

  • migrated execution backends into plugins, including a change in the respective command line interfaces (see the migration docs)

  • deprecates --use-conda and --use-singularity in favor of --software-deployment-method conda or --software-deployment-method apptainer and --software-deployment-method conda apptainer (see the migration docs)

  • profile support is now versioned, such that different profiles can be written for different minimum Snakemake versions (see the migration docs)

  • redesigned Snakemake API. It now uses a modern, dataclass based approach (see the migration docs)

Features

  • add ability to inject conda environments into running Snakefile (#2479) (6140e29)

  • add functionality for deploying sources if no shared FS is assumed (#2486) (76eac3c)

  • add option to control software deployment mode (shared or non shared FS) (#2525) (04ec2c0)

  • allow detailed configuration of shared FS usage (#2528) (0d34be9)

  • allow environment variables in string values of profile (e.g. paths may now contain elements like $USER). (58dc70c)

  • allow python expressions in –set-resources (#2521) (022a31e)

  • allow to set latency_wait in executor test suite (c0bca0b)

  • automatically upload workflow sources to default storage provider if no shared FS is used (a450c49)

  • Faster ci test setup (#2489) (4798e8a)

  • implement precommand (#2482) (ff0f979)

  • redesigned Snakemake API. It now uses a modern, dataclass based approach (#2403) (2be3bfa)

  • support for external executor plugins (#2305) (c9eaa4e)

  • version specific profile config files (profile/config.v8+.yaml with profile/config.yaml as fallback that matches any version) (#2498) (47e5811)

Bug Fixes

  • adapt to changes in snakemake-interface-executor-plugins (635c68a)

  • add storage provider args to deploy sources command (67178e3)

  • add testcase for script directive to work with Python 3.7 and corresponding fix. (0b4ae2e)

  • allow pepfile and pepschema to take pathlib (#2546) (ca91661)

  • also inherit rule proxies if there is no rulename modifier specified in a use rule statement (#2440) (1570289)

  • assume at most 8GB memory for default resources. This way, we avoid exploding memory requirements for large input files that are very unlikely to be put entirely into memory by any tool. (11c2ecc)

  • comparison to float in scheduler (ef44d84)

  • detect job paths that leave and then enter a group. Such paths are invalid because then the group depends on itself. (#2527) (5383a4d)

  • ensure that auto deployment of default storage provider works in containers with read-only root home. (1a347ff)

  • ensure that log and benchmark files are uploaded to storage as well (#2545) (6aabb5d)

  • ensure that targetjob is always forced. This fixes a bug causing run-directive rules to not being executed even when enforced via e.g. -R. (#2448) (b2a60d5)

  • fix cache handling and unlock handling (2f4d5e1)

  • fix nargs definition for –deploy-sources (fc252c8)

  • fix path handling when detective profiles (fe63881)

  • fix storage handling on windows by converting all paths to posix paths (#2519) (7864a76)

  • handle different f-string tokens in py3.12 (#2485) (f2c7613)

  • handle storage for local jobs; add test case (6d978ef)

  • handling of group jobs when obtaining temp input files (71be1de)

  • import (#2402) (2c831f1)

  • improved error handling for storage upload; fixed bugs caused by outdated calls to IOFile.exists(). (720bb84)

  • improved error messages in case of invalid storage queries (9671fd0)

  • in addition to localrules statement, infer that job is local if it has any input or output file that is marked as local (#2541) (e8b682b)

  • only deactivate conda inject envs upon workflow tear down (#2503) (e6dfdd4)

  • Panoptes –wms-monitor-arg (#2444) (98d2bdf)

  • proper reuse of rule proxies when importing several times from the same module (#2404) (e867dda)

  • Restore backward compatibility for Google Life Sciences executor (#2461) (5e3a464)

  • shadow “full” mode ignore symlinks (#2516) (1d58120)

  • show failed logs in executor testcases (92f7bf4)

  • Slack log service (#2537) (26eb4ba)

  • sort report (sub-)categories in lexicographical order (#2449) (d0705ad)

  • update minimum snakemake-interface-storage-plugins version (0ef7226)

  • use temporary directory (faster, more likely local, always writable) for persistence and source cache in case of remote execution without shared fs (#2502) (c8fa7ba)

  • wait for logs before showing them on error (a4ff328)

Documentation

  • document name directive with example (#2534) (cce5551)

  • fix syntax in cluster example (#2460) (64e9645)

  • notes on arm based machines in tutorial docs (0586f04)

  • rust: Fix typo on rust-script version (#2488) (a79dd94)

7.32.4 (2023-08-18)

Bug Fixes

  • always sort report (sub-)categories in lexicographical order

  • also inherit rule proxies if there is no rulename modifier specified in a use rule statement

  • ensure that targetjob is always forced. This fixes a bug causing run-directive rules to not being executed even when enforced via e.g. -R.

7.32.3 (2023-08-07)

Bug Fixes

  • fix bug occurring when using multiple use rule statements in combination with the rules object for referring to output of already defined rules.

7.32.2 (2023-08-07)

Bug Fixes

  • unnecessary set Snakefile in AzBatch executor (#2397) (78e6d6e)

7.32.1 (2023-08-05)

Bug Fixes

  • add missing spaces between lines that get concatenated. (#2268) (7238458)

  • better message about profile usage upon execution (#2391) (cf8aea5)

  • do not overwrite default resources setting in azure batch executor (#2395) (4aef3b9)

  • updating of non-dict config values gives error (#2364) (b33aeec)

  • wrong rule names when nesting module imports (#1817) (65c79a4)

Documentation

  • basics.rst: suggest VS Code instead of deprecated Atom as IDE (#2368) (1357316)

7.32.0 (2023-08-03)

Features

  • add support for Kubernetes service account name spec (#2254) (3370426)

Bug Fixes

  • Enable values with an = sign in default_resources (#2340) (c1c9229)

  • Escape workdir paths for potential spaces in paths (#2196) (9261f7e)

  • ga4gh executor resources (#2042) (ad6eaef)

  • print exceptions when job is not a shell job (#2385) (8a37b85)

  • remote-azblob-sasToken-Authorization (#1800) (bc854a7)

  • wms-monitor now gets data in correct json format (#2347) (7fafa7a)

Documentation

7.31.1 (2023-08-02)

Bug Fixes

  • require python >=3.7 again (the python 3.9 dependency was unnecessary) (#2372) (0d0e9c4)

Documentation

  • update CHANGELOG.md: add minimum Python version bump (#2370) (48e934d)

7.31.0 (2023-07-26)

Features

  • Add support for Google Service Accounts and GCE VM network configuration (#2318) (2b754aa)

7.30.2 (2023-07-20)

Breaking changes

  • Bump minimum Python version from 3.7 to 3.9 (#2369) (4608163)

Bug Fixes

  • do not allow setting benchmark and between-workflow caching for the same rule. The reason is that when the result is taken from cache, there is no way to fill the benchmark file with any reasonable values. (#2335) (e2d64fa)

  • ensure lazy evaluation of resource functions/callables (this also entails, for now, a removal of the thread statistics in the yellow job stats table); further, added some clarifying sentences about resource function evaluation to the docs (#2356) (4c591b7)

  • handle non-PEP440 versions of apptainer/singulariy (#2337) (dea6ba8)

  • remote GS builds too many inventories; io:collect_mtime always uses uncached mtime (#2266) (bad9115)

  • Solve apptainer version issue (#2333) (a876e0f)

  • SyntaxWarnings due to non-raw regex pattern strings (#2359) (a08c0b0)

Documentation

  • clarify minimum Snakemake version for profiles (86dc277)

  • clarify the channel priority in environment definition deployment.rst (#2352) (76aa964)

  • fix typo (stackoverflow issue) (#2365) (f770984)

  • note on using checkpoint mechanism only for input function, not for params or resources. (#2353) (4be2f9d)

7.30.1 (2023-06-28)

Bug Fixes

7.30.0 (2023-06-28)

Features

  • allow profiles to be YTE templates; adapt to eido 2.0 (#2325) (67d9ff2)

7.29.0 (2023-06-21)

Features

  • introduce –workflow-profile for additional workflow specific profiles that overwrite global profiles; add ability to define key-value CLI flags like –set-threads or –set-resources as multi-level dictionaries in profile config yaml files (#2310) (9675c17)

Bug Fixes

  • addressing #2197 by allowing 256 character account names in slurm (#2198) (ab58c65)

  • removed distutils from snakemake (#2312) (9b8c362)

  • Update init.py to move “file” param to “print” (#2291) (92352b6)

7.28.3 (2023-06-16)

Bug Fixes

  • Detect pandas availability to select serializer (#2300) (e08a771)

Performance Improvements

  • avoid superfluous mtime checks when the same file is referred to by multiple jobs (#2284) (eb6e2e1)

Documentation

  • update docs for azbatch and dockerhub ref (#2298) (908dbf1)

7.28.2 (2023-06-13)

Bug Fixes

  • fix pandas import handling in metadata persistence (27f7b40)

7.28.1 (2023-06-11)

Bug Fixes

  • Bump yte from >=1.0,<2.0 to >=1.5.1,<2.0 (#2275) (8c0b34f)

  • remove superfluous dependency (aad61a0)

7.28.0 (2023-06-11)

Features

7.27.0 (2023-06-07)

Features

  • Allow the environment variable SNAKEMAKE_CONDA_PREFIX to be present without –use-conda (#2263) (e4eba8d)

Bug Fixes

  • adapt linting rule to Python 3.11 (a3a5c58)

7.26.0 (2023-05-22)

Features

  • allow config files to be processed with YTE (#2269) (8e1c22f)

7.25.4 (2023-05-12)

Bug Fixes

  • fix scrolling behavior in landing page of report for large workflows (63c0c31)

  • report spacing (f3954b3)

Documentation

7.25.3 (2023-05-03)

Bug Fixes

  • fix missed wildcard constraints when using local rule inheritance (#2242) (8e94785)

7.25.2 (2023-04-28)

Bug Fixes

  • Fix inconsistencies between detailed summary and normal summary (#2218) (d903123)

  • Fix race condition when creating lock directory (#2225) (66ea4d1)

  • quote paths given to singularity in order to ensure that it does not fail when paths contain whitespace (#2190) (a572fb7)

Documentation

  • added changelog info for >v7.19.1 parsing error of “hh:mm:ss” time format in runtime resource (#2189) (2889f38)

  • update misc/vim/Readme with info for packer.nvim (#2095) (32166a7)

  • Update workflow syntax with priority directive (#2188) (af10db5)

7.25.1 (2023-04-28)

Bug Fixes

  • allow log directive in default target rule (#2191) (86e9624)

  • only consider global wildcard_constraints from the same module (#2235) (c412b71)

  • Use job.rule.name attribute to fill rule field in summary (#2217) (837c3fd)

Documentation

7.25.0 (2023-03-23)

Features

Bug Fixes

  • always make sure that the original path of source cached files is properly passed into metadata persistence records (#2179) (8bacbd0)

  • slurm batch job status queries (#2167) (0bb69e4)

Documentation

  • Change snakemake-tutorial download link to always be the latest (#2183) (ae8a8f4)

  • fix typos in –help (#2182) (09f0cbe)

  • Improve error message when rule contains multiple run/shell/script/notebook/wrapper/template_engine/cwl keywords (#2186) (cd5a3c4)

7.24.2 (2023-03-14)

Bug Fixes

  • fix index out of bounds error raised by usage of workflow.source_path called from input or params functions (thanks @AKBrueggemann) (#2170) (cf8e6e8)

  • limit length of failed logs decorations (#2125) (6fc9243)

  • raise error if callable is passed to expand. (#2171) (1f28476)

  • rounding for batch calculation (#2064) (cbdbf9b)

7.24.1 (2023-03-09)

Bug Fixes

  • better job status queries for slurm executor (#2136) (a4df38c)

  • get python version for script environment in a backwards compatible way that works down to python 2.7 (#2161) (44e59b9)

  • prevents DeprecationWarning caused by using old draft of json schema (#2152) (9791ffb)

Performance Improvements

  • Gfal2 remote provider using gfal2-python instead of gfal2-utils. (#2128) (0b9bfe5)

Documentation

7.24.0 (2023-03-01)

Features

  • limit the number of input/output files in job properties (#2149) (d93f091)

Bug Fixes

  • #2130 by patching the protect() method so the path of files in subdirectories is properly resolved during write-protection (#2131) (1a754fd)

  • sre_constants import because of deprecation (#2139) (3b326db)

  • ensure user and group rw permissions for metadata files and source cache (#2132) (cc51faa)

  • is_run error with local, group jobs (#2133) (31bfcd5)

  • require toposort >= 1.10 (#2145) (3cb54b8)

Documentation

7.23.1 (2023-02-18)

Bug Fixes

7.23.0 (2023-02-18)

Features

  • changed report layout to display menu always left of the results. For fullscreen, one can still hide the menu, which leads to automatic growth of the results (#2116) (d771b1b)

  • Publish docker images for amd64 & arm64 (#2105) (4c898f5)

Bug Fixes

  • use text/markdown for long_description_content_type (#2112) (0241075)

Performance Improvements

  • Improve execution speed of cleanup_workdir (in dag) (#2103) (1fbc5f5)

7.22.0 (2023-02-12)

Features

Bug Fixes

  • assume shared filesystem by default when running with –flux (#2075) (4bec2fd)

  • properly handle NA values for paramspaces (#2098) (6b6a880)

7.21.0 (2023-01-30)

Features

  • ability to encode paramspaces into a single wildcard, via the newly introduced single_wildcard argument of Paramspace. (#2069) (728ab3c)

  • allow input, output, and params to be used in functions passed to report mark arguments (#2081) (93ff8b6)

Bug Fixes

  • more robust encoding of params in persistent metadata storage. This way, pandas parameters do not lead to spurious rerun triggers. (#2080) (106a4c3)

  • more robust parsing of sacct output in slurm executor (#2036) (fe651f8)

  • Postprocess job groups in toposorted order for correct touch times (#2073) (10b5849)

7.20.0 (2023-01-18)

Features

  • add tes token (#1966) (59a8fa0)

  • Add token auth to GitLab/GitHub hosting providers (#1761) (e03a3b4), closes #1301

  • allow for human friendly resource definitions (e.g. mem=”5GB”, runtime=”1d”) which deprecates slurm constrained time format (e.g. runtime=”hh:mm:ss”) (#1861) (24610ac) (#2154)

Bug Fixes

  • :bug: - fix hyperlink (#2046) (9519d31)

  • Catch missing error stream in Slurm executor (#2063) (c21fc7e)

  • correctly parse empty values in config cli (#2032) (1b0689d)

  • Correctly parse UserDicts in executors (#2016) (e3926fa)

  • Fix handling of –jobs in no-exec state (#2029) (e8e8222)

  • make --show-failed-logs handle empty log files (#2039) (683c6f2), closes #2023

  • make python version check more robust (#2058) (e685621)

  • parsing error when last line is comment (#2054) (a928dd4)

  • prevent overriding of retries when set to 0 (#2053) (a328f3e)

  • propagate attempt count from group to subjobs (#2052) (da3f1c0)

  • remove overflow from rulegraph div in report (9a0aaa7)

  • skip type checks of missing dir in touch mode (#2051) (ae00c25)

  • slurm default_resources quoting (#2043) (47d3fc3)

  • Update list of python versions in classifiers (#2020) (7a98100)

  • use short argument name for --chdir for compatibility with Slurm <=v17 (#2040) (a9ed3ec)

  • human friendly resource definitions introduce inability to parse slurm specific time format (e.g. “hh:mm:ss”). New time format (e.g. “1d”) adds portability among various job schedulers and clusters (#2154)

Documentation

7.19.1 (2022-12-13)

Bug Fixes

  • improved default resources parsing (also allowing to deactivate a default resource via setting it to None) (#2006) (e6cdb32)

Documentation

7.19.0 (2022-12-13)

Features

  • add keyword to gridftp remote provide to specify the number or disable usage of multiple data stream (#1974) (3e6675d)

  • provide information about temp, pipe, and service files in –summary (#1977) (c7c7776)

  • native SLURM support (–slurm, see docs) (#1015) (c7ea059)

Bug Fixes

  • avoid logfile writing in case of dryrun; better hints in case of incomplete checkpoints (#1994) (a022705)

  • handle case where zenodo deposition does not return files (#2004) (b63c4a7)

  • issue #1882 WorkflowError: Metadata can’t be created as it already exists (Windows) (#1971) (d4484e6)

  • json validation error with markdown cells (#1986) (6c26f75)

7.18.2 (2022-11-10)

Bug Fixes

  • Change ratelimiter dependency to throttler (#1958) (50b8f16)

  • fixed problem with leaked modifications when inheriting multiple times from the same rule (#1957) (2475cbc)

  • forwarding –keep-incomplete to cluster executor (#1951) (2894c7d)

  • show input files on job error (#1949) (ad21631)

7.18.1 (2022-11-03)

Bug Fixes

  • regression ValueError introduced with 7.17.2 (#1947) (53a4fca)

7.18.0 (2022-10-31)

Features

  • first try to match output files against input files while persisting wildcard values from the consuming job. This can dramatically reduce ambiuity problems. Thanks to @descostesn! (#1939) (d093907)

7.17.2 (2022-10-28)

Bug Fixes

  • Consider source cache when setting search path for python scripts. This allows to import from Python modules next to scripts while deploying the workflow as a snakemake module, even from remote locations. (#1940) (27be1d4)

7.17.1 (2022-10-28)

Bug Fixes

  • change source cache entries to keep the original name and folder structure, such that imports from e.g. scripts also work with remote modules (if specified as additional input files with workflow.source_path) (#1936) (c34f3f6)

7.17.0 (2022-10-27)

Features

  • allow to define the cache mode per rule (this enables to exclude software envs from the caching hash value, which can be handy e.g. for download rules where the software version does not affect the result) (#1933) (715e618)

Performance Improvements

  • cached os.pathconf() call in _record_path() (#1920) (551badb)

7.16.2 (2022-10-26)

Bug Fixes

  • fix false rerun triggering downstream of checkpoints due to spurious parameter, code or software env changes (638ea86)

  • remove redundant dot in expand call in multiext documentation (#1921) (278beaa)

7.16.1 (2022-10-18)

Bug Fixes

  • conda create –no-shortcuts absent on Linux/MacOS (regression from #1046) (#1916) (8a86a1e)

  • fix typo in line display of exceptions (#1912) (55e38a6)

7.16.0 (2022-10-14)

Features

Bug Fixes

  • allow report generation to handle pathlib objects (#1904) (7c34656)

  • fix false reruns after checkpoints (#1907) (dc5af12)

7.15.2 (2022-10-08)

Bug Fixes

  • Comparison of rules and non-rule instances (#1894) (bf01ece)

  • delay evaluation of tmpdir to actual job execution, and not submission. This way, tmpdir can be dependent on the node context. (#1860) (4203556)

  • ensure that rule name string instead of object is passed to tabulate package (#1898) (f9ff157)

  • issue 1846 (#1888) (da2dfbd)

  • lexicographically sorted rule display with –list, and trimmed rule docstrings (#1880) (32128ae)

Performance Improvements

  • Average NamedList getitem performance improvement (#1825) (10451b7)

7.15.1 (2022-10-04)

Bug Fixes

Documentation

7.15.0 (2022-10-04)

Features

Bug Fixes

Documentation

7.14.2 (2022-09-26)

Bug Fixes

  • reduce resource requirements for kubernetes tests (#1876) (cb4b78a)

7.14.1 (2022-09-23)

Bug Fixes

  • allocation of local ssds in k8s tests (#1870) (d0de4dc)

  • allow script directive to take pathlib Path (#1869) (12cdc96)

  • catch errors in remote.AUTO provider list (#1834) (c613ed2)

  • consistently use text output in conda shell commands and various little fixes for failing test cases due to conda package changes (#1864) (4234fe7)

  • declare associative arrays (#1844) (90ae449)

  • fix falsely triggered reruns if input files are obtained via workflow.source_path() (#1862) (2dc2e6a)

  • fixed typos (#1847) (a1e49b6)

  • k8s container volume mounts as list (#1868) (5c54df3)

  • None type error when invoking Workflow object manually (#1731) (dc45ccb)

  • request disk_mb resource from k8s (#1858) (f68f166)

  • respect shebang lines in post-deploy scripts (see deployment docs) (#1841) (c26c4b6)

7.14.0 (2022-08-27)

Features

  • add support for bash scripts in the script directive (beyond small shell commands) (#1821) (c4cf8fd)

Documentation

7.13.0 (2022-08-25)

Features

  • add gitfile option to make it possible to use local git repos when importing modules (#1376) (1a3b91f)

Bug Fixes

  • allow to use {wildcards} for group jobs in cluster config (#1555) (f0ec73d)

  • avoid “Admin” prompt when using conda on windows (#1046) (552fadf)

  • handle benmark bug that arise with singularity (#1671) (10ef7c4)

  • Open Snakefile for reading with explicit encoding specified (#1146) (ec1d859)

  • remove superfluous comma causing TypeError in conda-frontend error message (#1804) (87b013c)

Documentation

  • explain SNAKEMAKE_PROFILE environment variable (2b32bba)

  • update contribution docs (09a5595)

7.12.1 (2022-08-09)

Bug Fixes

  • Fix case of multiple scattergather processes (#1799) (417aad4)

  • more comprehensive error reporting for RuleExceptions (#1802) (1cd9512)

7.12.0 (2022-07-29)

Features

Bug Fixes

  • Fix technical bugs in resource-scope documentation (#1784) (878420c)

  • move max_status_checks_per_second attribute setting before the wait thread of cluster backends is started to avoid missing attribute errors (#1775) (a48e9d0)

7.11.0 (2022-07-27)

Features

  • improved resource handling in groups and ability to define resource scopes (global or per node), see docs and –help (#1218) (a8014d0)

Bug Fixes

  • fixed conda frontend detection and checking to also work with latest mambaforge (#1781) (225e68c)

7.10.0 (2022-07-26)

Features

  • Support conda environment definitions to be passed as function pointers, similar to input, params, and resources (#1300) (6f582f1)

Bug Fixes

  • fix regression in workflow source acquisition of google life science executor (#1773) (c07732e)

  • limit filename length of temporary files generated by the persistence backend (metadata, incomplete markers, etc.) (#1780) (59053e7)

7.9.0 (2022-07-19)

Features

  • make it possible to exclude rules that will be imported when using ‘use rule’ statement (#1717) (d9e0611)

Bug Fixes

  • add lock free mechanism for avoiding race conditions when writing persistence information; consider corrupt metadata records as non-existent (#1745) (71fe952)

  • conda python interpreter path on Windows (#1711) (155c9d6)

  • ensures that REncoder also checks for numpy.bool_ in encode_value (#1749) (10a6e1d)

  • Move quiet default after profile parsing (#1764) (6ade76d)

7.8.5 (2022-06-30)

Documentation

  • fix long description type for pypi (set to markdown) (d8d9b8f)

7.8.4 (2022-06-30)

Bug Fixes

  • only display a warning in case of non-strict channel priorities (#1752) (b84fa33)

  • pass triggers and resources to subworkflow (#1733) (fa7fb75)

  • add pyproject.toml to use setuptools features (#1725) (454bfd1)

Documentation

7.8.3 (2022-06-20)

Bug Fixes

  • allow apptainer as a successor to singularity. (#1706) (bcbdb0b)

  • improved provenance trigger info (#1720) (29d959d)

  • small changes to make docs checkpoint example functional (#1714) (1d4909e)

7.8.2 (2022-06-08)

Bug Fixes

  • fixed bug in needrun computation of jobs downstream of checkpoints (#1704) (c634b78)

7.8.1 (2022-05-31)

Bug Fixes

  • handling of remaining jobs when using –keep-going (#1693) (87e4303)

  • more robust calculation of number of jobs until ready for execution (#1691) (fdfc717)

  • propagate rerun trigger info to cluster jobs; fix a bug leading to software stack trigger generating false positives in case of conda environments; fixed display of info message in case of provenance triggered reruns (#1686) (503c70c)

  • set channel priority in container system wide (#1690) (41175b3)

7.8.0 (2022-05-24)

Features

  • automatically rerun jobs if parameters, code, input file set, or software stack changed (thanks to @cclienty and @timtroendle). This also increases performance of DAG building by handling job “needrun” updates level wise, while avoiding to perform a full check for those jobs that are already downstream of a job that has been determined to require a rerun. (#1663) (4c11893)

  • enable the definition of conda pin files in order to freeze an environment. This can drastically increase the robustness because it allows to freeze an environment at a working state. (#1667) (53972bf)

Bug Fixes

  • fail with error if conda installation is not set to strict channel priorities (#1672) (f1ffbf2)

  • fix errors occurring when referring to input func via rules..input (#1669) (28a4795)

  • parsing error when combining single line directive with multi-line directive in use rule statements (#1662) (26e57d6)

7.7.0 (2022-05-16)

Features

  • add flag ensure that allows to annotate that certain output files should be non-empty or agree with a given checksum (#1651) (76f69d9)

  • for small files, compare checksums to determine if job needs to run if input file is newer than output file (#1568) (1ae85c6)

  • LockException (#1276) (f5e6fa6)

  • new directive “retries” for annotating the number of times a job shall be restarted after a failure (#1649) (c8d81d0)

Bug Fixes

Documentation

  • singularity sometimes uses system /tmp explanation (#1588) (170c1d9)

7.6.2 (2022-05-06)

Bug Fixes

  • fixed permission issues when using zenodo remote provider to access restricted depositions (#1634) (510f534)

7.6.1 (2022-05-04)

Bug Fixes

  • check for skipped rules in case of local rule inheritance (#1631) (9083ac1)

7.6.0 (2022-05-03)

Features

  • enable restricted access support in zenodo remote provider (#1623) (692caf9)

Bug Fixes

  • avoid erroneous too early deletion of parent directories in case of failed jobs (thanks to @SichongP). (#1601) (b0917e6)

  • ensure that rule inheritance considers the same globals and other settings as parent module (#1621) (104cab9)

  • issue 1615 - Switch formatting condition for dictionary (#1617) (0771062)

  • multiext prefix computation in case it is used within a module that defines an additional prefix (#1609) (fc6dfc6)

  • remove redundant print (#1608) (cc7e0e3)

7.5.0 (2022-04-26)

Features

Bug Fixes

  • properly use configfiles specified via CLI also if configfile specified via configfile directive is not present (1e0649a)

Documentation

7.4.0 (2022-04-22)

Features

  • Allow paramspace to separate filename params with custom separator (#1299) (8236e80)

Bug Fixes

7.3.8 (2022-04-06)

Bug Fixes

  • support multiple input files for template_engine rules (#1571) (aee7cf2)

7.3.7 (2022-04-05)

Bug Fixes

  • allow labels function to return None (#1565) (fef74d6)

  • do not wrap whitespace in result info headers of reports (653d0d0)

  • fixed detection of norun rules inside of modules (#1566) (d2223d4)

  • properly use retry mechanism in source cache (#1564) (624a83d)

7.3.6 (2022-04-02)

Bug Fixes

  • always recalculate job resources before job is scheduled as input might have changed or not have been present initially (#1552) (44aacdb)

  • fixed handling of input functions and unpack when using the prefix setting of module definitions (#1553) (d561e04)

  • fixed parsing of subsequent use rule statements directly beneath each other (#1548) (77d5a08)

  • fix spurious missing file errors when using google storage (#1541) (1b3ede1)

  • proper error message if resource types do not match (#1556) (1112321)

  • quote workdir in job exec prefix to allow to spaces in the workdir (#1547) (c3a593e)

  • report error and possible cause if metadata cleanup fails (#1554) (6866134)

7.3.5 (2022-03-31)

Bug Fixes

  • do not remove existing temp files in case of dryrun (#1543) (e820f97)

  • fixed bug in missing input file handling for cluster jobs (#1544) (40e2eb2)

Documentation

  • explain automatic decompression strategy for http remote provider (e6826b6)

7.3.4 (2022-03-30)

Bug Fixes

  • better error messages in case of missing files after latency period (#1528) (5b394c0)

  • correct handling of exceptions in input functions that are generators (#1536) (d9a56aa)

  • obtaining conda prefix when using in combination with singularity (#1535) (99b22d3)

  • proper error message in case of missing git when checking for source files (#1534) (92887a3)

  • throw error message in case of target rule that depends on a pipe. (#1532) (b9e9a7e)

Documentation

7.3.3 (2022-03-28)

Bug Fixes

  • better error message in case of failing to create conda env (#1526) (e7a461c)

  • fix singularity logging messages causing conda fail (#1523) (7797595)

  • more robust handling of incompletely evaluated parameters (any interaction with them will result in a string now). (#1525) (3d4c768)

Documentation

  • details on benchmarked results (64fea09)

7.3.2 (2022-03-25)

Bug Fixes

  • fixed code change detection (#1513) (67298c6)

  • modify dag and workflow display in report to also work for big DAGs (#1517) (1364dfb)

Documentation

  • Clarify the use of conda with notebook directive (#1515) (aefb1eb)

7.3.1 (2022-03-23)

Bug Fixes

  • add about page to report, including embedded packages and licenses (#1511) (142a452)

  • in google live science backend, save multiple logs per rule name and overwrite existing logs (#1504) (9e92d63)

  • in rules from imported modules, exclude modified paths from module prefixing (#1494) (1e73db0)

  • Replaced pathlib relative_to with os.relpath (#1505) (dc65e29)

  • update for minimum of Python 3.7 (#1509) (62024e2)

7.3.0 (2022-03-21)

Features

  • Support for machine_type for kubernetes executor (#1291) (12d6f67)

Bug Fixes

  • always wait for input files before starting jobs, also upon local execution and within group jobs. This should add further robustness against NFS latency issues. (#1486) (cab2adb)

  • cleaned up and rewritten execution backend structure, (fixing #1475, #860, #1007, #1008) (PR #1491) (e87cc97)

  • do not skip local conda env creation per se when having no shared FS, because it is still needed for local jobs. Instead, decide for each env whether it is needed locally or not. (#1490) (3f03c5d)

  • fixed temp file deletion for group jobs (#1487) (d030443)

  • improve robustness when retrieving remote source files, fixed usage of local git repos as wrapper prefixes (in collaboration with @cokelaer and @Smeds) (#1495) (e16531d)

  • mtime inventory for google storage was accidentally setting a float instead of a proper mtime object (#1484) (7c762c7)

  • render empty caption if nothing defined in report flag (013a6e8)

Documentation

7.2.1 (2022-03-14)

Bug Fixes

  • add missing report.templates.components module to setup.py (cb4e3fe)

Documentation

  • add install info of development (git) version to docs (#1477) (2a2d6cd)

7.2.0 (2022-03-13)

Features

  • improved reports: more interactive and modern interface, ability to define a label based representation of files (#1470) (d09df0c)

Bug Fixes

  • always deploy conda envs in main process when assuming a shared file system (fixes issue #1463) (#1472) (79788eb)

  • do not wait for named or containerized conda envs (#1473) (6b1d09c)

  • implement lock-free source file caching. This avoids hangs on network file systems like NFS. (#1464) (9520e98)

7.1.1 (2022-03-07)

Bug Fixes

  • quote jobid passed to status script to support multi-cluster Slurm setup (#1459) (0232201)

7.1.0 (2022-03-04)

Features

  • Zenodo remote provider for transparent storage on and retrieval from Zenodo (#1455) (4586ef7)

Bug Fixes

  • disable mtime retrieval from github api for now. This quickly exceeds rate limits. (1858bb9)

  • display change warnings only for jobs that won’t be executed otherwise (086f60f)

  • work around segfault with >100 jobs in google life sciences backend (#1451) (2c0fee2)

7.0.4 (2022-03-03)

Bug Fixes

  • more details on input and output exceptions (missing input, protected output, etc.) (#1453) (8d64af2)

7.0.3 (2022-03-02)

Bug Fixes

  • fix a bug leading to duplicate conda env initializations; fix display of jobs and output files with changes (994b151)

  • preserve empty names input or output file lists in params or resource functions (0d19ab0)

  • remove accidental pdb statement (9c935f1)

  • remove deprecated and add missing arguments to internal functions (93a7e39)

7.0.2 (2022-03-01)

Bug Fixes

  • add local marker for input files in cufflinks example. fixes issue #1362 (90bc88b)

  • failure to properly apply default remote prefix in combination with the unpack marker (#1448) (82666f1)

  • set mtime for cached source files [WIP] (#1443) (dd27209)

  • small bug in snakemake.executors (#1440) (6e64292)

Documentation

  • fix list display in docs (3724367)

  • fix list display in docs (2dd0e91)

  • Fix typo and grammar mistake in scatter-gather section. (#1441) (f218aaa)

7.0.1 (2022-02-26)

Bug Fixes

  • avoid incomplete remote files in case of errors and automatically retry download and upload (#1432) (8fc23ed)

  • do not apply module prefix in case of remote files (5645b3f)

  • do not require –cores or –jobs to be set when –cleanup-metadata is used. (#1429) (9c73907)

  • more robust place for runtime source file cache (#1436) (2681f6f)

  • provide details on error when failing to evaluate default resources (#1430) (04f39a9)

  • provide proper error when using immediate submit in combination with checkpoint jobs. (#1437) (865cf0f)

Documentation

  • explain relative path interpretation (#1428) (add9a05)

  • Fix problems with code blocks and broken internal link. (#1424) (5d4e7d8)

  • temaplte rendering examples and available variables (#1431) (5995e9e)

  • update copyright year (#1427) (6b9f5da)

7.0.0 (2022-02-23)

⚠ BREAKING CHANGES

  • require at least Python 3.7 (fd5daae)

Features

  • adding service jobs, i.e. the ability to define jobs that provide a resource for consumers (like a shared memory device or a database), and will be automatically terminated by Snakemake once all consumers are finished. (see docs, #1413) (a471adb)

  • support for group local jobs by enabling optional groupid consideration in input functions (see docs, #1418) (5d45493)

  • Adding –cluster-cancel and –cluster-cancel-nargs (#1395) (0593de1)

  • cluster sidecar (#1397) (b992cd1)

  • template rendering integration (yte and jinja2) (#1410) (e1cbde5)

Bug Fixes

  • bug in pipe group handling that led to multiple assignments of the same group id to different groups; bug that accidentally added already running groups of the list of ready jobs (issue #1331) (#1332) (1a9b483)

  • display wrapper or external script code in report #1393 (#1404) (a007bd1)

  • do not pass SNAKEMAKE_PROFILE into cluster-submit (#1398) (#1407) (7189183)

  • issue with duplicated prefix for checkpoints on cloud (#1294) (8ed0c8c)

  • keep flags with apply_wildcards on cloned IOFile (#1416) (23c943f)

  • remove raise that limits using –config with dicts (#1341) (bd65057)

  • Repair MREs from #823 (#1203) (b007979)

  • warn on non-file-modification-date changes like params, code, or input files (#1419) (b5f53f0)

6.15.5 (2022-02-09)

Bug Fixes

  • convert conda env to string before checks (#1382) (7a8da9f)

  • fix pepfile handling in case of module usage (#1387) (f097a76)

6.15.4 (2022-02-09)

Bug Fixes

  • fix issue when generating unit tests for rules with directory output (#1385) (7db614f)

Documentation

  • fix tutorial setup instructions for MacOS. (#1383) (b57b749)

6.15.3 (2022-02-07)

Bug Fixes

  • skip global report caption when using a module (#1379) (a755cee)

6.15.2 (2022-02-05)

Bug Fixes

  • avoid mutable default argument (#1330) (978cc93)

  • don’t raise WorkflowError when entry is empty (#1368) (1fc6f7b)

  • fix assertion error in conda env file spec when applying wildcards (thanks @ddesvillechabrol) (#1377) (6200652)

  • fix None type error when invoking Workflow object manually (#1366) (fca3895)

  • XRootDHelper.exists supports non posix filesystem (object store) (#1348) (7a3ad2f)

Documentation

6.15.1 (2022-01-31)

Bug Fixes

  • consider post-deploy script for env hashing (#1363) (d50efd9)

6.15.0 (2022-01-29)

Features

  • adding default_target directive for declaring default target rules that are not the first rule in the workflow. (#1358) (638ec1a)

Bug Fixes

  • Draft notebook filename with wildcards and params. (#1352) (11d4dc8)

  • proper error message when defining cache eligibility for rules with multiple output files and no multiext declaration. (#1357) (47b5096)

Documentation

  • Command line arguments for configuration files (#1343) (ad8aaa4)

  • fix broken link in executor_tutorial/tutorial.rst (#1360) (c9be764)

6.14.0 (2022-01-26)

Features

Documentation

6.13.1 (2022-01-11)

Bug Fixes

  • –conda-frontend value not passed on to cluster jobs (#1317) (df46ddb)

  • atomic job error display (#1326) (aa2c265)

  • fix source cache handling for remote source files retrieved via github() or gitlab() tags. (#1322) (6e2ecd2)

  • typos in code examples (#1324) (60010e4)

6.13.0 (2021-12-21)

Features

  • allow prefix definition in module statements (#1310) (29e6540)

6.12.3 (2021-12-09)

Bug Fixes

  • fixed display of any exceptions and errors from within a workflow definition (23d40d9)

6.12.2 (2021-12-07)

Bug Fixes

  • rule inheritance within modules (did previously lead to key errors) (#1292) (603e0a8)

Documentation

  • Fix typo in rules.rst (—draft-notebook) (#1290) (f5c42cf)

6.12.1 (2021-11-29)

Bug Fixes

  • set default number of nodes to 1 in test cases (#1288) (f6e12b4)

6.12.0 (2021-11-29)

Features

  • add flag –draft-notebook for generating a skeleton notebook for manual editing (e.g. in VSCode). (#1284) (d279322)

Bug Fixes

Documentation

6.11.1 (2021-11-26)

Bug Fixes

  • provide temporary IPYTHONDIR for notebook execution in order to avoid race conditions in https://github.com/ipython/ipython/blob/master/IPython/paths.py#L20 upon execution of multiple notebooks at the same time. (#1280) (4d70da1)

Documentation

  • move psutil import into benchmark methods to avoid needing it as a dependency for doc building (6ffe38d)

  • require sphinx>=3 (1773875)

  • skip lazy property (2883718)

6.11.0 (2021-11-25)

Features

  • fail with an error if snakemake cannot write job metadata. (#1273) (cd968cd)

Bug Fixes

  • Adds fixes for the first two MREs in #823 (#1215) (cfd2f89)

  • env file usage after changes to source file handling (inspired by #1233 and #1211). (#1236) (3ac8e85)

  • fixed code change detection when using modules (#1264) (b571e09)

  • handle config file extension/overwriting more explicitly (#1251) (d0a7bf2)

  • Issue #1253 (problems editing Jupyter Notebooks) (#1255) (3398ddf)

  • more informative nothing to be done message (#1234) (368d265)

  • only consider context of shell command for technical switches if called from snakemake rules. (#1213) (4816a58)

  • R encoding of pathlib.Path objects (#1201) (bd516e9)

  • Use ‘snakemake.utils.update_config’ instead of ‘dict.update’ (#1126) (2658027)

6.10.0 (2021-10-21)

Features

  • Add more informative errors when evaluation of --default-resources fails (#1192) (b3c4e68)

Bug Fixes

  • add quotes to each item of the wait_for_files list (#1160) (72856ed)

  • caching process (#1225) (0825a29)

  • enable usage of job grouping in GLS (#1054) (d243c22)

  • Only –bind Snakemake when we’re working with a Python script (#1206) (1d79f62)

  • run dependencies with non-existent ancient files before the consuming job (#1202) (84d1f64), closes #946

  • status cmd repeats until killed by 11 different signals (#1207) (8b28b57)

  • typo in sourcecache use (#1229) (8b54bc5)

  • wms monitor arg parsing now accepts any kind of value (#1181) (313de93)

Documentation

  • Clarification of –cluster-stats docs & elaborating on the situation where job ids are not passed to the status script (#1221) (ed0e4a2)

  • Combine CHANGELOG.rst with CHANGELOG.md (#1228) (19f5a43)

  • Mention required openssl dep for rust-script (#1216) (fc8c5f6)

  • Unpin docutils version (#1230) (15a82bf)

6.9.1 (2021-09-30)

Bug Fixes

  • fix function call when creating report and hashes for between workflow caching (#1198) (a4f6836)

6.9.0 (2021-09-29)

Features

  • autoconvert Path objects to str when passing to R or Julia scripts (80ec513)

Bug Fixes

  • fix source retrieval during between workflow caching and report generation (2394ca4)

6.8.2 (2021-09-29)

Bug Fixes

  • fix path returned by get_source() (ee05315)

6.8.1 (2021-09-24)

Bug Fixes

  • async_run to allow nested event loops. (#1170) (5dc6bbd)

  • merging of pipe groups when multiple rules are chained together via pipes (#1173) (de91d2c)

  • potential memory corruption caused by Google storage objects accessed from different threads (#1174) (41a5071)

Performance Improvements

  • more extensive caching of source files, including wrappers. (#1182) (bdb75f8)

Documentation

6.8.0 (2021-09-06)

Features

Bug Fixes

  • AmbiguousRuleException bug caused by weak ordering of rules (#1124) (7f54c39)

  • Bugfix tes add files (#1133) (8892bf2)

  • Disable Persistence cache for snakemake jobs (#1159) (7110f9d)

  • efficient job status checking when using DRMAA API (this should yield much better parallelization and performance when using –drmaa) (#1156) (ac004cb)

  • improved error handling for cluster status scripts and smarter job selector choice in case of cluster submission (use greedy for single jobs). (#1142) (48d2dd9)

  • Initialize assignments dictionary when setting rule-based resources (#1154) (68c13fd)

  • key error when handling FileNotFoundError in input functions. (#1138) (d25f04d)

  • linting of remote snakefiles (#1131) (2104e10)

Performance Improvements

  • improve job selection performance in case of potential ambiguity that is resolved by comprehensive ruleorder statements. (#1147) (921f4f7)

6.7.0 (2021-08-12)

Features

  • Add support for rust scripts (enabling directly integrated ad-hoc robust high performance scripting) (#1053) (f0e8fa2)

Bug Fixes

  • Ga4gh tes bugfixes (#1127) (af21d6c)

  • improved display of percentage of done jobs (1fee8c0)

  • improved error message in case of target rule misspecification (83b1f5b)

Documentation

  • fix contributing executors link (#1112) (4bb58d1)

  • Fix typo in file path in remote files documentation (#1110) (9ce294f)

6.6.1 (2021-07-19)

Bug Fixes

  • avoid superfluous calls of conda info that have slowed down Snakemake since 6.4.1. (#1099) (e990927)

6.6.0 (2021-07-16)

Features

  • Allow to mark all output files as temp with –all-temp (#1097) (0ac3b38)

6.5.5 (2021-07-16)

Bug Fixes

6.5.4 (2021-07-16)

Fixes

  • Fixed –touch in combination with temp files (issue #1028) (@johanneskoester, @iromeo).

Documentation

  • Fix syntax error in docs/conf.py and update sphinx.ext.napoleon import (#1084) (3e3fac2)

  • Improved pepfile (pepschema) documentation (@stolarczyk).

[6.5.3] - 2021-07-06

  • Fixed a bug occurring when using –resources in the command line interface (@johanneskoester).

  • Minor improvements in the docs (@johanneskoester).

[6.5.2] - 2021-07-