Seqera

Coming soon to Nextflow: From command-line to cleanup

Watch on YouTube

In this News and Views episode, Phil Ewels talks to Ben Sherman about upcoming features in core Nextflow development: a new CLI v2 interface; submission of job arrays and intermediate file cleanup during runs.

CLI v2

  • The new command-line interface will be invoked with the nf command
  • It will co-exist with the current nextflow CLI, at least for a while
  • Expect argument / option parsing more in line with linux norms
    • No longer a single-dash for core flags and double-dash for pipeline parameters
    • Single dash for short flags, double for long-form flags
    • Core and pipeline options separated with --

See the pull-request here: Add CLI v2 #3600

Job Arrays

  • Soon you’ll be able to submit job arrays to HPC and cloud
  • Group the submission of any number (10s-1000s) of jobs
  • Still independent jobs, but submitted in one go

See the pull-request here: Implement job array using TaskRun as carrier #3905

Intermediate file cleanup

  • New ability to track which intermediate files are no longer needed
  • Allows files to be deleted from the work directory during a run
  • Reduces the peak storage required for large pipeline runs

See the pull-request here: Add support for temporary output paths #3818