The Open Science Blog

Bioinformaticians often run their Nextflow workflows on internal HPC clusters, typically using job schedulers like Slurm to manage tasks. However, as the need for scalability and flexibility grows, many are making the shift to public cloud platforms. In this post, we'll highlight key details from a recent real-world example to demonstrate details about this process.

Lynn Langit & David Haley

We thought it was high time that we gave some attention to EPI2ME Workflows - a set of professionally maintained Nextflow pipelines developed by the EPI2ME team at Oxford Nanopore Technologies (ONT).

Paolo Di Tommaso

Nextflow workflows are widely used across various fields, performing tasks ranging from data retrieval and cleaning to processing and downstream analyses. Generally, workflows which feed into other workflows are run independently. They are also often not written in a way they can be included directly as subworkflows into another workflow, making it challenging to connect them seamlessly. nf-cascade is a proof-of-concept pipeline that demonstrates how multiple existing workflows can be integrated into a single workflow. Specifically, the proof-of-concept connects the nf-core pipelines: nf-core/fetchngs, nf-core/taxprofiler, nf-core/mag, and nf-core/funcscan. The code can be generalised further to run and connect any existing workflow, without any modifications necessary to the workflows you want to run.

Mahesh Binzer-Panchal

I approach bioinformatics workflows in a whole different way now that I’ve used Nextflow. It has evolved into a vital tool in my toolbox, capable of handling massive genomic datasets and guaranteeing repeatability and scalability. Thanks to Nextflow, I can automate and streamline tasks that used to take days or even weeks, whether it’s merging disparate software applications or simplifying intricate pipelines. To go beyond the constraints of conventional scripting (bash, perl, Python, R) for genomic data analysis was the driving force for my decision to work with Nextflow. Since then, it’s opened up new possibilities, leading me to collaborations, innovation, and now this exciting new role as an Ambassador! I am Pritam Kumar Panda, a bioinformatician working at the German Cancer Research Center DKFZ, Heidelberg, Germany.

Pritam Kumar Panda

I was honored to be invited to the ISMB 2024 congress to speak at the session organised by the COSI (Community of Special Interest) of Bioinformatics Cores. This session brought together bioinformatics professionals from around the world who manage bioinformatics facilities in different institutions to share experiences, discuss challenges, and explore solutions for managing and analyzing large-scale biological data. In this session, I had the opportunity to introduce Nextflow, and discuss how its adoption can help bioinformatics cores to address some of the most common challenges they face.

Francesco Lescai

As the Nextflow community continues to grow, fostering a space where users can easily find help and share knowledge is more important than ever. In this post, we’ll explore our ongoing efforts to enhance the community forum, transitioning from Slack as the primary platform for peer-to-peer support.

Geraldine Van der Auwera

When I (Ben) was in grad school, I worked on a Nextflow pipeline called GEMmaker, an RNA-seq analysis pipeline similar to nf-core/rnaseq. We quickly ran into a problem, which is that on large runs, we were running out of storage! As it turns out, it wasn’t the final outputs, but the intermediate outputs (the BAM files, etc) that were taking up so much space, and we figured that if we could just delete those intermediate files sooner, we might be able to make it through a pipeline run without running out of storage. We were far from alone.

Ben Sherman

As a PhD student in bioinformatics, I aimed to build robust pipelines to analyze diverse datasets throughout my research. Initially, mastering Bash scripting was a time-consuming challenge, but this journey ultimately led me to become a Nextflow Ambassador, engaging actively with the expert Nextflow community.

Firas Zemzem

State of the Workflow 2024 Community Survey Results: Insights from 600+ Nextflow users about the state of workflow management for scientific data analysis

Evan Floden

As a Nextflow Ambassador and a PhD student working in bioinformatics, I’ve always believed in the power of collaboration. Over the past six months, I’ve had the privilege of working with another PhD student specializing in metagenomics environmental science. This collaboration began through a simple email after the other researcher discovered my contact information on the ambassadors’ list page.

Cristina Tuñi i Domínguez

After diving into the Nextflow community, I’ve seen how it benefits bioinformatics in places like South Africa, Brazil, and France. I’m confident it can do the same for Türkiye by fostering collaboration and speeding up research. Since I became a Nextflow Ambassador, I am happy and excited because I can contribute to this development!

Kübra Narcı

We release an “edge” version of Nextflow every month and a “stable” version every six months. The stable releases are recommended for production usage and represent a significant milestone. The release changelogs contain a lot of detail, so we thought we’d highlight some of the goodies that have just been released in Nextflow 24.04 stable. Let’s get into it!

Paolo Di Tommaso

Nextflow Ambassadors are passionate individuals within the Nextflow community who play a more active role in fostering collaboration, knowledge sharing, and engagement. We launched this program at the Nextflow Summit in Barcelona last year, and it’s been a great experience so far, so we’ve been recruiting more volunteers to expand the program.

Marcel Ribeiro-Dantas

This is a joint article contributed to the Seqera blog by Jon Manning of Seqera and Felix Krueger of Altos Labs describing the new nf-core/riboseq pipeline.

Jon Manning

Nextflow and nf-core provide frequent community training events to new users, which offer an opportunity to get started using and understanding Nextflow, Groovy and nf-core. These events are live-streamed and are available for on-demand viewing on YouTube, but what if you could join friends in person and watch it live?

Florian Wuennemann

Check out Nextflow's newest plugin, nf-schema! It's an enhanced version of nf-validation, utilizing JSON schemas to validate parameters and sample sheets. Unlike its predecessor, it supports the latest JSON schema draft and can convert pipeline-generated files. But what's the story behind its development?

Nicolas Vannieuwkerke

Improved Singularity/Apptainer support in Nextflow and Wave.

Marco De La Pierre

Ancient environmental DNA (eDNA) is currently a hot topic in archaeological, ecological, and metagenomic research fields. Recent eDNA studies have shown that authentic ‘ancient’ DNA can be recovered from soil and sediments even as far back as 2 million years ago.

James Fellows Yates

From December 2022 to March 2023, I was part of the second cohort of the Nextflow and nf-core mentorship program, which spanned four months and attracted participants globally. I could not have anticipated the extent to which my participation in this program and the associated learning experiences would positively change my professional growth.

Anabella Trigila

Reproducibility is an important attribute of all good science. This is especially true in the realm of bioinformatics, where software is hopefully being updated, and pipelines are ideally being maintained. Improvements and maintenance are great, but they also bring about an important question: Do bioinformatics tools and pipelines continue to run successfully and produce consistent results despite these changes? Fortunately for us, there is an existing approach to ensure software reproducibility: testing.

Carson Miller & Sateesh Peri

With the third round of the Nextflow and nf-core mentorship program now behind us, it’s time to pop the confetti and celebrate the outstanding achievements of our latest group of mentors and mentees!

Marcel Ribeiro-Dantas

After a three-year COVID-related hiatus from in-person events, Nextflow developers and users found their way to Barcelona this October for the 2022 Nextflow Summit. Held at Barcelona’s iconic Agbar tower, this was easily the most successful Nextflow community event yet!

Noel Ortiz

We are excited to announce the launch of the Nextflow Ambassador Program, a worldwide initiative designed to foster collaboration, knowledge sharing, and community growth. It is intended to recognize and support the efforts of our community leaders and marks another step forward in our mission to advance scientific research and empower researchers

Marcel Ribeiro-Dantas

We are very excited to introduce the Seqera community forum - the new home of the Nextflow community!

We're excited to welcome Geraldine to Seqera as Lead Developer Advocate to help the Nextflow community in the USA grow and thrive!

Geraldine Van der Auwera

I’m excited to announce that I’m joining Seqera as Lead Developer Advocate. My mission is to support the growth of the Nextflow user community, especially in the USA, which will involve running community events, conducting training sessions, managing communications and working globally with our partners across the field to ensure Nextflow users have what they need to be successful. I’ll be working remotely from Boston, in collaboration with Paolo, Phil and the rest of the Nextflow team.

Geraldine Van der Auwera

Save time and improve efficiency by automating end-to-end processes..

Adam Talbot

Staying agile and portable in the age of cloud with Nextflow and Tower.

Robert Lalonde

The Nextflow project originated from within an academic research group, so perhaps it’s no surprise that education is an essential part of the Nextflow and nf-core communities. Over the years, we have established several regular training resources: we have a weekly online seminar series called nf-core/bytesize and run hugely popular bi-annual Nextflow and nf-core community training online. In 2022, Seqera established a new community and growth team, funded in part by a grant from the Chan Zuckerberg Initiative “Essential Open Source Software for Science” grant. We are all former bioinformatics researchers from academia and part of our mission is to build resources and programs to support academic institutions.

Marcel Ribeiro-Dantas

An overview of the impact of AI in health care and an example Nextflow pipeline for training a machine learning model.

Ben Sherman

A comprehensive set of professional services related to Nextflow, Nextflow pipelines, and Nextflow Tower

Robert Lalonde

In this article we present the various storage solutions supported by Nextflow including on-prem and cloud file systems, parallel file systems, and cloud object stores. We also discuss Fusion file system 2.0, a new high-performance file system that can help simplify configuration, improve throughput, and reduce costs in the cloud.

Paolo Di Tommaso

Read about our State of the Workflow Community survey to get a sense of activities and preferences in the Nextflow user community.

Evan Floden

In mid-March, we conducted our bi-annual Nextflow and nf-core training and hackathon in what was unquestionably our best-attended community events to date. This year we had an impressive 1,345 participants attend the training from 76 countries. Attendees came from far and wide — from Algeria to Andorra to Zambia to Zimbabwe!

The global Nextflow and nf-core community is thriving with strong engagement in several countries. As we continue to expand and grow, we remain committed to prioritizing inclusivity and actively reaching groups with low representation.

Chris Hakkaart

A hands-on-tutorial explaining how Nextflow Tower can simplify GPU-enable training and inference.

Evan Floden

Hi, my name is Ben, and I’m a software engineer at Seqera Labs. I joined Seqera in November 2021 after finishing my Ph.D. at Clemson University. I work on a number of things at Seqera, but my primary role is that of a Nextflow core contributor.

Ben Sherman

Get an opportunity to win an Apple Macbook Air M2 or one of ten sustainability-certified Seqera merchandise packs

Evan Floden

While we can't predict the next pandemic, we can be certain that new threats will emerge. Learn how Nextflow is helping prepare us to fight future pandemics and outbreaks.

Evan Floden

An overview of the Fusion 2.0 file system and benchmarks comparing Fusion to alternative approaches to data handling in Nextflow.

Paolo Di Tommaso

We have talked about Google Cloud Batch before. Not only that, we were proud to announce Nextflow support to Google Cloud Batch right after it was publicly released, back in July 2022. How amazing is that? But we didn’t stop there! The Nextflow official documentation also provides a lot of useful information on how to use Google Cloud Batch as the compute environment for your Nextflow pipelines.

Marcel Ribeiro-Dantas

The ability to resume an analysis (i.e. caching) is one of the core strengths of Nextflow. When developing pipelines, this allows us to avoid re-running unchanged processes by simply appending -resume to the nextflow run command. Sometimes, tasks may be repeated for reasons that are unclear. In these cases it can help to look into the caching mechanism, to understand why a specific process was re-run.

Abhinav Sharma

Follow our step-by-step guide and learn how to build and deploy your own scientific containers with Nextflow.

Evan Floden

After a three-year COVID-related hiatus from in-person events, Nextflow developers and users found their way to Barcelona this October for the 2022 Nextflow Summit. Held at Barcelona’s iconic Agbar tower, this was easily the most successful Nextflow community event yet!

Noel Ortiz

Learn about Easy Genomics - an innovative effort by Wisconsin State Laboratory of Hygiene (WSLH), Two Bulls, and AWS based on Nextflow Tower.

Venkata Kampana

Learn how to easily deploy your Nextflow pipelines using the Seqera Platform and nf-core tools.

Evan Floden

Learn how Tower's new resource optimization feature can help you accelerate pipeline execution and dramatically reduce cloud spending.

Evan Floden

Containers have become an essential part of well-structured data analysis pipelines. They encapsulate applications and dependencies in portable, self-contained packages that can be easily distributed. Containers are also key to enabling predictable and reproducible results.

Paolo Di Tommaso

Nextflow is a powerful workflow manager that supports multiple container technologies, cloud providers and HPC job schedulers. It shouldn’t be a surprise that wide ranging functionality leads to a complex interface, but comes with the drawback of many subcommands and options to remember. For a first-time user (and sometimes even for some long-time users) it can be difficult to remember everything. This is not a new problem for the command-line; even very common applications such as grep and tar are famous for having a bewildering array of options.

Marcel Ribeiro-Dantas

Our recent The State of the Workflow 2022: Community Survey Results showed that Nextflow and nf-core have a strong global community with a high level of engagement in several countries. As the community continues to grow, we aim to prioritize inclusivity for everyone through active outreach to groups with low representation.

Chris Hakkaart

A deep dive into Nextflow on Azure Batch, how the integration works, and different deployment options.

Evan Floden

An overview of the omics disciplines and the role of Nextflow and cloud computing in advancing research

Harshil Patel

Run portable, reproducible bioinformatics pipelines in the Google Cloud.

Evan Floden

As recently announced, we are super excited to host a new Nextflow community event late this year! The Nextflow Summit will take place October 12-14, 2022 at the iconic Torre Glòries in Barcelona, with an associated nf-core hackathon beforehand.

The State of the Workflow 2022: The Nextflow and nf-core Community. The results from our Nextflow user survey, covering where and how people use Nextflow and nf-core.

Gord Sissons

Introducing Pipeline Secrets. Securely store and use sensitive information for pipeline executions with Tower.

Mattia Bosio

Software development is a constantly evolving process that requires continuous adaptation to keep pace with new technologies, user needs, and trends. Likewise, changes are needed in order to introduce new capabilities and guarantee a sustainable development process.

Paolo Di Tommaso

Phil Ewels outlines his journey, from his beginnings in the Babraham Institute, his time at SciLifeLab, to working full-time with Nextflow.

Phil Ewels

Win an Apple Macbook Air M1 along with one of ten Seqera merch packs

Evan Floden

Dr. Harshil Patel's journey to leading pipeline development at the home of Nextflow and nurturing the relationship between nf-core and Seqera.

Harshil Patel

The recent tweet introducing the Nextflow support for SQL databases raised a lot of positive reactions. In this post, I want to describe more in detail how this extension works.

Paolo Di Tommaso

A guest post on the June 2021 updates, including the march towards Nextflow DSL2, with Phil Ewels from the nf-core community.

Phil Ewels

In May we blogged about Five Nextflow Tips for HPC Users and now we continue the series with five additional tips for deploying Nextflow with on HPC batch schedulers.

Kevin Sayers

Nextflow is a powerful tool for developing scientific workflows for use on HPC systems. It provides a simple solution to deploy parallelized workloads at scale using an elegant reactive/functional programming model in a portable manner.

Kevin Sayers

Nextflow Tower has taken next step for multi-platform collaboration at scale with the release of the Workspace and Organizations features.

Paolo Di Tommaso

The State of Nextflow 2021 - the results from our Nextflow user survey, covering where and how people use Nextflow, their industry, and their teams.

Evan Floden

This guide is designed to walk you through a basic development setup for writing Nextflow pipelines.

Evan Floden

When the Nextflow project was created, one of the main drivers was to enable reproducible data pipelines that could be deployed across a wide range of execution platforms with minimal effort as well as to empower users to scale their data analysis while facilitating the migration to the cloud.

Paolo Di Tommaso

Break-through Science - Machine Learning and Imaging Pipelines using Nextflow, following a conversation with Felix Morency, the CTO of Imeka.

Robert Lalonde

State of Nextflow Survey 2021

Evan Floden

With the year nearly over, we thought it was about time to pull together the best-of-the-best guide for learning Nextflow in 2020. These resources will support anyone in the journey from total noob to Nextflow expert so this holiday season, give yourself or someone you know the gift of learning Nextflow!

The latest Nextflow version 2020.10.0 is the first stable release running on Groovy 3.

Paolo Di Tommaso

For most developers, the command line is synonymous with agility. While tools such as Nextflow Tower are opening up the ecosystem to a whole new set of users, the Nextflow CLI remains a bedrock for pipeline development. The CLI in Nextflow has been the core interface since the beginning; however, its full functionality was never extensively documented. Today we are excited to release the first iteration of the CLI documentation available on the Nextflow website.

Abhinav Sharma

We are thrilled to announce the stable release of Nextflow DSL 2 as part of the latest 20.07.1 version!

Paolo Di Tommaso

Continuing our series on understanding Nextflow resume, we wanted to delve deeper to show how you can report which tasks contribute to a given workflow output.

Evan Floden

An overview of cloud-native data pipelines and the need for portable workflows and for hybrid computing

Evan Floden

Day one! We are thrilled to announce the launch of Seqera, founded by Paolo Di Tommaso and Evan Floden.

Evan Floden

We are excited to announce the new Nextflow 19.04.0 stable release!

Paolo Di Tommaso

It’s time for the monthly Nextflow release for March, edge version 19.03. This is another great release with some cool new features, bug fixes and improvements.

Evan Floden

Google Cloud and WuXi NextCODE are dedicated to advancing the state of the art in biomedical informatics, especially through open source, which allows developers to collaborate broadly and deeply.

Paolo Di Tommaso

Today marks an important milestone in the Nextflow project. We are thrilled to announce three important changes to better meet users’ needs and ground the project on a solid foundation upon which to build a vibrant ecosystem of tools and data analysis applications for genomic research and beyond.

Paolo Di Tommaso

One key feature of Nextflow is the ability to automatically pull and execute a workflow application directly from a sharing platform such as GitHub. We realised this was critical to allow users to properly track code changes and releases and, above all, to enable the seamless sharing of workflow projects.

Paolo Di Tommaso

Over past week there was some discussion on social media regarding the Nextflow license and its impact on users’ workflow applications.

Paolo Di Tommaso

Nextflow aims to ease the development of large scale, reproducible workflows allowing developers to focus on the main application logic and to rely on best community tools and best practices.

Paolo Di Tommaso

This is a guest post authored by Maxime Garcia from the Science for Life Laboratory in Sweden. Max describes how they deploy complex cancer data analysis pipelines using Nextflow and Singularity. We are very happy to share their experience across the Nextflow community.

Maxime Garcia

The latest Nextflow release (0.26.0) includes built-in support for AWS Batch, a managed computing service that allows the execution of containerised workloads over the Amazon EC2 Container Service (ECS).

Paolo Di Tommaso

Last week saw the inaugural Nextflow meeting organised at the Centre for Genomic Regulation (CRG) in Barcelona. The event combined talks, demos, a tutorial/workshop for beginners as well as two hackathon sessions for more advanced users.

Evan Floden

Learn how to deploy an elastic computing cluster in the AWS cloud with Nextflow.

Paolo Di Tommaso

Nextflow is a framework that simplifies the writing of parallel and distributed computational pipelines in a portable and reproducible manner across different computing platforms, from a laptop to a cluster of computers.

Paolo Di Tommaso

Recently a new feature has been added to Nextflow that allows failing jobs to be rescheduled, automatically increasing the amount of computational resources requested.

Paolo Di Tommaso

As a new bioinformatics student with little formal computer science training, there are few things that scare me more than PhD committee meetings and having to run my code in a completely different operating environment.

Evan Floden

In a recent publication we assessed the impact of Docker containers technology on the performance of bioinformatic tools and data analysis workflows.

Paolo Di Tommaso