Bringing Nextflow to Google Cloud Platform with WuXi NextCODE

Google Cloud and WuXi NextCODE are dedicated to advancing the state of the art in biomedical informatics, especially through open source, which allows developers to collaborate broadly and deeply.

WuXi NextCODE is itself a user of Nextflow, and Google Cloud has many customers that use Nextflow. Together, we’ve collaborated to deliver Google Cloud Platform (GCP) support for Nextflow using the Google Pipelines API. Pipelines API is a managed computing service that allows the execution of containerized workloads on GCP.

Nextflow now provides built-in support for Google Pipelines API which allows the seamless deployment of a Nextflow pipeline in the cloud, offloading the process executions as pipelines running on Google's scalable infrastructure with a few commands. This makes it even easier for customers and partners like WuXi NextCODE to process biomedical data using Google Cloud.

Get started!

This feature is currently available in the Nextflow edge channel. Follow these steps to get started:

Install Nextflow from the edge channel exporting the variables shown below and then running the usual Nextflow installer Bash snippet:

export NXF_VER=18.12.0-edge
export NXF_MODE=google
curl https://get.nextflow.io | bash

Enable the Google Genomics API for your GCP projects.
Download and set credentials for your Genomics API-enabled project.
Change your nextflow.config file to use the Google Pipelines executor and specify the required config values for it as described in the documentation.
Finally, run your script with Nextflow like usual, specifying a Google Storage bucket as the pipeline work directory with the -work-dir option. For example:

nextflow run rnaseq-nf -work-dir gs://your-bucket/scratch

You can find more detailed info about available configuration settings and deployment options here.

We’re thrilled to make this contribution available to the Nextflow community!