Building an nf-core style modules library for your organization

One of the strongest selling points for developing your bioinformatics pipelines using Nextflow is the incredible set of developer tools and assets created by the open-source community nf-core. In addition to the 100+ community-maintained pipelines available off-the-shelf, the nf-core community also maintains a collection of 1400+ individual reusable modules. Each nf-core module includes a pre-written Nextflow process definition for a specific command line software tool, including container images, a conda environment definition, unit tests with test data, and documentation. In totality, all these features mean that adding commonly-used bioinformatics tools to your pipelines can be as simple as installing the nf-core CLI, looking up the appropriate modules on the nf-core website, and running nf-core modules install fastqc within your project directory.

FAIR Code

The nf-core approach to packaging code elegantly addresses one of the biggest hidden inefficiencies in the bioinformatics software community: duplicative effort caused by difficulty in reusing code. This difficulty in code reuse is similar to the challenges faced by other digital scientific assets (such as data), which makes it really interesting to look at the nf-core modules library through the lens of the FAIR Principles for scientific data. Not only do nf-core modules adhere to the principles of being Findable, Accessible, Interoperable, and Reusable, but you can also map specific nf-core CLI commands to each principle showing that the FAIR principles are supported with tooling.

Bringing FAIR code to your own organization

If you would like to bring nf-core’s approach to enabling code within your own organization, you can! The nf-core python CLI supports the --git-remote argument that allows you to install modules from any module repository with the appropriate structure. One example that you can explore to see an organization using this approach is Memorial Sloan Kettering Cancer Center omics-workflows repository.

Use nf-core tools to make your code FAIR

The easiest way to set up a modules library for your organization is to use the copier template located at nf-core/modules-template. Copier is a Python CLI for rendering project templates that has the advantage of allowing child projects to be updated with later improvements from the parent template. Here are the steps to get it set up:

1. Install copier:

pipx install copier

2. Use the copier template:

copier copy https://github.com/nf-core/modules-template
./my-new-modules-library

3. Follow the prompts of the copier CLI

4. Run the setup script:
cd my-new-modules-library; bash ./project_init.sh

Currently the modules template only supports CI/CD using GitHub, however, support for additional CI/CD systems is planned (and PRs are welcome).

One of the key files in an nf-core compatible modules repository that Copier will create for you is the file .nf-core.yml which contains the lines:

repository_type: modules
org_path: my-custom-org

This tells the nf-core CLI what type of repository it is working with, and what path it can use to find all the relevant modules (e.g. modules/my-custom-org/custom-module ).

Working with your modules library

After initializing the modules library, the best way to add new modules is using the command nf-core modules create and loosely following the corresponding instructions in the nf-core docs. Beyond adding the Nextflow code for your module, you’ll want to add some nf-test cases and fill out the meta.yml file which includes details about your module that can be rendered as documentation.

The additional overhead of adding tests and documentation to your modules may not be worth it for every bit of code. If a specific module will only ever be used in one specific pipeline and isn’t broadly useful, then it usually is better to keep it as a local module in its pipeline instead of adding it to a library. If you’re building a module for publicly available software, it’s also worth considering if the module might be able to be added to the nf-core library instead of a private library. When a module becomes part of nf-core/modules it can be maintained, and improved by the broader open-source community. However, for organizations modularizing proprietary code, using nf-core tooling is an incredibly powerful approach to building reusable components for your organization’s pipelines.

💡Get started with building your own modules library by visiting https://github.com/nf-core/modules-template.