61M
more variants added to the research aggregate
10%
improvement in variant pass rate
2x
improvement in pipeline execution time
Aim
To develop an aggregate data product from the National Genomic Research Library (NGRL) that enables researchers to derive scientific insights from rare disease genomes at scale.Challenges
- →Production Reliability: Compounding storage and container failures repeatedly stalled the post-aggregate build processing.
- →Workflow Diversity: A single environment needed to support vastly different Nextflow workflows with varying resource demands.
- →Storage Unpredictability: No way to control disk allocation across diverse workloads, causing frequent capacity errors from large intermediary files.
- →Container Overhead: Redundant image pulls via the public internet created I/O bottlenecks and widespread timeout failures at scale.
— Catherine Snow, Product Manager
Solution
Nextflow
Built standardized, custom pipelines that handled each stage of the research aggregate, from Dragen realignment through to QC, annotations, population structure and genetic similarity inference, and high-quality SNP identification.
Seqera
Orchestrated workflows at scale across multiple environments, providing infrastructure control without modifying the pipelines themselves.
Terraform Provider
Enabled fully reproducible infrastructure-as-code deployment across multiple environments, to replace an unmaintained third-party provider.
Results
Genomics England partnered with Seqera to develop tailored solutions across infrastructure configuration, container management, and deployment tooling. This enabled Genomics England to deliver research data aggregate, AggV3, to their researchers, with all post-aggregate supporting data generated using cloud-native Nextflow pipelines orchestrated by Seqera. The result was a near doubling of the throughput capacity, improved variant quality, and a significant increase in the breadth of genomic data available to researchers in the form of an aggregated dataset. By centralizing and standardizing production with Seqera, Genomics England delivered a fully traceable, reproducible data product now immediately accessible to their entire research community. It also marked the first time Genomics England had run this scale of data on a cloud-only infrastructure. Without Seqera Platform’s orchestration it would not have been possible to provide the supporting data in such a short timeframe.
61 Million
Opportunity
Accelerated Research Access
61 million more variants and nearly double the germline genomes are now immediately available to researchers.
Foundation for Future Discovery
A standardized, reproducible process means extending the aggregate with new data products becomes significantly simpler, compounding its value over time.
Pathway to Patient Impact
Supports Genomics England's mission for genomics to feature in up to half of all healthcare interactions by 2035.
About
Genomics England is a global leader in enabling genomic medicine and research, with a vision of a world where everyone benefits from genomic healthcare. Building on the 100,000 Genomes Project, it is helping to transform health by making genomics a routine part of care - unlocking better diagnoses, treatments, and prevention for everyone. It supports the NHS’s world-first national whole genome sequencing service and runs the growing National Genomic Research Library, alongside delivering numerous major genomics initiatives including the Generation Study. It works with the NHS, researchers, industry and its participants to enable immediate healthcare benefits and advances for the future – harnessing innovation to deliver the digital systems and evidence needed for genomics to guide up to half of all healthcare interactions by 2035.
To learn more, visit https://www.genomicsengland.co.uk/