Running life science pipelines efficiently at scale is hard. They are both data- and compute-intensive: a single run can combine tens of thousands of tasks, from jobs lasting hours to tasks that finish in seconds, all putting heavy pressure on file systems and object storage. They also chain together heterogeneous tools with tens or hundreds of different dependencies and containers.
The cloud adds another layer of complexity. Varied machine types, disks and network latencies produce an almost endless range of transient errors, and performance is far from predictable: a task that runs fast on one instance type can crawl on another because of slow disk, insufficient memory or congestion.
Earlier this year, we launched Seqera Intelligent Compute, a new-generation compute service that abstracts away the complexity of scaling Nextflow pipelines in the cloud. The result is more resource-efficient workflows: fewer instances per run, less unutilized capacity, and lower overall costs. We've previously published benchmarks against AWS Batch. Now we've extended that testing to the other two major cloud providers, comparing Intelligent Compute with Azure Batch and Google Cloud Batch across six nf-core pipelines: rnaseq, sarek, atacseq, chipseq, nanoseq, and viralrecon.
Intelligent Compute drives this improvements by replacing static, user-defined configurations with decisions made by the compute layer itself. It continuously profiles running pipelines and treats every run as new information, rather than reapplying the same fixed settings and retry logic regardless of what happened last time.
Fully Cloud-native Intelligent Compute is built directly on virtual machines and object storage, meaning there are no compute clusters to create and maintain, no infrastructure to configure in advance, fewer cloud permissions to grant, and no fixed quotas that restrict you.
Intelligent Sizing Based on each task's observed memory, CPU and disk usage, and its duration, Intelligent Compute selects the most efficient machine type and configuration at runtime, weighing the provider's current pricing and availability.
Zero manual tuning Right-sizing, job-packing and spot management work together automatically, so teams stop paying for compute they don't use and stop maintaining configurations that go out of date as pipelines evolve.
Automated recovery Failures are diagnosed and handled at the compute layer instead of needing to be handled in pipeline code: tasks that run out of memory are retried with a larger allocation, spot-preempted tasks are snapshotted and resumed on another instance and tasks that are repeatedly preempted move to on-demand.
Continuous improvement Every run sharpens predictions for the next, so resource allocation, job-packing and cost performance keep improving the more a pipeline is used, with no extra effort.
Benchmarking Intelligent Compute vs. Cloud Providers
We benchmarked Intelligent Compute against the native batch service of each cloud (AWS Batch, Azure Batch and Google Cloud Batch) using popular nf-core pipelines including rnaseq, sarek, atacseq, chipseq, nanoseq and viralrecon (test_full and test_full_nanopore profiles). Results are the mean of the three most recent runs on each platform, measured by cost per run, CPU hours and wall clock time.
AWS Batch vs. Intelligent Compute
Cost per run
AWS Batch
Intelligent Compute
% Change
rnaseq
$24.65
$9.61
−61%
sarek
$13.86
$7.79
−44%
atacseq
$8.34
$2.46
−71%
chipseq
$27.76
$9.84
−65%
nanoseq
$3.62
$2.15
−41%
viralrecon
$7.50
$3.46
−54%
viralrecon (nanopore)
$4.60
$1.61
−65%
CPU Hours
AWS Batch
Intelligent Compute
% Change
rnaseq
375 cpu-h
146 cpu-h
−61%
sarek
129 cpu-h
81 cpu-h
−37%
atacseq
112 cpu-h
34 cpu-h
−70%
chipseq
380 cpu-h
203 cpu-h
−47%
nanoseq
36 cpu-h
15 cpu-h
−57%
viralrecon
78 cpu-h
36 cpu-h
−53%
viralrecon (nanopore)
52 cpu-h
11 cpu-h
−78%
Wall Clock Time
AWS Batch
Intelligent Compute
% Change
rnaseq
4.96 h
2.73 h
−45%
sarek
12.45 h
5.85 h
−53%
atacseq
5.41 h
1.93 h
−64%
chipseq
5.72 h
3.43 h
−40%
nanoseq
2.50 h
1.44 h
−42%
viralrecon
1.01 h
0.93 h
−7%
viralrecon (nanopore)
0.52 h
0.38 h
−28%
Microsoft Azure Batch vs. Intelligent Compute
Cost per run
Azure Batch
Intelligent Compute
% Change
rnaseq
$35.37
$5.86
−83%
sarek
$28.61
$3.56
−88%
atacseq
$10.89
$1.32
−88%
chipseq
$43.44
$7.52
−83%
nanoseq
$5.21
$0.75
−86%
viralrecon
$24.68
$2.50
−90%
viralrecon (nanopore)
$11.26
$1.20
−89%
CPU Hours
Azure Batch
Intelligent Compute
% Change
rnaseq
279 cpu-h
172 cpu-h
−38%
sarek
172 cpu-h
77 cpu-h
−55%
atacseq
100 cpu-h
38 cpu-h
−62%
chipseq
415 cpu-h
216 cpu-h
−48%
nanoseq
31 cpu-h
25 cpu-h
−19%
viralrecon
86 cpu-h
38 cpu-h
−55%
viralrecon (nanopore)
83 cpu-h
15 cpu-h
−82%
Wall Clock Time
Azure Batch
Intelligent Compute
% Change
rnaseq
3.69 h
3.39 h
−8%
sarek
7.64 h
6.23 h
−18%
atacseq
3.86 h
2.37 h
−39%
chipseq
5.14 h
4.28 h
−17%
nanoseq
2.01 h
1.94 h
−4%
viralrecon
0.87 h
1.08 h
+24%
viralrecon (nanopore)
0.76 h
0.84 h
+11%
GCP vs. Intelligent Compute
Cost per run
GCP Batch
Intelligent Compute
% Change
rnaseq
$14.91
$6.34
−57%
sarek
$6.06
$3.46
−43%
atacseq
$7.05
$1.92
−73%
chipseq
$24.24
$9.68
−60%
nanoseq
$1.91
$0.53
−72%
viralrecon
$13.84
$2.68
−81%
viralrecon (nanopore)
$7.59
$0.97
−87%
CPU Hours
GCP Batch
Intelligent Compute
% Change
rnaseq
310 cpu-h
176 cpu-h
−43%
sarek
113 cpu-h
84 cpu-h
−26%
atacseq
84 cpu-h
36 cpu-h
−57%
chipseq
366 cpu-h
206 cpu-h
−44%
nanoseq
34 cpu-h
23 cpu-h
−32%
viralrecon
58 cpu-h
47 cpu-h
−19%
viralrecon (nanopore)
46 cpu-h
13 cpu-h
−71%
Wall Clock Time
Azure Batch
Intelligent Compute
% Change
rnaseq
5.34 h
3.69 h
−31%
sarek
10.73 h
7.11 h
−34%
atacseq
3.83 h
2.15 h
−44%
chipseq
5.64 h
2.15 h
−30%
nanoseq
2.30 h
1.91 h
−17%
viralrecon
3.39 h
1.05 h
−69%
viralrecon (nanopore)
3.27 h
0.89 h
−73%
What This Means for Your Pipelines
These benchmarks reflect what changes day-to-day: no more guessing at instance sizes, no retry scripts to write and maintain, and no manual intervention when spot instances get preempted. That work happens automatically at the compute layer, freeing up time to focus on the science rather than the infrastructure. The result is more experiments completed for the same budget and faster time to results, and because Intelligent Compute learns from every run, it keeps getting better the more a pipeline is used.
See what smarter compute can do for your organizationIntelligent Compute achieves levels of optimization that would take teams months of manual tuning to approach and allows teams to scale and run workflows in the most efficient way.
