7 tips for reducing your Nextflow cloud spend

For organizations operating in the cloud, managing spending is an ongoing concern. In Flexera’s annual State of the Cloud Report for 2023, 82% of survey respondents cited managing cloud spending as their #1 cloud-related challenge.¹ According to similar research from Forrester, healthcare organizations spend an average of $9.5M annually on cloud services.² A significant portion of this spending is attributable to genomics analysis.

This article provides an updated look at best practices for managing cloud spending with Nextflow and the Seqera platform.

1. Understand your run rate spending

If you can access your cloud billing through your cloud provider console, getting a handle on your current spending is a good place to start. Get familiar with your cloud provider's billing and cost management dashboard and understand your monthly costs related to compute, managed services, storage costs, bandwidth, and support costs.

For most organizations, compute costs are their largest single line item. By understanding your spending, you will be able to better identify cost anomalies and areas where you may be overspending. Tools such as AWS Cost Explorer, Microsoft Cost Management, and Google Cost Management can be helpful. If your cloud platform supports Cost Anomaly Detection or similar services, set up a cost monitor to ensure that you or appropriate people in your organization are alerted in case of cost overruns.

2. Maximize use of spot instances

Most Nextflow users are aware of the benefits of Spot or preemptible instances. Nextflow pipelines easily tolerate spot instance reclamation, so using these lower-cost instance types only makes sense. All major cloud batch services (AWS Batch, Azure Batch, and Google Cloud Batch) support spot (or similar) instance types.

An advantage of using Seqera is that AWS Batch environments can be configured automatically via Seqera Forge. This ensures that spot instances are configured optimally, taking the guesswork and risk of error out of the configuration.

While AWS and other cloud providers claim savings of up to 90% using spot instances, savings of between 50-70% are more typical, depending on demand, instance types, and the cloud region. Becoming familiar with AWS tools such as the spot instance advisor to monitor spot placement scores and your spot instance pricing history (up to three months) is a good idea. Average discounts can vary by instance type, region, and availability zone. Seqera users can decide what types of instances to use in their batch environments, and what regions to run in, so understanding these differences in average discounts can be helpful.

For pipelines that use GPUs, it might be worth looking at these carefully and treating them as a special case. GPU instances are expensive and in high demand for applications like crypto mining, inference, and AI model training. As a result, discounts may be minimal, and reclaim rates may be high. It may be more practical to use on-demand instances for long-running GPU workloads.

3. Use the Resource Optimizer

Getting resource requirements configured optimally right can be tedious. Resource needs vary based on the size and complexity of datasets. The default settings in many pipelines err on the side of caution and tend to over-provision CPU and memory resources to avoid runtime errors. For studies involving large numbers of samples, it can be advantageous to use Seqera’s built-in Resource Optimizer to generate an optimized set of process-level resource requirements and apply these to your nextflow.config file. In production-scale workloads, this type of resource optimization has demonstrated that pipeline costs can be reduced by up to 87%.³

4. Use Nextflow resume

Avoiding unnecessary computation is an obvious way to reduce cloud expenses. A core feature of Nextflow is the ability to cache task executions and reuse them in subsequent runs to minimize duplicate work. This feature is useful when recovering from errors and when iteratively developing pipelines. This functionality is similar to checkpointing, a common practice in HPC applications.

Resume functionality can be triggered via the command line, but the Seqera interface makes this easy for users. Long-running pipelines can be run from where they left off by resuming workflows after correcting underlying issues. This makes users more productive and dramatically reduces resource use in the cloud.

See also: Demystifying Nextflow resume

5. Pay attention to storage costs

For most organizations, storage is their second largest cloud-related expense after compute services. Nextflow supports multiple types of storage, including object stores, cloud-native file systems, and high-performance parallel file systems such as FSx for Lustre. This enables users to select their preferred technology and price point depending on their business needs. Most organizations opt for object stores such as Amazon S3, Azure Blobs, or Google Cloud storage. Object stores can be an order of magnitude less expensive than shared file systems, and are transparently supported by Nextflow. When selecting storage options, make sure you pay attention to the following:

• Select the right type of storage – cloud providers typically offer different classes of storage with different price, performance, tiering, and retrieval characteristics.
• Leverage cloud lifecycle management features for data that remains in the cloud to automatically move infrequently used data to lower cost cold storage.
• Be aware that even when using low-cost object storage, pipelines may still need block storage attached to instances, which will increase storage costs.
• Regularly review and clean your storage — outdated backups, unnecessary snapshots, or unnecessary duplicates of files.

6. Use the Fusion file system

One of the best ways to reduce storage costs is to use the Fusion file system. The Fusion file system leverages your preferred cloud object store and provides additional benefits above and beyond the optimizations suggested above.

Fusion avoids the need for block storage attached to local instances, delivers performance on par with high-performance parallel file systems, and reduces overall storage requirements, resulting in savings of up to 76% compared to alternative storage solutions.⁴

See Breakthrough performance and cost-efficiency with the new Fusion file system.

7. Track and analyze spending by organization and workload

We’ve all heard the phrase, “If you can’t measure it, you can’t manage it” and this is particularly true when it comes to cost management.

For large organizations, a best practice in managing cloud spending is to use multiple accounts.⁵ By using separate accounts for each business unit, organizations can more easily report, control, forecast and budget cloud expenditures. In AWS, customers can use the AWS Organizations service for consolidated management and reporting across multiple accounts and organizational units (OUs).

Seqera meshes perfectly with this management model by enabling separate workspaces to be mapped to each account. Since compute environments use cloud credentials specific to each account, costs are automatically tagged to the correct business users. Organizations still retain the flexibility to share access to pipelines, compute environments, and data with internal or external users.

In addition, Seqera provides flexible workspace-level tagging of both workloads and resources using Labels and Resource Labels respectively. Tools such as the AWS Tag Editor can be used to manage these labels and activate relevant tags in the AWS Billing and Cost Management console for granular reporting and analysis. Resource Labels in Seqera are supported with AWS, Azure, Google Batch, Google Life Sciences, and Kubernetes.

For additional tips on reducing costs in the cloud, download the eBook Six ways to help slash your cloud computing bill.