Making Agentic Data Analysis Reproducible and Scalable with Nextflow

Nextflow was designed to make scientific analysis reproducible, portable and scalable. Those properties are why it is now used to run thousands of pipelines across genomics and other fields including protein analysis, imaging, drug discovery, and regulated environments. That demand is accelerating. At the end of 2024, teams were running around 1.2 million Nextflow pipeline runs a month. In September 2026, that figure passed 6 million, a fivefold increase in under two years as AI has become part of how scientists write and run analyses.

The next step is bringing AI into the pipeline itself. We are now extending Nextflow to cover a new kind of compute. Nextflow now has a native agent construct: a pipeline step where an AI model decides at runtime which actions to take. Agent steps run under the same engine, and with the same guarantees, as every other step in the pipeline.

Nextflow monthly runs: At the end of 2024, teams were running around 1.2 million Nextflow pipeline runs a month. In September 2026, that figure passed 6 million

Figure 1. Graph showing Nextflow monthly runs: ~1.2 million Nextflow runs before the AI era, scaling to over 6 million runs in September 2026.

Why Agents Need What Workflow Systems Already Provide

AI agents are starting to be used in data analysis. An agent does not follow a fixed sequence of steps. It reads an instruction, chooses which operations to run, checks the intermediate results and adjusts what it does next. In genomics this approach is called agentic genomics. The agent picks from a library of validated tools and skills and is guided by instructions written in plain language, not by a plan written in advance.

Agents are useful for tasks that are hard to express as fixed rules. Examples include cleaning inconsistent sample metadata, triaging samples, interpreting quality reports and deciding when a result needs a second look. We believe they will become a common part of data pipelines.

Results produced by an agent still have to meet the same standards as any other analysis:

→Reproducibility: it must be possible to record and replay what ran, with which inputs.
→Scalability: a real study can involve thousands of samples, which is more than one agent session can handle.
→Portability: the analysis must run on a laptop, an HPC cluster or in the cloud, and sensitive data must be able to stay where it is.
→Observability and lineage: reviewers need to be able to see afterwards what the agent did.

Workflow systems were built to solve these problems. Most agent tools, however, run outside the workflow engine. Each one has its own way of running, retrying and logging work. Typically, a separate harness calls into the analysis over MCP or an API, with its own provenance, retries and concurrency, and no shared work directory with the rest of the pipeline. As a result, the guarantees that pipelines already provide either have to be built again for each tool or are missing.

The workarounds are costly. Wrapping an agent in a process by hand takes a lot of boilerplate: integrating the harness, caching on only the right inputs, validating output against a schema and recording lineage. And a single agent working through one long context does not scale to real workloads. Splitting the work across parallel agents and combining their results means building the parallelism and fan-in yourself.

An Agent Primitive for Nextflow

Nextflow already provides these guarantees for regular, deterministic tasks through containers, caching, support for many execution platforms, and lineage tracking. Agentic tasks need the same things, and they map directly: replaying an agent's decisions is caching, auditing them is lineage, distributing skills is module distribution, and running local-first is portability. We therefore did not build a separate system for agents. An agent is a step in the workflow: a node in the workflow graph, executed by the same engine as every other task.

An agent is declared next to a process in the same pipeline. A process runs a script that you write. An agent receives an instruction and works out how to carry it out. Both take inputs, produce outputs and are connected through channels.

Nextflow's dataflow model is a natural fit for agents. A node waits for its inputs and fires when values arrive. That means mapping an agent over a channel in parallel, or fanning its results back in with collect(), needs no separate orchestration layer.

Example: labelling sample metadata

In the following example, a process fetches the raw metadata of each sample from a public archive. An agent then reads each record and assigns standard labels for organism, tissue, disease and assay, using a vocabulary provided as a skill:

process FETCH_METADATA {
    input:
    accession: String

    output:
    stdout()

    script:
    """
    curl -s "https://www.ebi.ac.uk/ena/portal/api/filereport?accession=${accession}&result=read_run&format=json"
    """
}

agent CLASSIFY {
    model 'openai/gpt-5-mini'
    instruction 'You are a metadata curator. Label each run using the ontology skill. When the evidence is missing, answer "unknown" rather than guess.'
    skills 'metadata-ontology'

    input:
    meta: String

    output:
    labels: SampleLabels

    prompt:
    "Label this sequencing run: ${meta}"
}

record SampleLabels {
    accession:  String
    organism:   String
    tissue:     String
    disease:    String
    assay:      String
    confidence: Float
    rationale:  String
}

workflow {
    accessions = channel.of('SRR3192396', 'SRR031708', 'SRR891268')
    meta = FETCH_METADATA(accessions)
    CLASSIFY(meta).view()
}

The workflow block contains nothing specific to agents. Samples are processed in parallel, with one agent task per sample, in the same way as for a process. The agent returns a structured record (SampleLabels) that is checked against its declared fields, so later steps can rely on its shape. When the pipeline is run again with -resume, previous answers are reused and only the steps that changed are run again. Each agent run is recorded in the pipeline lineage, along with the model, instructions and skills it used.

The pattern extends naturally. A second agent can collect() the labelled records, review the cohort as a whole, and split confident rows from those that go to a review queue for a person to check. Because the vocabulary lives in a skill, an institution can swap in its own ontology without editing the pipeline.

Tools: real modules, not runtime discovery

Agents can call other Nextflow modules as tools. When the agent calls a module, Nextflow runs it as a normal task: in its container, on the configured executor and compute platform, in its own work directory, and with its own cache entry and lineage record. The agent can only call the tools listed in its definition, so its options are limited to a set that a person has reviewed. Tools are declared, not discovered at runtime.

Non-determinism is bounded, not hidden

A model's output can vary from one call to the next, so the agent construct constrains it at every level. Tools are declared rather than discovered, skills constrain how the agent reasons, and outputs are checked against a declared structure. When a pipeline is resumed, -resume replays the stored generation instead of calling the model again. Reruns are therefore reproducible, and they cost no tokens.

Configuration and portability

Agents are configured in the same way as processes, so they can run on any platform Nextflow supports without changes to the pipeline:

agent {
    model    = 'openai/gpt-5-mini'
    apiKey   = secrets.LLM_KEY
    executor = 'k8s'
    cpus     = 1
}

The same pipeline can run on a local cluster or a cloud batch service. The model provider can be changed in the configuration in the same way. The agent runner is a plugin, configured like an executor. You can swap a hosted model for a local one without changing pipeline code.

An agent can also be packaged as a module together with its skills and tools, so it can be shared, versioned and reused in other pipelines like any other Nextflow component. Modules are the agent's tools, skills are versioned bundles that travel with the agent, and the agent itself is a module you include, alias and configure. Agents and their skills will soon be publishable and shareable through the Nextflow Registry.

One Engine for Deterministic and Agentic Compute

Agents are becoming part of everyday data analysis, and they need the same reproducibility, scalability, portability and lineage as other pipeline steps. The agent construct brings both kinds of compute into Nextflow. The aim is to use a model only for the steps that need judgement, and to keep the rest of the pipeline as deterministic and reproducible as it is today.

Benefit	What it means
One pipeline, one run	Deterministic steps and agentic steps are orchestrated together, in the same pipeline and the same run.
Same guarantees	Both get containers, a work directory, retries, caching and resume, lineage tracking and checked outputs.
Token-free reruns	Resuming a pipeline replays stored generations instead of calling the model again, so reruns are reproducible and cost no tokens.
Reviewed tools	Tools are explicitly declared, not discovered at runtime.
Built-in parallelism	Agents compose like processes through channels, so running agents in parallel and combining their results already works.
Any platform, any mode	Agents are configured, run and scaled on every platform Nextflow supports. The runner is a plugin, configured like an executor, so a hosted model can be swapped for a local one without changing pipeline code.
Shareable modules	Agents and their skills package as modules, versioned and shared across pipelines.
Familiarity	Nextflow developers keep working in the language and tools they already use, with one additional kind of step.

New to Nextflow?Nextflow is the leading open-source workflow orchestrator that simplifies writing and deploying compute and data-intensive pipelines at scale on any infrastructure.

Learn about Nextflow