Why we ran Nextflow Frontier on BioML-bench: to show it respects its contract, to test our infrastructure under load, and to train our forward deployed scientists.
“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” Charles Goodhart
Nextflow Frontier delivered top results (or the statistical equivalent) on 17 of the 24 tasks of BioML-bench, and the 7 others are still running. On average across tasks, it does better than 92% of leaderboard entries, where the best of the other agentic systems does better than 75%.
We benchmarked Nextflow Frontier for three purposes: showing that it respects its contract, testing our infrastructure under load, and training our forward deployed scientists.
Showing Frontier respects its contract
Science has always been about asking the right questions.
Nextflow Frontier’s promise is that it is the best system at solving a problem. By problem, we mean an assembly of three things: resources (papers, publicly available data, tools…), a success metric, and a gold standard (a held-out dataset that includes a validation set, and - optionally - a blind test set).
We chose BioML-bench because, when we decided to run this campaign, it was the third-party evaluation set that best fit these criteria: it defines problems the way Nextflow Frontier expects them, other AI systems have already been compared on it, and it covers a wide enough range of problems.
Each task comes with a training set and a test set. Within those rules, Nextflow Frontier can use any other resource it finds. On a drug-metabolism task, for example, it found rat liver measurements in the same public study and used them to better predict how quickly the human liver breaks down a drug.
On the EGFR task, for instance, we gave Nextflow Frontier 496 chemical structures with their laboratory results, then asked it to rank 144 unseen molecules for activity against EGFR, a protein involved in cancer growth. Nextflow Frontier tried 92 ideas and ran 820 experiments over 80 hours. It ended up level with the best entry on the leaderboard (a billion-parameter neural network), using two light-weight chemistry models. Interestingly, it combined them using a statistics paper from 1986 on blending two models.
A good score does not mean Nextflow Frontier solved the scientific question behind a task. The findings are only as good as the question that is asked, and asking the right question is hard (it needs scientific rigor, the right negative controls, the right test sets…).
Many of the benchmarks we tested also have flaws in their design, which meant we had to check Nextflow Frontier’s own work: we retrained its models with cross-validation, re-implemented published solutions to compare against, made sure the science it created was sound, and went through trillions of tokens of agent traces looking for reward hacking.
And we did find some: early in the campaign, a few agents figured out that submitting a constant prediction or silently falling back on default inputs, was enough to get a result past our checks. We added controls for this and a variety of other workarounds into Frontier.
This is also why we built Frontier to bubble up evidence on whether a question is properly asked (is there leakage, missing data, noisy data?). That behavior is harder to measure on benchmarks, but it makes the difference in the real world.
When ALLOX ran Nextflow Frontier, it took several runs before we designed the right problem. Once we had it, the results were much more useful, and features that intuitively made sense became predictive.
“We gave Nextflow Frontier one of the hardest problems in our field: predicting how mutations change protein binding affinity. On our own held-out data, the model it built outperformed every published method we tested, including recent deep learning approaches. The scale of experimentation it enables changes what’s possible for a company like ours.”
André Faure
CTO & Co-founder, ALLOX
So our benchmark shows that Nextflow Frontier respects its most basic contract: to our knowledge, it is the best system in the world at solving a task given resources, a success metric, and a gold standard it can score itself against. Whether solving that task also means making scientific progress is another question, one where Nextflow Frontier works hand in hand with scientists, through a user experience built for trust and on a platform built for performance, reproducibility and rigor.
Testing our infrastructure
We ran the BioML Benchmark campaign over two weeks. During that time, Nextflow Frontier ran over 100,000 Nextflow pipelines across regions and cloud providers (peaking at over 9,000 in a single day), built over 12,000 container images adding up to hundreds of terabytes, and used trillions of LLM tokens. Running at that scale surfaced edge cases and errors we had never seen before, and fixing them improved every component of Frontier, including parts of Seqera’s current products.
We also developed a heavily locked-down version of Nextflow Frontier for customers who need it, fully cut off from the outside network: no proxy, no direct or indirect connection to the outside. We care about this because of what happened at OpenAI in July, when agents under evaluation escaped their sandbox through a package registry proxy and broke into Hugging Face to get the answers to the benchmark they were being tested on. Combined with Seqera’s experience building GxP-validated infrastructure, this means Frontier is ready to run in the strictest environments, which we believe is a requirement to contribute to the problems that matter most.
Testing ourselves
During those two weeks, Nextflow Frontier was operated by Seqera’s computational biology team, the same team that supports our customers every day. Nextflow Frontier explored on its own while the team guided the loop, and this is how our forward deployed scientists learned when Nextflow Frontier fails, when it reward hacks, and how to give feedback that improves its output. It is also how we built up the core piece of the system: the interface between the scientist’s brain and the machine.
We expect Nextflow Frontier to change the way scientists work, so our team have had to go through that transition first, before accompanying our customers through it.
We will share the results of the 7 remaining benchmarks in the coming weeks, and expect to continue to throw Frontier at interesting benchmarks and challenges as a way of refining the platform and - frankly - showcasing its rather astonishing power. That said, we expect most of our focus to now turn to solving our customers’ hardest problems, hand in hand with them. Benchmarks are fun, but it’s much more fun to work on things that truly matter, but are harder to measure - such as helping scientists ask better questions.