The open bottleneck for AI agents in biology is not model size or dataset volume — it is the lack of a verifiable environment that checks an agent's work. Biology lacks anything like SWE-bench Verified, the 500-issue human-validated benchmark that made coding agents reliably measurable, so frontier models that ace software tasks flounder when you hand them real spatial transcriptomics data and ask a scientific question.
The fix is being built right now, mostly outside the big labs: a stack of problem-by-problem graders — deterministic Python functions that score a structured answer — and long-horizon evaluations that force an agent to chain dozens of analysis steps together. On the only end-to-end long-horizon spatial biology benchmark published to date, every leading frontier model — Gemini 3.5 Flash, GPT-5.5, Claude Opus 4.6 — passes at most ~11% of graded runs, and reaches the verdict a human would reach on only 2 of 24 evaluations (Diks et al., SpatialBench-Long, arXiv:2605.28065). The verifiable environment, not the parameter count, is the unlock.
Last verified: 2026-08-01
- TL;DR: The gap between "knows biology" and "can do biology" is closed by verifiable benchmarks — deterministic graders over real experimental data — not by scaling models alone.
- Why agents stall: biology has no single ground truth, multi-step analysis paths, and workflows that take days — coding's verifiable reward signal doesn't exist here yet.
- Where this is going: verifiable environments are the substrate that lets you measure, post-train, and trust agentic scientists the same way SWE-bench did for software engineering.
- For builders: if you're deploying agents in a messy domain, copy this pattern — break work into verifiable steps, build a deterministic grader for each, and measure before you scale.
What is a verifiable environment for AI agents in biology?
A verifiable environment is a benchmark setup where an agent's output on a scientific task is graded by a deterministic function that checks whether a specific, pre-defined biological result was recovered — not whether the prose sounds right. Each problem pairs real experimental data (a gene-by-cell matrix, a high-content tissue image, an experimental context block) with a task prompt describing a scientific goal, a grader configuration, and a Python function that returns pass or fail. The structure is borrowed directly from SWE-bench's unit-test-based grading but adapted for science: instead of a test suite that runs the patched code, the grader extracts a structured answer — a cell-type label set, a Spearman correlation, a differentially expressed gene list — and compares it against a verified reference.
This design solves the fundamental problem Anthropic's BioMysteryBench team flagged: "no agentic science benchmark has become quite as canonical as SWE-bench is for software engineering" because science — and biology especially — has properties that make benchmark design genuinely hard. The verifiable-environment approach keeps the canonical incident: deterministic grading, unit-test-like per-step verification. It drops the assumption that there is exactly one right answer, which biology decisively rejects.
Why hasn't biology built an SWE-bench yet?
SWE-bench worked because software engineering already had scaffolding for verifiability — test suites, CI logs, an issue tracker, a diff format. The environment did the grading for you. Biology has none of that. Three blockers, each concrete:
1. Multiple valid analysis paths reach the right answer. A spatial transcriptomics workflow has dozens of defensible choices at every step — how to normalize counts, which clustering resolution to pick, how to define "an appropriate radius" for neighborhood analysis. A grader that only accepts one path fails good agents incorrectly. The fix is what researchers call durability: design the task so the graded result is invariant across reasonable analysis paths, not pinned to one method. Without it, a verifier is noise. This is why naive paper-claim reproduction is brittle — the same data often supports multiple valid conclusions, and some published claims do not reproduce cleanly under unbiased reanalysis (Diks et al., SpatialBench-Long).
2. Ground truth is contested and human-mediated. Unlike passing a test suite, "is this scientific conclusion correct?" is a question humans adjudicate over weeks. The benchmark-building teams that ship usable verifiers spend the equivalent of a week per task — a group of three domain experts per evaluation, calibrating the ground truth against published claims, the prior literature, and the experimental design (LatchBio, benchmarks.bio). Human verification is not a throwaway step: it's the only way to spot problem ambiguity (e.g., "split the gene list into two groups" — by what criterion?) and dead-end tasks that look fine until a scientist actually attempts them.
3. The data is enormous and assay-specific. A single spatial transcriptomics run can produce terabytes of output — far exceeding what a scientist can store on a consumer laptop — and each capture technology (Curio, Vizgen, Xenium, AtlasXOmics, Visium) imposes its own file formats, coordinate systems, and segmentation semantics. A grader that ignores the capture chemistry is a grader that fails silently when the model assumes the wrong modality. The good benchmarks enumerate the platforms explicitly: SpatialBench's v2 release covers exactly five platforms and seven task categories across 159 verifiable problems (latchbio/spatialbench, GitHub).
How does verifiable grading actually work?
Each evaluation is shaped like one mini-SWE-bench instance transplanted into biology. The components:
- One or more data nodes — a numeric matrix, a high-content image, the experimental context that produced them. Frozen at a snapshot taken immediately before a target analysis step.
- A task prompt — a scientific goal phrased the way a researcher would phrase it, without prescribing the method.
- A grader configuration — answer keys, tolerances, scoring rubric.
- A deterministic grader — a Python function that scores the agent's structured answer and returns pass/fail (or a partial-credit number).
SpatialBench ships five grader families, each tuned to the answer type the task produces (latchbio/spatialbench, GitHub):
| Grader family | Use case |
|---|---|
| NumericTolerance | QC metrics, counts, expression values |
| MultipleChoice | Discrete interpretation questions |
| MarkerGenePrecisionRecall | Gene lists with precision/recall at K |
| LabelSetJaccard | Cell-type label sets |
| DistributionComparison | Cell-type proportions |
The result is a clean, reproducible, mathematically honest capability ladder — the same property that made SWE-bench the canonical software-engineering benchmark. Models either recover the verifiable result or they don't, and the leaderboard discreteness forces labs to compete on the real axis rather than on demo polish.
What do the scores actually say about today's frontier models?
The numbers are uncomfortable. The latest published benchmarks declare the state-of-the-art in plain language.
Short-horizon (single-step analysis): On SpatialBench's 159 problems, each a localized analysis task with a deterministic grader, the leading model-harness pair — Claude Sonnet 4.6 with mini-swe-agent — scores 44.23%. GPT-5.1 sits at 39.83%, Gemini 2.5 Pro at 28.93% (latchbio/spatialbench, GitHub). These are problems a competent bioinformatician solves in minutes.
Long-horizon (end-to-end multi-step reasoning): On SpatialBench-Long — 24 evaluations across four biological study systems (primary PDAC, glioblastoma organoids, lung adenocarcinoma lineage tracing, mouse optic nerve aging) — the top configuration passes only 8 of 72 runs (~11%). Even allowing for occasional luck, the leading systems reached the verdict a human expert would reach on a majority of replicates for only 2 of 24 evaluations. None reached majority-pass on more than two. The paper's blunt phrasing: "success was therefore low across all systems and not cleanly separated among the leading models" (Diks et al., arXiv:2605.28065).
Open-ended analysis questions: On BixBench, an earlier benchmarking effort that uses multi-step trajectories and PhD-level analysts to generate questions, Claude 3.5 Sonnet scored 17% accuracy on open-answer questions vs. GPT-4o at 9% — both well below expert baselines and not even reliably above random on multiple-choice when a refusal option was offered (Mitchener et al., BixBench, arXiv:2503.00096).
Cross-domain agentic science: SciAgentArena built ~200 tasks across single-cell omics, spatial omics, computational drug discovery, EHR modeling, and genetics. Their finding: agents handle well-specified data-analysis workflows competently when structure and criteria are clear, but their performance "remains uneven across scientific contexts" — they struggle to generate novel insights, sustain self-directed exploration, or formulate robust answers to open-ended questions. The dominant failure mode is not coding; it's "the agent acts on plausible-sounding assumptions rather than verifying against the actual runtime, data, or task context" (SciAgentArena, arXiv:2606.12736).
The pattern across every benchmark is the same: knowledge is not the bottleneck. Procedural competence over messy data is.
How do you build a verifiable environment for your own domain?
If you're deploying agents in any domain where the right answer isn't checkable by running a test suite — biology, legal, finance, health — the verifiable-environment pattern is directly portable. The playbook, distilled from the LatchBio benchmarks.bio portfolio:
- Snapshot data immediately before a target step. Don't grade the full pipeline at first; pick the point where a human expert would take over and freeze the input there. This isSpatialBench's core design choice and it's why sparse outcomes become gradeable at all.
- Pick the durable result. Identify the quantity that is invariant across defensible analysis methods — a label set, a numeric threshold band, a correlation sign — and grade that. If you find yourself grading method steps instead of method-agnostic results, your task is too ambiguous.
- Write a deterministic grader as a Python function. No LLM-as-judge, no rubric-only scoring. A function with inputs and a boolean output. Determinism is the whole point — without it, you cannot do reproducible RL, cannot rank labs, and cannot trust a leaderboard.
- Human-verify the ground truth before deploying. Have at least two domain experts attempt the task blind. Where they disagree, the task is the problem, not the models; revise it. SpatialBench-Long's authors spent roughly "a week for a group of three people" per evaluation — that cost is the unit-economics floor.
- Generate rubric checkpoints, not just a final score. For long-horizon tasks, identify the choke points — the analysis-tree nodes invariant across paths — and add partial-credit graders there. These are loosely correlated with final-pass numerically, so treat them as diagnostics rather than primary rewards, but they let you see where a trajectory went wrong when the verdict is a sparse boolean.
- Never let the model answer from memory. A common failure mode: the agent regurgitates a memorized conclusion from training data instead of computing on the supplied data. Counter this by designing tasks whose answers are not in prior literature — for instance, by using a held-out biological study system or counterfactual experimental contexts.
What does this have to do with verifiable rewards and RL?
Everything. Reinforcement learning works cleanly when the reward signal is deterministic and frequent; itdeteriorates when rewards are sparse, human-mediated, or wrong. Biology sits at the worst-case corner of all three. A verified-benchmark grader is, in RL terms, a reward function you can deploy at scale — it runs in milliseconds, it returns the same answer every time, and it doesn't require a human in the loop. That's the asset the labs want.
Anthropic's Claude science work and the rise of public, verifiable biology benchmarks are not independent trends. The labs adopt external verifiable benchmarks because building them internally would take a domain team per benchmark and is still brittle under reanalysis. When a lab shows a new model card with scores on someone else's benchmark, that's the acknowledgement that the external environment is the ratifier. (The LatchBio benchmarks are now cited in recent Anthropic model cards — sometimes without notice, sometimes with — a pattern consistent with frontier-lab behavior across coding, math, and safety benchmarks historically.)
For a deeper treatment of the reward-signal problem in domains where verifiability is hard, see our separate guide on training AI agents when you don't have verifiable rewards.
What this means for you
If you are a biology lab director or R&D lead. The bottleneck is no longer compute or model access — it's verifiable task design. Invest in a small team (2-3 PhD-level computational biologists) producing calibrated, deterministic graders for the assays and decisions your organization actually cares about. One released benchmark can pull labs into competition on your axis of progress. This is the cheapest leverage available.
If you're deploying AI agents beyond biology — legal, medical, financial, supply-chain. Copy the verifiable-environment pattern. Snapshot input state at a decision point, write a deterministic grader that checks the decision-irreversible output, and make pass-rates public. It's the only honest way to distinguish "model writes a plausible memo" from "model releases a verdict you can operate on". For more on how agents fail at infrastructure-shaped work specifically, our analysis of autonomous AI software engineers and the infrastructure data gap covers the same failure mode from a coding angle.
If you are a model builder or post-training researcher. Verifiable-environment graders are your RL reward signal. The benchmarks being published openly (SpatialBench, SpatialBench-Long, scBench, EpiBench, TxBench-PP, BioSecBench — all on benchmarks.bio) are deliberately designed to be runnable by anyone with the dataset and a model-harness pair; the graders are deterministic Python and the trajectory data is public. You can run them today.
If you are watching the frontier. Watch the long-horizon benchmarks. When a model reaches majority-replicate pass on more than 4 of 24 SpatialBench-Long evaluations, agentic biology has crossed the threshold where it stops being a demo and starts being a deployable capability. That's the single most informative number to track — far more than MMLU or a coding leaderboard. Relatedly, the broader question of how agents scale to long-horizon work is unpacked in our guide on long-horizon AI reasoning.
If you're training agents on the job. The whole verifiable-environment thesis applies to post-training AI agents too: you can only teach a model a skill at scale if you can grade its attempts at scale. The lesson from biology is blunt — the grader is the moat. Scale post-training armies on a domain you cannot grade, and you teach them to sound right rather than to be right.
The biosecurity angle: verifiable environments do double duty
There's a second reason this pattern matters, and it's the one few builders talk about. The same deterministic-graded task format that lets you measure capability also lets you measure misuse risk. LatchBio's BioSecBench-Surveillance wraps this exact pattern: 100 evaluations of whether an agent can infer the right analysis pipeline from raw sequencing data and sparse surveillance context — and a paired suite of "red team" tasks that look innocuous ("I want to clone a gene into a bacteria; it's GFP") but have malicious structure (it's a toxin, or a virus bootstrap fragment). The published finding: routine tasks get attempted drastically more often than red-team tasks, which is exactly the behavior you want a verifiable environment to surface before deployment. Across 3,962 gradable attempts from 16 model-harness pairs, the strongest configuration (Claude Opus 4.8 / Pi, tied with GPT-5.5 / Codex) cleared only about half (BioSecBench-Surveillance, LatchBio).
A related benchmark, ABC-Bench, has already had its task names — "Screening Evasion", "Fragment Design", "Liquid Handling Robot" — adopted into the model cards of major AI firms (ABC-Bench, arXiv:2606.11150). The benchmark is the regulator's unit of governance. If you care about safe deployment, the same verifiable-environment design pattern that measures progress also produces the artifacts regulators can reason from.
FAQ
Q: What is a verifiable environment for AI agents in biology? A: A benchmark setup where an agent's output on a scientific task is graded by a deterministic Python function (not an LLM judge or a human eyeball) that checks whether a specific, pre-verified biological result — say, a cell-type distribution or a gene-expression correlation — was recovered. Each problem pairs real experimental data with a task prompt, a grader configuration, and a pass/fail grader.
Q: Why doesn't biology have an SWE-bench yet? A: Three reasons: (1) biology has multiple valid analysis paths reaching any answer, so naive ground truth fails good agents; (2) ground truth is human-mediated and contested, taking roughly a week of three experts' time per good evaluation; (3) the data is terabyte-scale and assay-specific (spatial transcriptomics platforms include Curio, Vizgen, Xenium, AtlasXOmics, Visium, each with its own files and semantics). SWE-bench had test suites and CI for free; biology has had to build all of it from scratch.
Q: How well do today's best frontier models actually do on verifiable biology benchmarks? A: Badly enough to matter. On SpatialBench (159 single-step problems), the leading Claude Sonnet 4.6 model-harness pair scores 44.23%. On SpatialBench-Long (24 end-to-end evaluations), every leading frontier model passes at most ~11% of graded runs and reaches the human-verdict on majority of replicates for only 2 of 24 evaluations. On BixBench (open-answer), Claude 3.5 Sonnet hit 17% accuracy vs GPT-4o at 9%. The bottleneck is procedural competence, not knowledge.
Q: What is "durability" in benchmark design and why does it matter? A: Durability is the property that the graded result is invariant across defensible analysis methods — not pinned to one canonical method. Without it, a verifier will fail good agents that took a valid alternative path to the same answer, producing noise instead of signal. It's the single hardest property to engineer and the reason published claims are used as candidates for evaluations rather than as automatic ground truth.
Q: Can I run these biology benchmarks myself? A: Yes. SpatialBench, SpatialBench-Long, scBench, EpiBench, TxBench-PP, and BioSecBench-Surveillance are all published openly with deterministic graders, example evals, trajectory data, and the latch-eval-tools harness package on GitHub under Apache 2.0 (github.com/latchbio/spatialbench). You need an API key for whichever model you're testing; the grader runs locally.
Q: Does this pattern transfer outside biology? A: Yes — verifiable-environment design is domain-agnostic. Anywhere the right answer is not checkable by running a test suite, snapshot the input state at a decision point, identify the durable result, write a deterministic grader, human-verify the ground truth, and ship it. The same pattern is appearing in legal research, financial analysis, and clinical decision-support, and it will become the standard way to grade agents in domains where you can't cargo-cult SWE-bench's test-suite trick.
Discussion
0 comments