If you build AI agents the way you write traditional software, you will ship broken things. The honest verdict from 2026 is that agents are a form of machine learning, and their performance is best treated as a blackbox artifact managed through empirical evaluation — rollouts in environments, not unit tests on deterministic functions. Every company that uses computers should be building its own agent eval today, because the moment you have one, you can invite every model to compete on your terms and stop trusting vendor benchmarks.
Last verified: 2026-07-30 · Volatile facts (framework versions, benchmark scores) flagged inline.
TL;DR
- Agents are non-deterministic by design: run the same agent twice and you can get two valid but different paths. Deterministic unit tests can't catch that.
- The right mental model borrows from ML: training data → environments, model weights → skills/prompts/tools, loss function → environment rewards, gradient step → a pull request.
- An environment is three pieces: an instruction, a sandbox (virtual computer), and a verifier (programmatic tests or rubric that grades the outcome).
- A rollout is one run of the agent through an environment; aggregate reward across many to get a reliable eval signal the way you'd average across a validation set.
- Frameworks like Harbor (Apache-2.0, by the Terminal-Bench team at Laude Institute) standardize this so environments are interoperable between teams.
Why is building an AI agent more like machine learning than software engineering?
In 2018 software engineering meant you knew what the code would do before you ran it. A regex that extracts phone numbers from text will print the same output on the same input a million times in a row — you could review a 400-line pull request by reading it, because the behavior was fully specified by the code. That era is ending. Swap the regex for a model call ("extract any phone number from this text") and the program gets more robust at handling weird formats, but you can no longer say with 100% confidence what it will print across a million runs. The uncertainty only compounds as task complexity grows. This is a confirmed shift others have documented well: agent evaluation has to grade the trajectory the agent takes, not just the final answer, because a correct answer reached through the wrong tool calls is a latent incident wearing a green checkmark.
The mental shift, articulated by François Chollet in May 2026, is that "agentic coding is a form of machine learning. Generated code is best treated as a blackbox artifact whose behavior and generalization should be managed via empirical evaluation, like with any ML model." (Source: fchollet on X, May 9 2026). Generalize that one level further and the claim holds for all agents, not just coding ones: agent performance itself is a blackbox artifact.
The ML-to-agent mapping
| Machine learning concept | Agent-building equivalent | What changes |
|---|---|---|
| Training data | Environments (the tasks you practice on) | Tasks live in sandboxes, not static datasets |
| Test / validation set | Eval environments (environments you don't train on) | Same format as training — just held out |
| Model weights | Skills, prompts, tools, model choice | The "parameters" you tune to improve the agent |
| Loss function | Environment rewards and feedback | Rewards come from a verifier, not a closed-form metric |
| Optimizer / backprop | Context-based optimization (running the agent in a loop, or tools like DSPy) | Improvement comes from iterative prompts/PRs, not gradient descent |
| One gradient step | A pull request into your repo | Each iteration is a human- or agent-authored change |
| Overfitting | Reward hacking, or overfitting to a narrow eval | The model games the reward without generalizing |
The payoff of this framing is practical: every tooling idea ML engineering already solved has an agent-development analogue that is only now being built. Databricks-scale companies were created for the ML column; the agent column is even bigger in population.
How do you actually evaluate an agent?
You evaluate an agent the same way you evaluate a machine learning model: run it on a dataset of held-out tasks, score the outcomes, and aggregate. For agents, each "task" is an environment and each "run" is a rollout. An environment is three concrete pieces:
- An instruction — what the agent is supposed to do.
- A sandbox — a virtual computer where the agent tries to do it (a container, a VM, a directory). All modern agents act on computers, so the sandbox gives them a real surface to work on without touching production.
- A verifier — a programmatic test, rubric, or judge that determines whether the agent actually did the thing, within a time or step budget.
A rollout is the full cycle: hand the sandbox to the agent → the agent runs until a stopping condition, producing a trajectory (the sequence of actions it took) → hand the sandbox to the verifier → the verifier returns a reward or set of rewards → stop the sandbox. Aggregate rewards across many rollouts over an eval set and you have a reliable signal — the same way you'd average a metric across a validation set rather than trusting a single run.
This matters because of reliability math: per-step accuracy compounds multiplicatively. An agent that is 95% reliable on each individual step succeeds end-to-end only ~36% of the time across a 20-step chain. A single "it worked once" demo tells you almost nothing; you need dozens or hundreds of rollouts to know what the agent actually does on the distribution of tasks it will face in production.
What is the environment-sandbox-verifier pattern in practice?
The environment-sandbox-verifier layout was popularized by Harbor, an open-source framework (Apache-2.0) from the same team that built Terminal-Bench at the Laude Institute. The current release is v0.16.1 (Zenodo DOI 10.5281/zenodo.20953922). Source: github.com/harbor-framework/harbor.
What Harbor actually provides
- A format for specifying agentic environments as a standardized file directory. The layout has become a de facto common language in the environment space because it lets tasks pass between hands and stay interoperable.
- A framework for performing rollouts in parallel using any agent with any model in any sandbox on any task. This matters operationally: rollouts are slow, and parallelism is the only way to tighten the eval loop fast enough to be useful.
- A registry of eval/training environment sets, so you don't start from zero.
Harbor supports more than the bare-bones loop: multi-step rollouts, verification run in a separate sandbox, artifact collection, and simulated users. But every flavor is built off the same core cycle of agent → trajectory → verifier → reward.
What should your company evaluate?
Satya Nadella's framing on this is the most useful one for business leaders: build the eval that matters to you, then invite all models to show up against it — "you don't have to trust brand, you don't have to trust somebody else's eval, you don't have to trust a public eval." (Paraphrased from a June 2026 interview; the substance is verified via Stratechery's coverage of Nadella's June 4 2026 remarks.) The moment you have your own eval, the power shifts to you: you can balance the cost-performance tradeoff on your own terms, not a vendor's.
There are four concrete categories of agent eval that companies are building today, and at least one applies to almost every business:
| Eval category | What it answers | Who needs it |
|---|---|---|
| Agents build your product | How well do coding agents work on your internal codebase? | Any software company (Ramp's RampBench is the canonical example — they built an internal-codebase eval to pick the coding agent that performs best on their use cases instead of maximizing token spend) |
| Agents use your product | How well do agents operate your product's headless/API mode? | Any product company moving toward agent-friendly APIs (Stripe, HubSpot model): iterate on the product to make it more usable by agents and your product becomes more valuable |
| Agents power product features | How well do the agent-driven features in your product perform? | Companies shipping AI features to end users |
| Agents automate internal processes | How reliably can an agent automate a workflow you currently do by hand? | Literally any company that does anything on a computer |
If you do something on a computer, you should be testing whether AI can automate part or all of it. That automation either increases productivity (raising the value your company generates) or it tells you exactly where you still need people. Either outcome starts with the eval. This is the same logic we unpack in our guide to building an agent benchmark from your own production traces: public leaderboards measure other people's problems; your eval has to measure yours.
How do you turn rollouts into a better agent?
Rollouts aren't just for measuring — they're the raw material for improvement. There are four well-trodden paths from eval data to a better agent, and they map directly onto ML training loops:
Trace review for prompt/agent iteration. Run a batch of rollouts, read the trajectories where the agent failed, find the recurring mistakes, and write the next prompt or PR that fixes them. This is the "gradient step as a pull request" in the ML-to-agent mapping. It's the most accessible path — you can start today with no training infrastructure.
Supervised fine-tuning (SFT) on successful trajectories. Take the trajectories where the agent succeeded, convert them into training data, and fine-tune. The trajectories are now the "training data" half of the environment concept.
Reinforcement learning using trajectory tokens as reward. Take the reward (and the trajectory, tokenized) and use it as the RL signal. Tinker (from Thinking Machines Lab) launched a Harbor integration with a cookbook recipe for exactly this — tinker-cookbook/recipes/harbor_rl. Source: Thinking Machines Lab docs.
Evolutionary / hill-climbing optimization. Take the text feedback from a trajectory and eval and use harnesses like DSPy or Scale to do automated hill climbing on the agent's prompts and tools. This is the agent analogue of hyperparameter search.
The unifying insight: you do not have to pick between "prompt engineering" and "model training." The same rollout data feeds all four. The self-improving loop — rollout → trajectory → feedback → better agent → next rollout — is exactly what we describe in our signal-to-PR playbook for self-improving agent loops: the trajectories are the signal, the PR is the gradient step.
What benchmarks and tools already run on this pattern?
The environment format is the interoperability layer that lets one team's benchmarks run inside another team's infrastructure. A non-exhaustive map of what's live on the Harbor ecosystem as of July 2026:
| Benchmark / tool | Who builds it | What it measures | Primary source |
|---|---|---|---|
| Terminal-Bench 2.0 | Laude Institute (Merrill, Shaw et al.) | 89 real-world software engineering tasks; frontier models score below 65% | tbench.ai, arXiv:2601.11868 |
| Harbor | Laude Institute (same team) | The eval/RL environment framework + registry itself | github.com/harbor-framework/harbor |
| FrontierCode | Cognition AI | Ultra-long-horizon software engineering; three tiers (Extended 150 / Main 100 / Diamond 50); 81% lower false-positive rate vs SWE-Bench Pro; Claude Opus 4.8 leads Diamond at 13.4% | cognition.ai/blog/frontier-code (released June 8 2026) |
| SWE Atlas | Scale AI | 284 tasks across Q&A, test writing, and refactoring; built on SWE-Bench Pro | scale.com/blog/swe-atlas |
| Senior SWE-Bench | Snorkel AI | Under-specified tasks requiring agents to function under ambiguity with behavioral feedback (a taste judge validates) | senior-swe-bench.snorkel.ai |
| Poolside evals | Poolside | All of Poolside's model-training evaluations run on Harbor | Poolside public usage, July 2026 |
⚠️ Note: the agent-eval ecosystem is moving fast and names are easy to confuse. "Frontier-Bench" (a Harbor-repo project) is distinct from Cognition's FrontierCode. SWE Atlas is not the same product as SWE-Bench Pro. And "Terminal-Bench 2.1" is sometimes referenced in talks but the published release is 2.0 — treat any other version number as unverified until you see it on the official source.
What is the agentic map-reduce use case (beyond eval)?
Because a rollout is just "run any agent with any model in any sandbox on any task," the same infrastructure that evaluates agents can also do work at scale. The emerging pattern is agentic map-reduce: distribute thousands of agent runs across parallel sandboxes, then intelligently aggregate. Use cases seen in the wild:
- Scan every trajectory in a batch of rollouts to detect reward hacking.
- Process a large batch of receipts for reimbursement.
- Search semantically across a folder of notes for where you wrote something.
- Analyze a pile of pull requests to answer a question about them.
Harbor ships a feature for this — Harbor Exec — and the pattern generalizes to anything orchestrable. One real example: take every Codex session from a month, run a cheap fast agent (Cursor CLI) on each as the map step to write an analysis.json with the mistake reason and correction, then run an accurate summarizer (Claude Fable 5) as the reduce step to fold those into a concise feedback file. That feedback then seeds the next batch of eval tasks — closing the loop from "what did the agent do wrong" to "here is the eval that catches it next time."
What this means for you
- If you lead a team: the fastest leverage move in 2026 is to build one eval environment that captures a real task your team does by hand. Pick the easiest task to sandbox and grade, not the most impressive. The moment it works, add two more. Your eval is your defensible ground — not which model you happened to license.
- If you're a developer: start with one Harbor environment (or any rollout harness) and run 20–50 rollouts of your current agent on it. The recurring failure modes you find in the trajectories will be a better to-do list than anything a benchmark leaderboard gives you.
- If you're buying agents: never trust a vendor's own eval at face value. Ask for the rollout-level data — task, trajectory, reward — on a set of tasks that resembles your work. If they can't produce it, you're trusting brand, not evidence.
- For your roadmap: budget evaluation and monitoring at roughly 10–20% of agent development time. The teams that treat eval as a one-time checkbox are the ones shipping agents that pass the demo and break in production.
FAQ
Q: Why can't I just unit-test an AI agent the way I unit-test regular code? A: Because agents are non-deterministic: run the same agent on the same task twice and you can get two different but both-valid paths to the answer. Unit tests assume the same input always yields the same output, which is exactly the assumption agents break. You need repeated rollouts aggregated into a distribution, not a single pass/fail.
Q: What is a "rollout" in agent evaluation? A: A rollout is one full run of an agent through an environment: you hand a sandbox to the agent, it acts until a stopping condition and produces a trajectory, a verifier grades the outcome, and you record the reward. Aggregate reward across many rollouts and you get a reliable eval signal — the agent equivalent of averaging a metric across a validation set.
Q: Do I need to train a model to benefit from rollouts, or is reading the trajectories enough? A: Reading the trajectories is enough to start. The cheapest improvement loop is: run rollouts → read the failing trajectories → find recurring mistakes → write the prompt or PR that fixes them. SFT and RL are higher-leverage but require training infrastructure; trajectory review works on day one.
Q: What's the difference between an eval and a benchmark? A: A benchmark is a fixed, public task set with a leaderboard, used to compare models on problems someone else defined. An eval is your set of tasks that resemble your work. You cite benchmarks to compare models; you build an eval to decide which model you should actually run in production. Public benchmarks measure the world's problems; your eval measures yours.
Q: Is Harbor the only option, or are there alternatives? A: Harbor is one of several. The open-source ecosystem also includes OpenAI Evals (MIT), DeepEval (Apache-2.0), and Arize Phoenix, plus hosted platforms like LangSmith and Braintrust. Harbor's distinguishing feature is the standardized environment format and the rollout-as-a-unit primitive, which makes environments interoperable across teams and training pipelines. Pick based on whether you want a framework that points at your own agent (LangSmith, Braintrust) or an environment-first harness for parallel rollouts (Harbor).
Q: How many rollouts do I need for a trustworthy eval? A: Enough to see the distribution, not just the mode. For a simple task, 20–50 rollouts will surface the dominant failure modes. For anything that ships to users, aim for at least enough that your reward mean and the spread across runs stop moving meaningfully when you add more. One successful demo is never sufficient evidence an agent works.
Discussion
0 comments