Verdict: Agent-as-a-judge is the third evaluation layer every production AI team needs in 2026. Deterministic checks catch format errors. LLM-as-a-judge catches semantic issues on a single output. But neither can evaluate an agent's trajectory — the multi-step path of tool calls, reasoning, and recovery across an entire trace. Agent-as-a-judge closes that gap by deploying an evaluator agent that reads the full trace dynamically, discovers failure patterns no fixed rubric would catch, and even proposes fixes. The 2025 ICML paper that formalized the framework found agent judges aligned with human consensus far more closely than LLM judges (Judge Shift of 0.27% vs 31.24% on task-dependency requirements). The bottom line: when your system is an agent, your evaluator should be one too.
- Three evaluation layers: code-based (deterministic), LLM-as-a-judge (semantic), agent-as-a-judge (trajectory)
- Agent-as-a-judge reads full traces dynamically — no fixed rubric or column mapping required
- Key agent failures it catches: infinite loops, context loss, inefficient tool selection, "doesn't know when done"
- ICML 2025 research showed agent judges matched human consensus; LLM judges deviated up to 31%
- Production tools: Arize AX Signal, Arize Agent-as-a-Judge (Enterprise beta), open-source frameworks emerging
- Volatile: product availability and features change — last checked July 2026
What Is Agent-as-a-Judge Evaluation?
Agent-as-a-judge is an evaluation method where an AI agent reviews another AI agent's complete execution trace — including tool calls, reasoning steps, intermediate outputs, and final results — to score quality, find failures, and propose improvements. Unlike LLM-as-a-judge, which applies a fixed rubric to a single input/output pair, an agent-as-a-judge dynamically explores the trace, pulls relevant context, and adapts its analysis to what actually happened (arXiv:2410.10934).
The framework was formalized in the 2025 ICML paper "Agent-as-a-Judge: Evaluate Agents with Agents" by Zhuge et al. The authors describe it as "an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process" (PMLR 267).
The core insight is simple: determinism went out the window when agents started taking different paths on every run. A fixed rubric applied to the final output cannot tell you whether the agent took 3 efficient steps or 15 redundant ones to get there.
Why LLM-as-a-Judge Is Not Enough for Agentic Systems
LLM-as-a-judge works by sending a judge prompt with variables mapped to specific columns (input, output, expected answer) and receiving a score back. It was designed for a world where AI systems answered a prompt and returned a single response. That world ended in 2024.
The seminal "Judging LLM-as-a-Judge" paper (Zheng et al., NeurIPS 2023) documented three biases that remain unresolved: position bias (judges favor the first-presented answer, with consistency as low as 23.8% for some models when answer order is swapped), verbosity bias (judges prefer longer responses even when they add no substance), and self-enhancement bias (models favor their own outputs) (arXiv:2306.05685).
These biases are manageable when you are scoring one response. But agentic systems introduce a fundamentally different problem. Here is what changed:
| Capability added | When | What it broke in evals |
|---|---|---|
| Single prompt → response | 2023 | Nothing — LLM-as-a-judge was built for this |
| Tool calling | 2024 | Final output alone no longer reveals quality; the tool-selection path matters |
| Multi-step reasoning | 2024 | A correct answer via flawed reasoning is a reliability risk at scale |
| Long-horizon sub-agents | 2024–2025 | Traces span hundreds of steps; fixed rubrics cannot inspect them all |
| Dynamic UI generation | 2025+ | Every user interaction creates a unique trajectory — no two runs are the same |
When an agent forgets context, gets stuck calling the same tool repeatedly, or fails to recognize a task is complete, these are trajectory-level failures. An LLM-as-a-judge scoring only the final output will mark the run as successful because the output looks fine — it never saw the 12 unnecessary API calls that preceded it.
How Does Agent-as-a-Judge Compare to Other Evaluation Methods?
Agent-as-a-judge is not a replacement for deterministic or LLM-as-a-judge evals — it is a third layer that covers what the other two cannot. Here is how the three evaluator types divide the work:
| Dimension | Code-based (Deterministic) | LLM-as-a-Judge | Agent-as-a-Judge |
|---|---|---|---|
| What it scores | Format, schema, regex, latency, exact-match | Semantic quality of a single output (faithfulness, tone, helpfulness) | Full agent trajectory: tool choices, reasoning path, recovery, efficiency |
| How it works | Rules applied in milliseconds, zero variance | One judge LLM call per span/trace with column-mapped variables | An agent harness reads exported traces at runtime and scores from natural-language instructions |
| Determinism | Fully deterministic | Non-deterministic (same input can yield different scores) | Non-deterministic, but explores context dynamically |
| Setup | Write a rule or unit test | Template + column mappings | Scoring instructions in plain language — no column mapping |
| Best for | JSON schema validation, SQL syntax, PII checks, latency thresholds | High-volume repeatable checks with stable inputs | Multi-step reasoning, nuanced criteria, context-dependent judgment |
| Cost | Near zero | Per-call LLM cost | Higher (longer agent runs) — use selectively, not on every span |
| What it misses | Anything semantic or contextual | Trajectory quality, multi-step dependencies, inefficiency | Simple format checks (overkill for those) |
Source: Arize AX documentation on evaluator types, Arize resource hub comparison.
The routing decision is straightforward: if a check has a verifiable correct answer, code runs it. If it requires reading for meaning on a single output, an LLM judge scores it. If you need to evaluate an agent's full reasoning path, tool selection, and recovery behavior across a trace, that is where agent-as-a-judge earns its cost.
What Evidence Supports Agent-as-a-Judge Over LLM-as-a-Judge?
The ICML 2025 paper by Zhuge et al. is the primary empirical study. The authors built the DevAI dataset of 100 development tasks with 551 testable requirements, then evaluated three code-generating agent systems (AutoCodeRover, OpenHands, and a baseline) using three judging methods: human, LLM-as-a-judge, and agent-as-a-judge.
Key findings:
Agent-as-a-Judge outperformed LLM-as-a-Judge across all task categories. The "Judge Shift" metric (deviation from human consensus) was as low as 0.27% for agent-as-a-judge on requirement-dependency tasks, versus 31.24% for LLM-as-a-judge on the same tasks (arXiv:2410.10934, Table 3).
Agent-as-a-Judge matched individual human judges in reliability. The authors found it was "as reliable as our human evaluation baseline" — a significant result given the cost difference.
The biggest advantage was on tasks with dependencies. When one requirement depended on another being met first, LLM-as-a-judge could not trace the dependency chain. The agent judge could follow the execution path and verify each step.
LLM-as-a-judge scored based on surface features. It gave high marks to solutions that "looked complete" even when intermediate steps were broken — the verbosity and self-enhancement biases documented in 2023 persisted.
Additionally, the ACM SIGKDD 2025 survey "Evaluation and Benchmarking of LLM Agents" catalogued the broader landscape, noting that agent evaluation requires five perspectives: core LLM capabilities, application-specific benchmarks, generalist agent evaluation, benchmark dimensions, and evaluation frameworks (ACM DL). Agent-as-a-judge addresses the gaps in trajectory and multi-step evaluation that traditional benchmarks miss.
What Specific Agent Failures Does Agent-as-a-Judge Catch?
These are the failure modes that deterministic checks and LLM-as-a-judge consistently miss but agent-as-a-judge surfaces:
1. Infinite or Excessive Tool-Call Loops
An agent calls the same API repeatedly — perhaps a search function — without converging on an answer. The final output may be acceptable, but the trajectory reveals 15 calls where 2 would suffice. An agent-as-a-judge reads the full trace, identifies the repeated pattern, and flags the inefficient loop. A fixed rubric scoring only the output sees a correct answer and passes it.
2. Context Loss Across Long Sessions
An agent that starts a task correctly but forgets earlier user instructions by turn 8. LLM-as-a-judge scoring the final response sees a coherent, helpful answer. Agent-as-a-judge reads the full conversation trace, compares the final response against the original request, and flags the missing context.
3. "Doesn't Know When to Stop"
The agent completes the task but continues taking actions — running additional searches, making redundant tool calls, or generating unnecessary UI elements. This is a trajectory-level failure: the agent reached the goal but did not recognize it. Agent-as-a-judge can identify the point of completion and flag every action after it as waste.
4. Inefficient Tool Selection
The agent had the right tool available but chose a slower or more expensive path. For example, calling a full web-search API when a local lookup would have answered. Agent-as-a-judge can compare the tool selected against alternatives visible in the trace and flag suboptimal choices.
5. Recovery Quality
When an agent hits an error mid-task and recovers, how good was the recovery? Did it retry intelligently, or panic and abandon the approach? LLM-as-a-judge sees only the pre-error and post-error outputs. Agent-as-a-judge reads the error, the reasoning steps, the retry attempt, and scores the recovery strategy itself.
How Do You Build an Agent-as-a-Judge Evaluation?
Building an agent-as-a-judge eval involves five steps. The exact implementation depends on your tooling, but the pattern is consistent.
Step 1: Instrument Your Agent With Tracing
Before you can evaluate a trajectory, you need to capture one. Use OpenTelemetry-based tracing (OpenInference spans) to record every model call, tool invocation, retrieval, input, output, timing, and error in your agent's execution. Your traces are the evidence base — the richer they are, the better your agent judge can work.
If you are building from scratch, the open-source Arize Phoenix project provides OpenInference instrumentation for most popular frameworks. Pipecat, LangChain, LlamaIndex, and others have native or community OpenTelemetry support.
Step 2: Define What "Good" and "Bad" Look Like in Plain Language
Unlike LLM-as-a-judge, where you write a template with {variable} placeholders and map columns, agent-as-a-judge takes natural-language scoring instructions. You describe the quality criteria in prose:
"Review the agent's full trace. Check whether it used the most efficient tool for each step, whether it avoided repeating the same call more than twice, and whether it recognized task completion without unnecessary additional actions. Flag any step where a simpler alternative was available. Score the trajectory from 1–5."
This is a key advantage: expressing nuanced criteria in prose is easier than wiring them into column maps, and the agent judge can interpret them dynamically per trace.
Step 3: Choose an Agent Harness
The agent-as-a-judge runs inside an agent harness — a coding agent or similar tool that can read exported traces, reason over them, and write results back. As of July 2026, the supported harnesses include:
- Claude Code (Anthropic) — supported in Arize AX's Agent-as-a-Judge evaluator
- Cursor, Codex, Hermes, and OpenClaw — listed as coming soon to Arize AX
Source: Arize AX Agent-as-a-Judge docs.
If you are building your own, the open-source implementation from the ICML 2025 paper is available at github.com/metauto-ai/agent-as-a-judge.
Step 4: Run the Evaluator Against Production Traces
Attach the agent-as-a-judge evaluator to an online eval task on your traced project. Configure the date range, query filter, and sampling rate — the same controls as template eval tasks. On each run, the harness reads exported spans for the task window, applies your scoring instructions, and publishes eval columns back onto the spans.
Because agent-as-a-judge is more expensive than a single LLM call, sample strategically. Run deterministic checks on 100% of traces, LLM-as-a-judge on a high percentage, and agent-as-a-judge on a sample or on traces flagged by the other two layers.
Step 5: Close the Loop — From Evaluation to Fix
The most powerful extension is turning evaluation output into action. Arize's Signal product (launched July 2026) continuously reviews production traces, groups failures into ranked issues with evidence and root-cause analysis, and — with repository access — can open a pull request with a proposed fix (Arize blog, July 29 2026).
The loop is: investigate → propose → review → ship → observe again. The engineer still makes the call; they just start with a diagnosis instead of a blank terminal.
What Production Tools Support Agent-as-a-Judge Today?
| Tool | What it does | Availability (July 2026) |
|---|---|---|
| Arize AX Agent-as-a-Judge | Runs evaluations with a Claude Code harness that reads traces at runtime; no column mapping; writes results as eval columns | Closed Enterprise beta |
| Arize AX Signal | Continuously reviews production traces, groups failures into ranked issues with evidence + root-cause + proposed fix; can open PRs with repo access | Free and Pro tiers (full managed agents: Enterprise beta) |
| Arize Phoenix (open-source) | OpenInference tracing, datasets, LLM-as-a-judge evals; agent-as-a-judge support in development | Open-source (GitHub) |
| Pipecat Evals | Behavioral testing for voice/text agents with scripted conversations, semantic assertions, LLM judge; integrates with Arize for online evals | Open-source |
| metauto-ai/agent-as-a-judge | Reference implementation from the ICML 2025 paper | Open-source (GitHub) |
Sources: Arize AX docs, Arize Signal blog, Pipecat Evals docs, ICML 2025 paper repo.
Volatile: product availability, pricing, and feature sets change frequently. Verify directly with vendors before committing.
What This Means for You
If you are building AI agents production: Start with code-based evals for format and safety checks — they are fast, cheap, and deterministic. Add LLM-as-a-judge for semantic quality scoring on the final output. Then layer agent-as-a-judge on a sample of production traces to catch trajectory-level failures the other two miss. You do not need all three on every span; you need each one where it fits. If your agent takes multi-step paths with tool calls, you need agent-as-a-judge — no fixed rubric will catch a loop or a missed completion signal.
If you are a small team or solo builder: You do not need Arize Enterprise to start. Use the open-source Phoenix for tracing, the reference agent-as-a-judge implementation for trajectory evaluation, and LLM-as-a-judge for the high-volume checks. Instrument your agent with OpenTelemetry from day one — the traces are the foundation for everything else. See our guide on building LLM agent evals that work for the foundational setup before adding the agent judge layer.
If you are evaluating between eval strategies: The key question is not "which method should I use" but "which method for which check." Code catches structure, LLM catches semantics, agent catches trajectory. Most production gaps come from using one method for everything, not from picking the wrong method. Our closed-loop evaluation guide walks through building the full pipeline.
For teams already running evals on production traces, the next step is closing the loop. Evaluation without action is a dashboard; evaluation that feeds a proposed fix is an improvement system. If you are exploring self-improving agent architectures more broadly, see our comparison of self-improving AI agents vs static copilots.
FAQ
Q: What is the difference between LLM-as-a-judge and agent-as-a-judge?
A: LLM-as-a-judge sends a single judge prompt with mapped variables to score one input/output pair using a fixed rubric. Agent-as-a-judge deploys an evaluator agent that reads the full execution trace dynamically, explores context across multiple steps, and adapts its analysis to what actually happened — catching trajectory-level failures like loops, context loss, and inefficient tool selection that a fixed rubric cannot see (arXiv:2410.10934).
Q: Do I need to replace my existing LLM-as-a-judge evals with agent-as-a-judge?
A: No. Agent-as-a-judge is a third layer, not a replacement. Use code-based evals for deterministic format checks, LLM-as-a-judge for high-volume semantic scoring, and agent-as-a-judge selectively on traces where trajectory quality matters. Most teams use all three on the same project (Arize evaluator types docs).
Q: Is agent-as-a-judge more expensive than LLM-as-a-judge?
A: Yes. An agent judge runs a longer, multi-step process to explore a trace, so per-evaluation cost is higher. Use it on a sampled subset of traces or on traces flagged by cheaper methods — not on every span. The trade-off is that it catches failures the cheaper methods systematically miss.
Q: What research proves agent-as-a-judge is better for agentic systems?
A: The ICML 2025 paper "Agent-as-a-Judge: Evaluate Agents with Agents" by Zhuge et al. found that agent-as-a-judge matched human judge consensus far more closely than LLM-as-a-judge, with Judge Shift as low as 0.27% vs 31.24% on task-dependency requirements. The framework was benchmarked on the DevAI dataset of 100 development tasks across three code-generating agent systems (PMLR 267).
Q: Can agent-as-a-judge propose fixes, not just find problems?
A: Yes. Production tools like Arize Signal go beyond flagging issues — they group failures into ranked issues with evidence, root-cause analysis, and proposed fixes. With repository access, the agent can open a pull request with the fix. The engineer still reviews and ships; the agent does the investigation. This creates an improvement loop: investigate, propose, review, ship, observe again (Arize blog, July 2026).
Q: How do I start with agent-as-a-judge if I am not an Arize Enterprise customer?
A: Use the open-source reference implementation at github.com/metauto-ai/agent-as-a-judge from the ICML 2025 paper. For tracing, use the open-source Arize Phoenix. Start by instrumenting your agent with OpenTelemetry, then run the agent judge against traces where you suspect trajectory-level failures. Our guide on building agent benchmarks from production traces covers the trace-to-eval pipeline.
Discussion
0 comments