AI coding tools do not make every developer faster. A randomized controlled trial published in July 2025 found that experienced open-source developers completed real tasks 19% slower when allowed to use AI tools — even though those same developers estimated afterward that AI had made them 20% faster. Six months later, Anthropic analyzed roughly 400,000 Claude Code sessions and reached the opposite conclusion for a different population: experts who already understand the problem finish more often and let the agent run longer autonomously, while novices quit.
Both studies are correct. The difference is context — and that context is the single most important thing to understand about AI coding productivity in 2026. This article breaks down what the evidence actually says, why the skill that matters has shifted from writing code to directing and verifying agents, and what to build with that shift.
The two studies, side by side
The METR randomized controlled trial (July 2025). Researchers atMETR recruited 16 experienced open-source developers, each with about five years of experience contributing to their specific repositories, and randomly assigned 246 real tasks (bug fixes, features, refactors) to either an "AI allowed" or "AI disallowed" condition. The tasks averaged about two hours each. When AI was permitted, developers mostly used Cursor Pro with Claude 3.5 or 3.7 Sonnet — the frontier models of early 2025.
The measured result: developers took 19% longer with AI. Before the study they forecast AI would save 24% of the time. After the study they still estimated AI had saved 20%. The gap between perception and the clock is the finding that traveled.
The Anthropic session analysis (June 2026). Anthropic ran a privacy-preserving analysis of roughly 398,000 interactive Claude Code sessions from about 235,000 people between October 2025 and April 2026 — a window of markedly more capable agentic harnesses. A classifier rated each session's apparent task-specific expertise on a five-point novice-to-expert scale, inferred from three signals: how precisely the user framed instructions, what they asked Claude to verify, and the direction of corrections (whether the user corrected Claude, or Claude corrected the user).
The result: completion probability scales steeply with rated expertise. Expert-rated sessions triggered about 12 Claude actions per prompt and produced roughly 3,200 words of output; novice sessions triggered about 5 actions and 600 words. Same harness, same observe-think-act loop — the difference was entirely in how the prompt was framed.
Why both can be true
Three axes flip the sign of the productivity effect, and they line up almost perfectly with the differences between the two studies.
Repo familiarity. METR studied experts on mature codebases they had deep context on — the regime where a human already holds the mental model an agent has to reconstruct from scratch. On home turf, the developer's own fluency is the asset the tool partly displaces. Anthropic's population tackled problems that were often novel to the user, where the agent's breadth offsets the user's missing implementation detail.
Tooling maturity. METR used early-2025 AI tooling (Cursor Pro with Claude 3.5/3.7 Sonnet). Anthropic's window spans October 2025 to April 2026 — agentic harnesses like Claude Code with tool use, multi-file edits, and verification loops. METR itself flagged a February 2026 follow-up showing some evidence of speedup, though selection effects complicated interpretation.
Problem novelty. Familiar mature codebase plus early-2025 tooling produced a slowdown. Novel-to-user problems plus post-October-2025 agentic tooling produced a speedup for experts. The scarce input is not typing speed — it is the understanding of the problem itself.
The reconciliation is contextual, not contradictory. As Anthropic's researchers frame it, "agentic coding tools do not flatten the gap between beginners and veterans — they widen it." The scarce input moves from hands-on coding toward task and domain expertise.
The skill that actually matters now: directing and verifying
If the bottleneck has moved, the skill has moved with it. The Anthropic data is explicit about what the expert prompts do differently: they specify constraints up front, hand Claude a checkable success condition, and steer with corrections rather than rewrites. The agent then does more per turn before it has to stop and ask.
Experienced Claude Code users run full auto-approve in over 40% of sessions, versus roughly 20% for new users. Longer leashes are something developers earn through better specification and verification habits, not something a model hands out by default. The 99.9th-percentile turn duration nearly doubled from under 25 minutes to over 45 minutes between October 2025 and January 2026 — not because the model got slower, but because experienced users learned when it was safe to let the loop run longer.
This is the core shift: the valuable skill is no longer "can you write this function" but "can you specify a checkable success condition, recognize when the agent's output is wrong, and correct course without rewriting everything." Writing code is now the cheap part. The expensive, scarce, leverage-multiplying skill is judgment — and the evidence says the people who already had it got faster, while the people who did not got slower or quit.
A real incident: the expired-certificate playbook
The transcript behind this article walks through a concrete episode that illustrates the shift. A production service went down because HTTPS certificates had expired — a classic operational failure that has nothing to do with writing novel code and everything to do with directing an agent through a fix under pressure.
The pattern that worked: the developer did not hand the agent a vague "fix the certs" instruction. They scoped the task narrowly (renew the specific expiring certificate), named the tool (Let's Encrypt certbot), specified the verification (the site returns 200 over HTTPS after renewal), and let the agent execute. The agent handled the mechanical work — installing certbot, running the renewal, validating the certificate chain. The developer handled the judgment: which certificate, which host, what "done" looked like, and whether the result was safe to ship.
This is the shape of productive AI-assisted work in 2026. It is not "let the agent figure it out." It is "give the agent a bounded task with a checkable success condition, then verify." The agent is a fast, tireless executor. The human is the director and the verifier. When the human directs well, the work is dramatically faster. When the human directs poorly, the agent generates plausible-looking output that is wrong, and the human spends longer fixing it than they would have spent writing it from scratch — which is exactly the METR finding.
What to actually build: the 7-level competency ladder
If you want to get faster with AI coding tools, the evidence says you do not need a better model. You need a better operating discipline. Here is the competency ladder the data points to, from the session patterns of the experts Anthropic studied.
Level 1 — Specify a checkable success condition. Before you send any prompt, write down what "done" looks like in a form the agent can verify: "the test suite passes," "the endpoint returns 200 with this response shape," "the build compiles with no warnings." Vague success criteria produce vague output. A checkable condition is the single highest-leverage habit in the data.
Level 2 — Constrain the scope. Tell the agent which files to touch and which to leave alone. Experts in the Anthropic corpus did not let the agent roam the whole codebase — they scoped the task to the module, named the boundaries, and reviewed the diff against those boundaries. Unbounded agents produce unrelated formatting changes, dependency upgrades, and hard-to-review noise.
Level 3 — Steer with corrections, not rewrites. When the agent gets something wrong, do not throw away its work and start over. Correct the specific mistake: "you imported the wrong auth library, use X instead of Y, the rest is fine." Rewriting wastes the context both of you just built. Corrections preserve it. Experts ran longer autonomous loops because they corrected efficiently, not because they tolerated bad output.
Level 4 — Verify before you trust. The METR developers who reported a 20% speedup while actually being 19% slower were experiencing a real cognitive bias: the fast first step (code generation) is memorable, the slower verification step is not. Build verification into your definition of done. Run the test suite. Hit the endpoint. Read the diff. The output is not finished because the agent said so.
Level 5 — Delegate the boring execution, keep the judgment. Once you can do levels 1-4 reliably, you can let the agent run longer autonomously. This is where the 40% auto-approve rate among experts comes from. It is earned by specifying constraints, scoping the work, and verifying — not granted by the model.
Level 6 — Orchestrate parallel agents for independent work. When two sub-tasks do not share files, run them in parallel. Tools like Orca put each agent in its own git worktree so they cannot clobber each other, then let you compare diffs and merge the winner. Parallel agents are not free — they multiply token cost and review burden — but for genuinely independent work (one agent fixes the bug, another writes the regression test, a third audits the affected module) they compress wall-clock time dramatically.
Level 7 — Build systems and processes that make agents safe by default. This is where forward deployed engineering and agent-ready codebases enter the picture. The highest-leverage work is not using an agent better yourself — it is making the codebase, the CI, and the review process agent-safe so that everyone on the team can let agents run longer with less risk. Verification density, not model capability, is the binding constraint.
The model does not matter as much as you think
Once you are past level 3, the choice of model matters less than the discipline. The July 2026 Codex vs Claude Code comparison puts the point in numbers: GPT-5.5 leads Terminal-Bench 2.0 at 82.7% versus Claude Opus 4.7 at 69.4%, while Claude Opus 4.8 leads SWE-bench Pro at 69.2% versus GPT-5.5 at 58.6%. Both tools ship GA multi-agent workflows. Both start at $20 a month. The right answer for most work is not "which model is best" — it is "which model fits the specific task, and am I directing and verifying it well."
Claude Opus 5 ships a five-level effort dial that lets you cut cost 40–66% by routing easy work to low effort and hard work to high effort — which only matters if you already know which is which. That judgment is the skill. The free OpenAI Codex paths and the Hermes Agent OS for orchestration are the tools, not the answer.
The honest takeaway
The METR trial is not a verdict that AI coding tools are bad. It is a verdict that AI coding tools, used without the directing-and-verifying skill, on familiar codebases, with early-2025 tooling, make experienced developers slower. The Anthropic study is not a verdict that AI coding tools are good. It is a verdict that AI coding tools, used with that skill, on novel problems, with late-2025 agentic harnesses, make experts faster and novices quit.
Both are the same finding stated from two sides: the tool is a multiplier on judgment. Good judgment gets multiplied. Bad judgment gets multiplied. No judgment gets replaced. Build the judgment — the ability to specify a checkable success condition, constrain scope, steer with corrections, and verify before you trust — and the tools you pick after that are a matter of cost and fit, not of salvation.
The question for 2026 is not "which AI coding tool makes me fastest." It is "am I the expert in the Anthropic study, or the expert in the METR study?" The difference is not the model. It is you.
Frequently asked questions
Do AI coding tools make developers faster or slower?
It depends on context. A July 2025 randomized trial by METR found experienced developers were 19% slower with early-2025 AI tools on familiar codebases. A June 2026 Anthropic analysis of roughly 400,000 Claude Code sessions found experts finish more often with late-2025 agentic tooling on novel problems. The difference is repo familiarity, tooling maturity, and problem novelty.
What is the METR AI productivity study?
METR ran a randomized controlled trial published in July 2025 (arXiv 2507.09089) in which 16 experienced open-source developers completed 246 real tasks, each randomly assigned to allow or disallow AI tools. Developers took 19% longer with AI but estimated they had been 20% faster. The gap between perception and the clock is the central finding.
What does the Anthropic Claude Code study show?
Anthropic's June 2026 study analyzed roughly 398,000 Claude Code sessions from October 2025 to April 2026. Expert-rated sessions triggered 12 agent actions per prompt and finished a higher share of tasks; novice sessions triggered 5 actions and quit more often. The study concludes agentic coding widens the expert-novice gap rather than closing it.
If AI tools can slow me down, should I stop using them?
Not necessarily. The METR finding applies to experienced developers on familiar mature codebases using early-2025 tooling. The evidence says the skill that matters is directing and verifying an agent — specifying a checkable success condition, constraining scope, steering with corrections, and verifying output. Build that skill and the multiplier turns positive.
What is the single most important habit for productive AI-assisted coding?
Specify a checkable success condition before you send the prompt. Write down what "done" looks like in a form the agent can verify: a passing test suite, a 200 response with a specific shape, a clean build. Vague success criteria produce vague output; checkable criteria let you verify before you trust, which is the habit the METR developers were missing.
Discussion
0 comments