Verdict: The single most useful framework for predicting which AI application layer markets will mature next is verifiability — specifically, how easily a domain's tasks can be decomposed into checkable steps, whether the field agrees on what "correct" means, and how often the real world hands you fresh verified examples. AI coding matured first not because code is easy, but because GitHub solved all three of those problems at once. Finance, healthcare, and law are next precisely because they score low on the same axes and are locked inside enterprise workflows nobody has captured yet. Meanwhile, most public benchmarks are quietly compromised by training-data contamination and Goodhart's-law optimisation, meaning the one number a vendor quotes you is a single noisy sample from a distribution nobody has measured.
Last verified: 2026-07-30 · AI coding was the first mature app-layer market; finance, healthcare, and law are mid-maturation; scientific discovery and "taste" domains remain hardest. · Benchmark saturation is now the norm (GPQA Diamond: PhD humans ~65%, GPT-5.6 Sol ~93%+). · Data vendors selling "hard" contrived data and billing it as real captured real-workflow data is the industry's open secret. · Your moat is not the data — it is the pipeline into real-world work plus the infrastructure to keep retraining on it as base models migrate.
What Is Verifier's Law and Why Does It Matter for AI Application Markets?
Verifier's Law states that the ease of training an AI model to perform a task is proportional to how verifiable the task is — because in reinforcement learning, the ability to verify a solution is exactly equivalent to the ability to create an RL environment. The law was articulated by Jason Wei (then at OpenAI, now at Meta's Superintelligence Lab) in a blog post titled "Asymmetry of verification and verifier's rule," published July 15, 2025.
In plain terms: if you can check the answer quickly, objectively, and at scale, you can grind on it with reinforcement learning until the model gets it right. If you cannot, you are stuck — no matter how much compute or money you throw at the problem. The law implies a jagged frontier of intelligence: AI will appear superhuman on easily verifiable tasks (math competitions, coding exercises, structured QA) and plateau on fuzzy ones (taste, long-horizon business judgment, open-ended research).
The Three Axes of Verifiability
Wei's original formulation breaks verifiability into five properties — objective truth, fast verification, scalable verification, low noise, and continuous rewards. A working framework used by data-market analysts (building on Wei's post and subsequent corroborating research) reduces these to three decision-useful axes you can apply to any professional domain today:
- Asymmetry of verification — how hard is it to decompose the task into individually checkable steps? A coding task can be broken into unit tests. A legal memo may require two senior partners to read 30 pages to know if the reasoning holds.
- Veracity of verification — does the field agree on what "correct" means? In math, yes. In design, taste, and strategic business decisions, emphatically not. This is the hardest axis to game.
- Proliferation of verification — how often does the real world produce fresh, independently verified examples of the work? GitHub generates millions of commits with test results weekly. A private equity LBO memo produces one example per deal, locked in a confidentiality vault.
| Domain | Asymmetry | Veracity | Proliferation | Maturity |
|---|---|---|---|---|
| Coding | High (unit tests) | High (code runs or it does not) | Very high (GitHub) | Mature |
| Search/QA | High | High | Very high (web) | Mature |
| Finance (quant) | High | High | Medium (proprietary) | Maturing |
| Healthcare/clinical | Medium | Medium | Low (sealed records) | Maturing |
| Law | Low | Medium | Low (sealed briefs) | Early |
| Cybersecurity | Medium | High | Low (private vuln DBs) | Early |
| Scientific discovery | Low | Medium | Very low | Emerging |
| "Taste" (design, brand, creative) | Very low | Very low | Very low | Hardest |
If you can classify a profession's tasks across these three axes and apply a long-horizon bar — tasks that require sequential decisions across many dependent steps, heterogeneous tool calls, genuine state transitions, and mandatory failure recovery — you have a predictive map for which application layer markets will mature next. No single leaderboard number will show you this. It requires looking at the raw data-buying behaviour of the labs themselves.
Why Did Coding Become the First Mature AI Application Market?
Coding scored high on all three axes simultaneously, and that was not an accident — it was the lucky inheritance of Web 2.0. GitHub, launched in 2008, gave the AI era exactly the resource it needed: an enormous, public corpus of code paired with unit tests (objective, decomposable correctness), community consensus on what "working code" means, and effectively infinite examples where commit messages serve as free reasoning traces.
This is why the trajectory went code → search → finance → healthcare and law → cybersecurity and scientific discovery. Each subsequent domain scores lower on one or more axes, and the data is progressively more locked inside enterprise workflows that no public Web 2.0 system ever captured. That is the whole predictive game.
The practical implication: if you are a builder or investor trying to spot the next AI-native software category, do not chase what the model providers say is next — watch where the data-buying dollars go. That is an upstream indicator that something is being built. The correlation is strong enough that domain-specific data vendor emergence precedes a new application-layer product by roughly 2-3 months. It also connects directly to how we evaluate AI agents on long-horizon tasks — the same verifiability axes that decide which markets mature also decide which eval harnesses are trustworthy.
How Real Is Your Training Data? Type 1 vs Type 2
A critical distinction unpacked by data-market analysts is Type 1 (captured) versus Type 2 (contrived) data:
- Type 1 is pure capture of real working workflows — GitHub commits, session replays, production traces — with minimal reward shaping by people who are not themselves domain experts. The realism is inherited from the work itself. Type 1 is what moves a model from 20% to 80% competence on a real task.
- Type 2 is contrived: you hire experts, seat them in an arbitrary setting, manufacture example tasks, and label them. Type 2 is the right place to start a model when it is reading at a first-grade level — anybody can teach it. But beyond a point, contrived data has diminishing returns because the artificial structure does not match production's messiness.
The dirty secret of the data industry is that everybody sells Type 2 and bills it as Type 1. The reason: real captured workflow data is rare, expensive, and often entangled with privacy and enterprise constraints. A vendor can manufacture Type 2 data at scale for a fraction of the cost and rebrand it. The result is data that looks dense in a benchmark sweep and barely moves the needle on real-world work.
The structural reason this matters: a data set is only valuable for as long as the frontier stays where the data was captured. Once the frontier moves, last year's data stops being useful. The only durable source of real captured data is a live business partner whose workflows you continuously ingest — not a dead startup's codebase that you acquire.
How Do Most AI Benchmarks Get Quietly Compromised?
Most public benchmarks you see are compromised in ways that do not show up on the headline score. The dominant pattern — studied and catalogued across academic and industry research — is training-data contamination: the benchmark's test set is published on the open web to be reproducible, and is eventually swallowed by the next web-scale crawl. The model memorises the questions rather than reasoning to them.
The 2025 CACM analysis "Goodhart's Law Comes for Every Benchmark You Trust" documents this directly: even benchmarks with explicit anti-contamination mechanisms (such as BIG-bench's canary strings) had their questions absorbed into training data anyway, because the mechanism designed to catch cheating did not survive contact with a web-scale crawler.
The problem compounds with a profit motive. The dominant Type 2 data-market recipe is: hire domain experts; have them generate plausible tasks with an LLM's help; solve those tasks; cherry-pick the ones where the model diverged; package them as a "hard" North Star benchmark; then sell the data to hill-climb on that same benchmark.
This is Goodhart's Law with a profit motive: the moment the measure becomes a target, and the target is set by people who are not true domain experts, the benchmark stops measuring anything real. Charles Goodhart articulated the principle in 1975 in the context of British monetary policy: "When a measure becomes a target, it ceases to be a good measure."
Practical symptoms to look for when a vendor hands you a benchmark number:
- Only looks at a single isolated in-distribution question — contrived benchmarks cannot test whether a model sustains correct reasoning across a long dependent episode, because they were designed to be solved in isolation.
- High false-positive and false-negative rates hidden behind a high mean — a single benchmark number under a single scaffolding is essentially one sample from a distribution. The variance is hidden. A widely-noted pattern is that the same model scores within a few points of itself on the same task, but fails in opposite directions — nailing methodology while losing arithmetic, or vice versa. This is why treating agent evaluation as a rollout — not a one-off — is increasingly the discipline frontier teams demand.
- Cross-harness differencing — when you run the same task under a different scaffolding or different infrastructure, performance diverges. This is the primary source of "benchmark psychosis": two equally good models look different because the test harness injects noise, not because one model is better.
A Worked Real-World Example
Consider three real-world finance tasks — an ARR waterfall reconciliation, an LBO valuation memo, and a long-short pair trade — paired with deterministic verifiers plus an LLM-as-judge applied carefully. The tasks are long-horizon, non-trivially verifiable, and representative of what top funds actually need.
Run these against multiple frontier models under a properly agnostic rubric, and a clear pattern emerges:
- Newer is not always better. A more recent flagship can score lower than its predecessor on certain long-horizon rubrics — in some cases due to over-engineered self-reflection loops that interfere with the task structure.
- Models can score within three points on the same task and fail in opposite directions: one nails the arithmetic but loses the methodology; the other nails the methodology and drops the arithmetic.
- That direction-of-failure signal tells you exactly where each lab's post-training pipeline poured its effort, and can inform what data you need to buy next.
No leaderboard number gives you any of this. The receipts from such benchmarking feed directly into RL environment reports that labs consume. The leaderboard numbers the public sees are an aggregate that hides which sub-capabilities are actually moving.
How Do You Tell Which AI Application Layer Domain Is Next?
Use this three-part checklist — the same one data-market professionals use to read the upstream indicator.
Step 1: Classify tasks along the three verifiability axes
For each candidate domain, ask: how asymmetric is verification? How much veracity (field consensus on "correct") exists? How proliferative is the verification (how often does the real world hand you fresh verified examples)? Domains that score low on all three are the hardest, but also where the moat is deepest once cracked.
Step 2: Apply a long-horizon bar (not a single-question test)
Real labs use long-horizon tasks — not single-question benchmarks. The signals to look for in the raw data:
- Sequential decisions against a single entity across the whole task
- An inferable expert action per step (so you can shape rewards)
- Outcomes recorded by an independent party (not self-graded)
- Economically available high-wage work — meaning the domain is valuable enough that solving it pays
Watch for heterogeneous tool calls that are not interchangeable, state transitions that genuinely constrain future actions, and mandatory failure recovery — tasks where the agent cannot paper over a mistake. Most vendor "long-horizon" specs fail on at least one of these counts.
Step 3: Watch where data-buying dollars move
When a major lab quietly shifts data spend to a new vendor specialising in (say) cybersecurity environments, an application-layer product in that domain typically surfaces 2-3 months later. This is one of the highest-leverage observation channels for reading the market.
Counterexample to keep you honest: robotics
Robotics is the modality-not-settled counterexample. Whether the right capture modality is ego (first-person camera), teleop (operator-driven), or demonstration-based (such as Universal Manipulation Interface approaches) is an unsolved research question. A domain where the modality is not yet settled is a domain where vendor selection is entangled with research risk — a much harder portfolio bet than a data problem alone. Choose the vertical where the research question is already decided before betting your data spend.
Is the Foundation Model Layer Going to Lock In, or Open Up?
A relevant counterpoint sometimes raised by data-market analysts: historically, no pioneer of an infrastructure technology has held more than 10% of the market in the long run. Railroads built company towns and charged tyrannical rents. The automobile arrived, and they got regulated into utility status. AWS and Google consolidated the cloud infrastructure layer but never captured the application layer on top of it. This pattern suggests foundation model lock-in is far from guaranteed.
Today's frontier labs face genuine pressures: anti-distillation clauses in their terms, export and exclusivity negotiations with nation-states and enterprises, and rapid quality erosion from open-weight competitors. When open-weight models reach near-parity on real-world rubrics — which is already happening in some workloads — enterprise customers gain a decoupling lever: they can start owning their own intelligence rather than renting it from labs.
The implication for builders: models differ on efficiency and modality; they are not fungible like electricity. The layer that is being created right now is an abstraction layer for enterprises to manage their own multi-model, multi-base deployments — an emerging infrastructure need that does not yet exist. If you want a concrete take on when it makes sense to stay on the frontier API and when to decouple, the LLM cost optimisation guide covers the token-economics math behind that decision.
What Does "Successful Data Company" Even Mean After the Pivot?
The take-away from data-market analyses is uncomfortable for the data industry itself: successful data companies are all pivoting to become enterprise workflow and infrastructure companies. The durable value does not accrue to raw data — it accrues to the services and application layer that uses real-world work. Pure-play data businesses do not stay pure-play data businesses, because the data alone depreciates as the frontier moves.
The emerging infrastructure stack enterprises will need includes:
- Routing and serving infrastructure for small models targeted by cost, latency, and performance profile — a multi-model router.
- RL dataset management across base-model migration — when you swap to a new open-source base model, you must re-run post-training automatically, not start from scratch.
- Translation systems that convert messy business context into evals — the bespoke systems that interpret an enterprise's idiosyncratic workflows into a reward signal an RL loop can consume. The metaphor here is the Antikythera mechanism: an intricate, hand-built translation device that bridges incompatible representations.
This is where the next decade of B2B AI infrastructure spending is headed if the enterprise decoupling thesis holds.
What This Means for You
- If you are a researcher: stop outsourcing your definition of "realistic data" to the same vendor you buy your evals and tasks from. That is letting the task-test writer grade the task. Build or own your own verifiers tied to your production environment.
- If you are a builder: your moat is not the data — it is the pipeline into real-world work plus the infrastructure to keep retraining on it as the models improve underneath you. Treat your relationship with a real working business (yours, or a partner's) as the durable asset, not the training set you snapshot today.
- If you are evaluating vendor claims: ignore single leaderboard numbers. Demand to see rubric-level scores under multiple harnesses, or run your own long-horizon benchmarks against your real distribution. The variance between harnesses and across failure types is more diagnostic than the mean. Our closed-loop AI agent evaluation guide covers how to run this continuously in production, and the synthetic data Pipeline guide shows where synthetic augmentation helps versus where it actively degrades model capabilities.
FAQ
Q: What is Verifier's Law in AI? A: Verifier's Law, articulated by Jason Wei in July 2025, states that the ease of training an AI model on a task is proportional to how verifiable the task is — because in RL, the ability to verify a solution is equivalent to being able to build an environment to train on it. Tasks that are fast, objective, and scalable to verify will be solved by AI; tasks where "correct" is contested or expensive to judge will plateau.
Q: Why did AI coding mature before finance or healthcare? A: Coding inherited GitHub — a public corpus matching unit tests (objective, decomposable correctness), community consensus on what working code means, and effectively infinite examples with commit messages as free reasoning traces. Finance and healthcare score lower on verifiability, and their data is locked inside enterprise workflows no public Web 2.0 system ever captured.
Q: What is the difference between Type 1 and Type 2 training data? A: Type 1 data is pure captured real production workflows with minimal reward shaping by non-experts; realism is inherited from the work itself. Type 2 data is contrived: experts hired to manufacture example tasks in an arbitrary setting. Type 1 moves a model from poor to competent; Type 2 is the right place to start a model that is still weak, but plateaus faster. The industry's open secret is that almost all vendors sell Type 2 and bill it as Type 1.
Q: Why are AI benchmarks unreliable as decision inputs? A: Most public benchmarks suffer from training-data contamination (the test set was absorbed by the model's training crawl) and Goodhart's-law optimisation (vendors hill-climb on the same benchmark they are scored on). A single benchmark number is one noisy sample from an unmeasured distribution. Two equally good models can score within three points and fail in opposite directions — a signal the headline number hides.
Q: How do I predict which AI application layer domain matures next? A: Classify a domain's professional tasks across three verifiability axes (asymmetry of verification, veracity of verification, proliferation of verification), apply a long-horizon bar (sequential decisions, heterogeneous tools, state constraints, mandatory failure recovery), and watch where the big labs are shifting data-buying dollars — those vendors tend to precede a product launch by 2-3 months.
Q: Are foundation model providers going to lock in the market permanently? A: The historical pattern suggests not. No pioneer of infrastructure technology has held more than 10% market share in the long run. Open-weight model quality is approaching parity on many real-world rubrics, giving enterprises a decoupling lever. The durable infrastructure need is not more data — it is the multi-model serving, retraining, and business-context-to-eval translation layers that enable an enterprise to own its own intelligence rather than rent it.
Discussion
0 comments