Verdict: Today's frontier AI models excel at short, verifiable tasks — answering questions, writing code snippets, summarizing text. But when you ask an AI agent to sustain reasoning over hours or weeks of interleaved decisions (trading a portfolio across a season, debugging a complex system end-to-end, researching and writing a multi-part report), performance collapses. Every frontier model given £100,000 to bet on Premier League football for a year lost money. The gap between "good at a one-shot prompt" and "good at a multi-hour task" is the most important unsolved problem in AI. The fix is not just bigger models — it requires new algorithms (value-based RL), new infrastructure (RL environment platforms), and careful compute orchestration between training and inference.
Last verified: 2026-08-01
- AI task-completion horizons are doubling every ~7 months (METR, March 2025)
- All frontier models lost money on KellyBench, a long-horizon betting benchmark (General Reasoning, April 2026)
- Three technical bottlenecks: gradient variance, sparse rewards, and context-window exhaustion
- Key fix: value-model bootstrapping + context compaction + pipeline RL
- Pricing/limits change often — last checked August 2026
What Is Long-Horizon Reasoning in AI?
Long-horizon reasoning is the ability of an AI agent to maintain coherent, goal-directed behavior across many sequential steps — hundreds or thousands of decisions over hours, days, or even weeks. Unlike a single prompt-response exchange, long-horizon tasks involve intermediate planning, course correction, memory management, and adapting to outcomes that unfold over time.
The word "horizon" comes from reinforcement learning, where the "time horizon" defines how many steps into the future an agent must plan. A chatbot answering a question has a horizon of 1 — one response. A coding agent fixing a bug has a horizon of maybe 10 — read the error, locate the file, edit, test, iterate. But an agent tasked with researching a market, building a model, backtesting it, and deploying it over a full year has a horizon in the thousands.
METR, an AI safety research organization, measured this precisely. In a March 2025 study, they found that the length of tasks AI agents can complete autonomously with 50% reliability has been doubling approximately every 7 months for six years (METR, arXiv:2503.14499). As of early 2026, the best frontier model (Claude Opus 4.6) reached a 50% success horizon of roughly 12 hours — meaning it reliably completes tasks that take a human expert about half a day. That sounds impressive until you realize most real-world knowledge work takes days to weeks.
Why Do Current AI Models Fail at Long-Horizon Tasks?
Three structural bottlenecks cause long-horizon failure: gradient variance that grows with trajectory length, sparse rewards that arrive only at the very end, and context windows that run out of room before the task is done.
Bottleneck 1: Gradient variance scales with length. When a reinforcement learning agent takes thousands of steps before getting a reward, the statistical noise in the learning signal grows enormous. An agent that makes 500 sequential decisions and fails learns almost nothing about which of those 500 steps were responsible. This is the classic credit assignment problem, and it gets exponentially worse as the horizon extends.
Bottleneck 2: Sparse rewards. Most RL training for language models uses verifiable rewards — a clear yes/no at the end (was the math answer correct? did the code pass tests?). That works for short tasks. But for a task that takes a full season of betting, the only reward signal comes at the very end: did you make money? By that point, the agent has made hundreds of decisions, and the single binary outcome tells it almost nothing about which individual decisions were good or bad.
Bottleneck 3: Context window exhaustion. Even with modern context windows of 1 million tokens (which sounds enormous), a long-horizon agent generating reasoning traces, tool calls, observations, and intermediate conclusions over many hours can easily exceed that limit. Consider Fermat's Last Theorem: mathematician Andrew Wiles spent years reading papers, writing notes, taking walks, and revising proofs before completing it. Converted to tokens — every paper read, every scratchpad calculation, every false start — that's easily tens of billions of tokens. Today's largest context window is 1 million. The gap is three to four orders of magnitude (arXiv:2604.27865).
How Does RLHF Separate Good Base Models From Great Products?
The most important lesson from recent AI history is that a strong base model is necessary but not sufficient. Reinforcement learning from human feedback (RLHF) — or its newer variant, reinforcement learning with verifiable rewards (RLVR) — is what turns a capable model into a useful product.
The clearest natural experiment came in November 2022. Two weeks before ChatGPT launched, Meta released Galactica, a 120-billion-parameter model trained on 106 billion tokens of curated scientific text. Galactica was genuinely state-of-the-art: it scored 41.3% on mathematical MMLU (vs. Chinchilla's 35.7%), 20.4% on the MATH benchmark (vs. PaLM 540B's 8.8%), and 68.2% on LaTeX equation prediction (vs. GPT-3's 49.0%) — all with a fraction of the compute (Taylor et al., arXiv:2211.09085). But Galactica shipped as a bare base model with a demo. Users prompted it to write about absurd topics, it hallucinated confidently, and the public backlash was immediate.
Two weeks later, ChatGPT launched — built on a similar-quality base model (GPT-3.5) but with a critical addition: an RLHF pipeline that trained the model to be helpful, harmless, and honest. The difference in reception was night and day. RLHF was the ingredient that made LLMs products for the first time.
This wasn't even the first time RL proved its value. OpenAI's InstructGPT, released in 2022, showed that a 1.3-billion-parameter model trained with RLHF was preferred by humans over the 175-billion-parameter GPT-3 — a 100x parameter efficiency gain attributable entirely to reinforcement learning (Ouyang et al., arXiv:2203.02155).
| Model | Parameters | Training Method | Key Result |
|---|---|---|---|
| InstructGPT (1.3B) | 1.3 billion | RLHF | Preferred over 175B GPT-3 in human eval |
| Galactica (120B) | 120 billion | Base model only | SOTA on MATH/MMLU but product failure |
| ChatGPT | ~175 billion | RLHF on GPT-3.5 base | World-changing product success |
| DeepSeek-R1 | 671 billion | GRPO with verifiable rewards | Open-source reasoning breakthrough |
Sources: Ouyang et al., 2022; Taylor et al., 2022; Guo et al., 2025
Galactica also pioneered two ideas that later became mainstream. First, data efficiency: it trained on 106 billion tokens of curated scientific text, not the trillions of web tokens used by Chinchilla and GPT-3. Second, multi-epoch training: it trained for 4.25 epochs, challenging the then-consensus that repeated data was harmful. Later research formalized this — Muennighoff et al. found that up to 4 epochs of repeated data causes negligible loss, with diminishing returns starting around 16 epochs (Muennighoff et al., arXiv:2305.16264). Galactica also introduced "thinking tokens" — internal working memory tags (<work> tokens) where the model spends inference compute reasoning before producing an answer, a direct precursor to the thinking tags in modern reasoning models.
What Is the "Bitter Lesson" and Why Does It Apply Here?
Rich Sutton's 2019 essay "The Bitter Lesson" argues that general methods leveraging computation ultimately outperform domain-specific human knowledge by a large margin (Sutton, 2019). The lesson: don't build clever hand-engineered features; instead, scale compute and let general learning algorithms discover structure.
This lesson explains why early RL experiments on language models — work that applied PPO with verifiable rewards to mathematical reasoning as early as 2023 — didn't produce the emergent reflective behavior seen in OpenAI's o1 or DeepSeek-R1. The same recipe that produced state-of-the-art math results in 2023, using Llama 2 base models with 4,000-token context windows, failed to produce the backtracking, self-correction, and extended chain-of-thought reasoning that became the hallmark of reasoning models.
The difference wasn't the algorithm. It was the scale of three things: better base models, more RL compute, and bigger context windows. When DeepSeek-R1 applied GRPO (Group Relative Policy Optimization) with verifiable rewards on top of the much stronger DeepSeek-v3 base model with vastly larger context, the emergent reasoning behavior appeared automatically (Guo et al., arXiv:2501.12948).
The practical implication: if you're building AI applications, investing in better base models and more RL compute will typically beat investing in clever prompt engineering or custom reasoning pipelines. This also has a sociological dimension — whoever holds the best base model can see further into the research frontier, spotting behaviors and capabilities that teams with weaker models cannot yet reach. That compounding advantage is hard to break.
What Is KellyBench and What Did It Prove?
KellyBench is a long-horizon benchmark released in April 2026 by General Reasoning, a London-based AI research company. It placed frontier AI models in a simulated sports-betting environment for the full 2023–24 English Premier League season, giving each model a £100,000 bankroll and asking it to maximize long-term growth by betting on individual matches across 380 games over nine months.
Every single frontier model lost money.
The results were reported on the front page of the Financial Times and captured public attention precisely because betting is a domain with clear, real-world outcomes — unlike coding benchmarks where success is somewhat subjective (General Reasoning, KellyBench, April 2026; arXiv:2604.27865).
| Strategy | ROI | Final Bankroll | Notes |
|---|---|---|---|
| Human quant | +5.1% | £105,118 | Professional sports bettor |
| AI Researcher | -4.3% | £95,705 | Human AI researcher |
| GPT-5.4 (avg) | -7.9% | £92,063 | 5-seed average |
| Dixon-Coles model | -15.4% | £84,567 | Classic 2000s statistical baseline |
Source: KellyBench paper, arXiv:2604.27865
A classic Dixon-Coles statistical model from the 2000s outperformed three out of five frontier models. The human quant walked away with a profit. Even the best-performing AI model finished at -11% ROI.
Three reasons explain why models failed so badly:
- Industry bias toward coding and procedural tasks. Most benchmarks ask models to "do this, fix that" — tasks with one or two valid solutions and no room for creative, open-ended reasoning.
- Too few open-ended tasks in training. The real world involves complexity, uncertainty, and changing conditions that benchmarks don't capture.
- No simulation of other agents. Real-world decisions happen in multi-agent environments. Football betting involves bookmakers adjusting odds, other betters, injuries, weather — none of which single-agent training captures.
How Does Context Compaction Extend the Effective Horizon?
Context compaction is the primary technique for extending an agent's effective working memory beyond its context window. The idea is straightforward: let the agent generate tokens until it nears the context limit, then summarize the relevant information, reset the beginning of the context, and let the agent continue generating on top of the summary.
This creates a form of progressive memory compression. Critical content — tool calls, state changes, key decisions — is preserved verbatim. Routine content like intermediate reasoning traces is compressed into structured summaries. The agent retains enough signal to support future replanning without consuming middleware budget.
The advantage of applying RL to compaction is that you can train the agent to both compact effectively and complete the task simultaneously, killing two birds with one stone. The compaction itself becomes a learned skill rather than a hand-coded heuristic.
However, compaction alone doesn't solve the underlying RL optimization problems:
- Gradient variance still scales with length — even with compaction, the effective trajectory is still long
- Sparse rewards remain sparse — the agent still gets a binary signal only at the very end
- Variable trajectory length adds instability to the optimization
What Are Value Models and How Do They Help Long-Horizon RL?
Value models (also called critics) are a second neural network trained alongside the policy model to estimate the expected future reward at any intermediate step. They address three long-horizon problems simultaneously:
Reducing gradient variance: By providing a running estimate of how good each state is, the value model turns a sparse end-of-episode signal into dense per-step feedback. Instead of learning "the whole 500-step episode failed," the agent learns "step 47 was slightly worse than expected."
Bootstrapping before episode end: A value model can generate expected-reward signals before the full episode completes. Think of it as a dopamine system — the agent gets a sense of whether it's on the right track before reaching the final outcome. This is critical when inference takes weeks, because you can start training the model while episodes are still running.
Encouraging batch diversity: Value-based training naturally produces more diverse trajectories, which stabilizes learning and prevents the policy from collapsing to a single mode.
The trade-off is complexity. GRPO (used by DeepSeek-R1) eliminates the need for a separate value model by computing relative advantages from Monte Carlo rollouts. This is simpler and works well for short-to-medium horizon tasks. But for truly long horizons (weeks of inference), GRPO's lack of a value model means you must wait for episodes to finish before training — wasting GPU time. Introducing a value model adds training overhead but allows the pipeline to keep GPUs productive. Investing in data quality and compute efficiency matters as much as algorithmic novelty — see our guide to data quality as a compute multiplier.
How Does Pipeline RL Manage the Compute Trade-Off?
Pipeline RL is a technique for overlapping inference and training to maximize GPU utilization during long-horizon RL. In traditional RL, you run inference to completion, then start training. For short tasks that's fine. For long-horizon tasks where inference can take weeks, this means GPUs sit idle waiting for episodes to finish.
Pipeline RL works by starting to train on completed sequences while remaining sequences are still being generated. This introduces off-policy drift — the policy being trained starts to diverge from the policy that generated the data. From practical experience, up to roughly 8 steps of off-policy is tolerable before training quality degrades significantly.
But long-horizon episodes that take weeks will inevitably exceed the 8-step budget. At that point, without bootstrapping, your only options are:
- Let GPUs idle (wasteful)
- Accept more off-policy drift (degrades learning)
Value-model bootstrapping offers a third option: use the value model to generate intermediate training signals before episodes complete, keeping GPUs productive while introducing some value-model bias. Every approach involves a trade-off between GPU utilization and training quality. There is no free lunch — for more on how compute choices affect your AI stack, see our DeepSeek V4 Flash agentic upgrade guide.
What Infrastructure Does Long-Horizon RL Need?
Long-horizon RL requires treating environments as first-class infrastructure — not an afterthought. The standard approach of training on a single environment doesn't scale; you need hundreds of diverse environments to develop general long-horizon capabilities.
Tools that help bridge the context limitation include:
- File-system tools: A scratchpad where the agent can write reasoning thoughts to disk rather than keeping them in context, freeing up working memory.
- Self-search tools: Allow the agent to search over its previous trajectory, finding relevant earlier decisions without keeping them all in context.
- Archive tools: In tasks like automated research, the agent can build on previous results. But safeguards are needed to prevent the agent from cheating by grabbing previous answers without actually reasoning through them.
Platforms like OpenReward (openreward.ai), launched by General Reasoning in March 2026, host 330+ RL environments behind a single API endpoint, serving both training and evaluation needs for frontier labs and open-source researchers. For more on serving agent infrastructure at scale, see our guide on serving AI agents at scale with MiniMax M3 (General Reasoning, March 2026). This is part of a broader shift toward treating environments as critical infrastructure — alongside algorithms and compute — for the next wave of AI capabilities.
How Fast Are AI Task Horizons Growing?
METR's March 2025 study found that the length of tasks AI agents can complete autonomously at 50% reliability has been doubling approximately every 7 months (196 days, later refined to 187.8 days in a March 2026 update) for the past six years (METR, arXiv:2503.14499).
In practical terms:
- 2019: Tasks taking human experts ~1-2 minutes
- 2022: Tasks taking ~10-15 minutes
- 2024: Tasks taking ~1 hour
- Early 2026: Tasks taking ~12 hours (Claude Opus 4.6, best frontier)
At this rate, reliable multi-day task completion is expected by 2026–2027, and multi-week tasks by 2028–2029. However, METR's data measures software engineering tasks — relatively structured and verifiable. Open-ended, creative, multi-agent tasks (like KellyBench's betting environment) may progress more slowly.
The key insight: the doubling is exponential and consistent, but reliability drops sharply as task length increases. An agent might have a 50% success rate at 12-hour tasks but only a 10% success rate at 24-hour tasks. The "long-horizon wall" isn't a cliff — it's a probability gradient that falls faster than the doubling trend pushes it forward. For a deeper look at what happens when models automate complex workflows, see our guide on what comes after RLHF.
What This Means for You
If you're building AI-powered workflows for your business or clients, here's how to plan around the long-horizon gap:
Design for short horizons today. Break long tasks into sub-tasks that take 30 minutes or less for an AI agent. Use human checkpoints between sub-tasks. Don't expect a single agent call to handle a multi-hour workflow end-to-end.
Verifiable rewards beat open-ended goals. If you can define success as a clear, checkable outcome (code passes tests, answer matches expected output, data format validates), RL-style training and evaluation works. If your task is genuinely open-ended ("research this market and tell me what to do"), expect much worse performance and plan for heavy human oversight.
Watch the doubling curve. If METR's 7-month doubling holds, agents reliable at full workdays arrive by late 2026, and full work weeks by 2028. Plan your product roadmap to absorb these capabilities as they arrive, but don't bet the business on them being ready tomorrow.
Invest in your own RL environments. If you're training or fine-tuning models for your domain, the quality and realism of your training environment matters as much as your algorithms. A well-designed simulation of your actual workflow — with realistic obstacles, multi-agent dynamics, and time-varying conditions — will produce dramatically better agents than a generic coding benchmark.
FAQ
Q: What is a long-horizon task in AI?
A: A long-horizon task is any goal that requires an AI agent to make hundreds or thousands of sequential decisions over an extended period — hours, days, or weeks — rather than answering a single prompt in one shot. Examples include trading a portfolio across a season, debugging a large system end-to-end, or conducting weeks of research and synthesis.
Q: Why do AI models fail at long-horizon tasks?
A: Three bottlenecks: gradient variance in RL training scales with trajectory length (making learning signals noisy), rewards arrive only at the very end (sparse reward / credit assignment problem), and even million-token context windows run out before long tasks finish. These compound: the longer the task, the worse all three problems get.
Q: What is context compaction and how does it help?
A: Context compaction lets an agent fill its context window, summarize the key information, reset the beginning of the context, and continue generating on top of the summary. It extends effective working memory beyond the raw context limit, though it doesn't solve the underlying RL optimization challenges.
Q: What is the difference between GRPO and value-model-based RL?
A: GRPO (used by DeepSeek-R1) computes relative advantages from multiple rollouts of the same prompt, eliminating the need for a separate value model. It's simpler and works well for short-to-medium tasks. Value-model-based RL trains a second network to estimate expected future rewards, enabling per-step feedback and bootstrapping before episode completion — more complex but better suited for long-horizon tasks where waiting for episodes to finish would waste GPU time.
Q: How fast is AI task-completion ability improving?
A: METR measured a doubling time of approximately 7 months for autonomous task completion at 50% reliability. As of early 2026, the best frontier model reliably completes tasks taking a human expert about 12 hours. If the trend continues, multi-day reliability arrives by late 2026 and multi-week by 2028.
Q: What is KellyBench and what did it prove?
A: KellyBench is a benchmark released in April 2026 that gave frontier AI models £100,000 to bet on Premier League football across a full season. Every model lost money, while a human quant achieved +5.1% ROI. It demonstrated that current models collapse on open-ended, multi-agent, non-stationary tasks — exposing the long-horizon capability gap.
Discussion
0 comments