Verdict: The "base model" — a large language model pre-trained on vast quantities of scraped web text to encode general human knowledge — is not dead, but its job description has changed entirely. Between 2020 and 2026, the proportion of raw web text in frontier pre-training data fell from roughly 85% to as low as 15%, replaced by code, math, synthetic data, and instruction-formatted content. The base model's purpose is no longer to be an encyclopedia of the internet; it is to provide a set of atomic skills and representations that reinforcement learning (RL) can then sharpen into a capable reasoning agent.
Last verified: 2026-08-01 · Base models shifted from knowledge encyclopedias to reasoning/agentic priors · Web text share dropped from ~85% (GPT-3) to ~15% in some 2025–26 models · RL compute now rivals or exceeds supervised learning in frontier model budgets · Synthetic data is mainstream in pre-training · "Mid-training" emerged as a distinct training phase
What Was the "Base Model" and Why Is It Changing?
The base model was the product of pre-training: the phase where a transformer learns to predict the next token across trillions of words scraped from the web. The assumption was that the more diverse, high-quality text the model consumed, the better its "world knowledge" and general capabilities would be. Post-training (instruction tuning, RLHF) was a relatively light overlay — shaping the model's output format and tone so it could function as a chat assistant.
That assumption drove the first generation of LLMs. GPT-3 (2020) was trained on a data mix that was approximately 60% filtered Common Crawl, 22% WebText2 (Reddit outbound links with 3+ upvotes), 16% books corpora, and 3% Wikipedia (Brown et al., 2020; Mozilla Foundation, 2024). Web text, including Wikipedia, made up roughly 85% of the total training tokens. Pre-training was the bulk of the compute budget, and post-training was "a cherry on top" — shaping interactions without fundamentally changing what the model knew.
By 2024–2026, that recipe had been turned on its head. What comes after RLHF is no longer just human feedback — it is large-scale reinforcement learning with verifiable rewards, and it is eating a growing share of the compute budget.
How Much Has Pre-Training Data Actually Changed?
The shift is most visible in the disclosed data mixes of open-weight models:
| Model | Year | Web Text Share | Code Share | Math/Reasoning Share | Synthetic Data | Source |
|---|---|---|---|---|---|---|
| GPT-3 | 2020 | ~85% (Common Crawl + WebText + Wikipedia) | 0% (no dedicated code) | Minimal | None | GPT-3 paper |
| Llama 3 (405B) | 2024 | ~50% (general knowledge) | 17% | 25% (math + reasoning) | Limited | Llama 3 paper |
| Kimi K2 | 2025 | Reduced (Web Text is one of four domains) | Significant (Code domain) | Significant (Math domain) | Large-scale rephrasing pipeline | Kimi K2 paper |
Kimi K2, released July 2025 by Moonshot AI, is a 1-trillion-parameter Mixture-of-Experts model that was pre-trained on 15.5 trillion tokens. Its pre-training data spans four domains — Web Text, Code, Mathematics, and Knowledge — with a synthetic data rephrasing pipeline that amplifies high-quality tokens without inducing overfitting (Kimi K2 technical report, arXiv:2507.20534). The rephrasing approach works: K2's team found that rephrasing data 10 times with a single training pass outperformed raw text repeated for 10 epochs on SimpleQA accuracy (28.94% vs. 23.76%).
The trend is clear: code and STEM data — which teach the model transferable reasoning structures — have partially displaced generic web text. The web is no longer the primary source of capability.
Why Did Web Text Lose Its Dominance?
Three forces drove the shift away from raw web text as the dominant pre-training ingredient:
1. Web text has diminishing returns. Once a model has seen a sufficiently large corpus of internet text, additional web pages teach it more facts but not more reasoning. The Llama 3 team found that the most impactful data for downstream performance was math, reasoning, and code — not more web pages (Llama 3 paper, arXiv:2407.21783). They used learning-rate annealing on high-quality domain-specific data at the end of pre-training, and found that this improved Llama 3 8B's performance on GSM8k (grade-school math) by 24.0%.
2. Synthetic data got better — and safer. A large-scale 2025 study by FAIR at Meta (Kang et al.) trained over 1,000 LLM variants across more than 100,000 GPU hours to systematically evaluate synthetic data in pre-training. The key finding: mixing approximately 30% rephrased synthetic data with 70% natural web text can accelerate convergence by 5–10× to reach the same validation loss, with no signs of model collapse on rephrased data at practical scales (Kang et al., EMNLP 2025). This is not theoretical — it is deployed at scale in models like Kimi K2.
3. RL changed the target. When OpenAI released o1 in September 2024 and DeepSeek released R1 in January 2025, they demonstrated that reinforcement learning could dramatically improve a model's reasoning and agentic performance — far beyond what post-training alignment previously achieved (OpenAI o1 blog, 2024; DeepSeek R1 paper, arXiv:2501.12948). If RL is where capabilities are unlocked, the base model's job is to provide the raw materials — atomic skills like code execution, step-by-step reasoning, and tool-use patterns — that RL then composes into complex behavior.
What Is "Mid-Training" and Where Does It Fit?
A new phase has emerged between traditional pre-training and post-training: mid-training. A 2025 survey from Peking University and Meituan defines it as "a distinct phase that bridges general pre-training and specialized post-training" with a "transitional intent" — amplifying specialized skills like math and coding while preserving general competencies through careful data mixture management (Tu et al., arXiv:2510.23081).
Mid-training delivers steeper performance gains with less data and compute than pre-training. IBM Research published a study in April 2026 called PRISM that found mid-training with just 27 billion tokens (small compared to pre-training's 15+ trillion) significantly improved reasoning capabilities — but the data mixture mattered enormously (IBM Research, 2026).
The practical implication: the old three-phase pipeline (pre-train → fine-tune → align) is becoming a four-phase pipeline (pre-train → mid-train → post-train with SFT → RL), and the boundaries between phases are blurring. Some labs are pulling post-training-style SFT data back into pre-training itself, so the model encounters instruction-formatted content from the very beginning.
How Is Reinforcement Learning Changing the Compute Budget?
The most consequential shift is not in data mixes — it is in where the compute goes:
| Era | Pre-Training Compute | Post-Training Compute | RL's Role |
|---|---|---|---|
| GPT-3 era (2020) | ~95% of budget | ~5% | None |
| ChatGPT era (2022–2023) | ~80–90% | ~10–20% | RLHF for alignment |
| Reasoning era (2024–2026) | ~50–60% | ~40–50% | RL with verifiable rewards; can exceed supervised learning compute |
Research from 2025–2026 shows that post-training has grown into a massive undertaking. A 2026 position paper from researchers at multiple universities argues that "post-training has become such a critical component [that it] has effectively transformed back into a form of supervised learning," consuming hundreds of billions of tokens (arXiv:2606.07527). The paper shows that models post-trained from scratch (without pre-training) achieve non-trivial performance, suggesting that what was once a light touch-up is now a fundamental capability-building phase.
This aligns with what the head of Xiaomi's MiMo Labs described: pre-training and post-training in the final model have roughly equal compute allocation. Some models take this further — concentrating more RL compute into the model than it ever saw in supervised learning.
The post-training pipeline for agentic models is now where models learn their most valuable skills: tool use, multi-step reasoning, code execution, and environment interaction. AI agents that work with real tools depend on RL-shaped capabilities, not raw memorized knowledge.
For builders, this means the data quality bottleneck has shifted: it is no longer just about how much web text you can scrape — it is about how good your RL environment, reward signal, and synthetic data generation pipelines are. See also how to train AI agents without verifiable rewards.
What Is the New Paradigm for Understanding LLM Training?
Rather than thinking in terms of "pre-training vs. post-training" or "base model vs. aligned model," it is more useful to think in terms of two fundamental learning paradigms:
Supervised learning (next-token prediction): The model learns to predict the next token across a curated mix of web text, code, math, synthetic data, and instruction-formatted content. This builds foundational representations and atomic skills.
Reinforcement learning: The model interacts with environments, receives rewards for correct behavior, and learns to compose its atomic skills into complex capabilities — reasoning chains, tool use, multi-step problem solving.
The supervised learning phase is increasingly being optimized for RL, not as an end in itself. Research from 2025 shows that the base model needs exposure to the atomic skills it will compose during RL, and given a sufficiently challenging environment, it can learn to extrapolate from there (e3: Learning to Explore, arXiv:2506.09026). This is a fundamental reframing: pre-training is no longer about building the best possible model — it is about building the best possible starting point for RL.
What Does Synthetic Data Actually Do in Pre-Training?
Synthetic data in pre-training is not about blindly generating text and feeding it to the model. Two approaches have proven effective at scale:
Rephrasing: Take a seed data item and generate synthetic rephrasings of the same information, so the model sees the same content expressed in multiple ways. This increases token utility without overfitting. Kimi K2 deployed this at scale across its entire pre-training corpus. FAIR's study found the optimal ratio is approximately 30% rephrased synthetic data mixed with 70% natural web text (Kang et al., EMNLP 2025).
Textbook-style generation: Generate entirely new educational content (textbooks, worked examples, reasoning traces) to teach specific skills. Microsoft's Phi series pioneered this approach. However, the FAIR study found that textbook-style synthetic data alone results in notably higher loss on many downstream domains — it works best as a supplement, not a replacement for natural text.
The key insight is that synthetic data is not a monolith. Rephrased data scales well; purely generated data risks model collapse if used without natural text. The choice of training data is now an engineering discipline, not just a data collection exercise.
What This Means for You
If you are building with LLMs — whether you are fine-tuning open models, choosing between API providers, or deciding which capabilities to build in-house — this shift has practical consequences:
Model selection is now about RL quality, not just pre-training scale. A smaller base model with excellent post-training RL can outperform a larger model with mediocre post-training. Evaluate models on your actual tasks (reasoning, agentic tool use, multi-step problem solving), not on parameter counts or pre-training token counts.
Synthetic data is a tool you can use. If you are fine-tuning or building domain-specific models, rephrasing pipelines can amplify your training data. The ~30% synthetic / 70% natural ratio is a starting point validated at Meta scale. You do not need a frontier lab's budget to benefit.
The "web text = knowledge" mental model is outdated. A model trained primarily on web text will be a better encyclopedia but a worse reasoner. If your use case involves code, math, logic, or agentic workflows, look for models whose training data and post-training emphasize those domains.
Mid-training is a lever for specialization. If you are adapting a base model to a specific domain, mid-training (high-quality domain data at a moderate compute budget, between pre-training and full post-training) can deliver better results per FLOP than either expanding pre-training or aggressive fine-tuning.
Your RL environment matters more than your model size. The biggest performance gains in 2025–2026 came from better RL training environments (reward signals, tool-use sandboxes, multi-turn agentic trajectories) — not from bigger base models. Invest in the evaluation and environment infrastructure.
FAQ
Q: Is the base model truly dead? A: No, but its role has fundamentally changed. The base model is no longer the primary source of a model's capabilities — it is the starting point that RL builds on. Pre-training still matters (models post-trained from scratch underperform models with good pre-training), but the balance of compute and data has shifted toward post-training RL and synthetic data.
Q: How much has the pre-training data mix changed since GPT-3? A: Dramatically. GPT-3's training data was approximately 85% web text (Common Crawl, WebText, Wikipedia). By 2024, Llama 3 reduced general-knowledge web text to ~50% and added 25% math/reasoning and 17% code. Some 2025–26 models have pushed web text as low as 15% of the mix, replacing it with code, math, and synthetic data.
Q: Can synthetic data really replace web text in pre-training? A: Partially. Research shows that rephrased synthetic data mixed at ~30% with 70% natural web text can accelerate convergence 5–10×. However, purely synthetic textbook-style data alone causes higher loss on downstream tasks and can lead to model collapse. Synthetic data is a powerful supplement, not a full replacement.
Q: What is mid-training and why should I care? A: Mid-training is a distinct training phase between pre-training and post-training that amplifies specialized capabilities (math, code, reasoning) while preserving general competencies. It delivers steeper performance gains per compute unit than pre-training. For domain-specific models, it is a high-leverage step.
Q: Is reinforcement learning now more important than pre-training? A: It depends on the model's purpose. For reasoning and agentic models, RL compute now rivals or exceeds supervised learning compute. But RL depends on a good base model — pre-training provides the atomic skills RL composes. The two are complementary: pre-training builds the foundation, RL builds the capabilities.
Q: What should builders look for when choosing an LLM in 2026? A: Look beyond parameter count and pre-training scale. Evaluate the model's post-training quality (RL environments, synthetic data pipelines, agentic benchmarks like SWE-Bench and Tau-Bench). A smaller model with better post-training will often outperform a larger model with weaker post-training on reasoning and tool-use tasks.
Discussion
0 comments