Running AI agents locally on phones — rather than in the cloud — solves three problems at once: it eliminates the round-trip latency that breaks real-time gameplay, keeps player data private on the device, and unlocks something entirely new: a game that dynamically adapts its accessibility settings to each player's physical and cognitive needs as they play. The approach, called local agentic theory, treats an AI agent as a perceiving-predicting-acting loop that runs entirely within a mobile device's frame budget, using on-device neural processing to sense player frustration and rewrite the game's interface live.
Last verified: 2026-07-29
- On-device AI inference eliminates the cloud round-trip, shrinking agent decisions to within the 16.67ms frame budget at 60 Hz (Android Developers).
- Apple Core ML and Google ML Kit let models run entirely on-device using the Neural Engine (ANE) and Tensor chips — no internet required (Apple Core ML docs; Google ML Kit).
- WCAG 2.2, the current international standard, now faces an evolution to WCAG 3.0 with a graded Bronze/Silver/Gold model that drops the binary pass/fail (W3C WCAG 3.0 draft).
- Reinforcement learning models like EfficientZero V2 achieve state-of-the-art sample efficiency but still can't reason like a general agent — a gap measured by the ARC-AGI benchmark (ICML 2024; ARC Prize 2024 report).
Why run AI agents on-device instead of in the cloud?
Cloud-based AI inference requires two network round-trips per decision: the device sends game state upstream, the model processes it, and the result comes back. For a game running at 60 frames per second, that's 60 potential decisions per second — each one waiting on a network call. On-device inference compresses that to the time it takes the device's own chips to compute, which is measured in single-digit milliseconds on modern hardware. Core ML models can achieve sub-10ms latency for vision tasks on the Apple Neural Engine (Inference Systems comparison).
The three advantages are practical, not theoretical — and they extend the 5-Layer Local AI Stack that builders are already using for edge inference:
- Speed — No HTTP round-trip. The agent perceives, predicts, and acts within the 16.67ms frame window at 60 Hz. If the player is in a subway tunnel with no signal, the game still works — same as any local AI setup.
- Privacy — Player biometric data (gaze tracking, tap patterns, processing speed) never leaves the device. No telemetry upload required.
- Personalization — A local agent can learn one player's habits and adjust the game to them over time, without sending that personal data to a server — the same pattern that makes self-improving AI agents powerful for individual workflows.
How is an agentic AI system different from reinforcement learning?
Reinforcement learning (RL) trains a model by running thousands of iterations that adjust the model's internal weights until it plays a game well. Think of DeepMind's AlphaGo, which learned Go through self-play and defeated world champion Lee Sedol in 2016 (Google DeepMind). AlphaZero generalized this to chess and shogi. MuZero went further, learning without being told the rules. All of these are weight-changing systems: the model itself is modified during training.
An agentic system takes a fundamentally different approach. Instead of grinding through iterations to change weights, the agent uses in-context learning: it reasons over the game's current state using a language model, makes tool calls to interact with the game, and adapts to new situations dynamically. There's no reward system to grind out. This is the same perceive-decide-act loop that powers production agents in other domains — just compressed into milliseconds.
The agentic loop: perceive → predict → decide → act
An agentic game system runs a continuous loop:
| Step | What happens | Constraint |
|---|---|---|
| Perceive | The agent reads the current game state (screen, grid, positions) | Must fit in memory budget |
| Predict | The agent predicts what will happen next (enemy moves, bullet trajectories) | Must complete within frame time |
| Decide | The agent chooses an action based on the prediction | Must not exceed energy budget |
| Act | The agent executes the action in the game | Must not cause visible jank |
A simple example: an agent playing Space Invaders perceives the row of invaders approaching, predicts which one will fire next, decides to move left if a bullet is approaching, acts by triggering the movement, then loops back to perceive the new state. The entire loop must finish within one frame.
Sample-efficient RL vs. agentic reasoning
EfficientZero V2, published at ICML 2024, is the state-of-the-art in sample-efficient reinforcement learning — it learns complex game tasks with far less data than traditional RL models, outperforming DreamerV3 on 50 of 66 benchmark tasks across Atari, Proprio Control, and Vision Control (EfficientZero V2, PMLR 235). But even EfficientZero V2 is still a weight-changing system: it needs training cycles to become good at a game.
An agentic system doesn't need training cycles for a specific game. It reasons about the game at runtime using general intelligence — which is where the ARC-AGI benchmark matters. The ARC Prize 2024 competition saw state-of-the-art scores rise from 33% to 55.5% on the private evaluation set, with the Grand Prize for 85% remaining unclaimed (ARC Prize 2024 Technical Report). The benchmark measures fluid intelligence — the ability to solve novel tasks with minimal prior examples — which is exactly what an agentic game player needs. Until base model intelligence improves significantly, agents will still struggle with the most complex game scenarios.
What are the hardware constraints for local agentic game AI?
On-device agents face a three-way constraint that the video's framework calls a constraint graph: space, time, and energy. Each one pulls against the others, and the agent's design must balance them with soft constraints (penalize lightly if you need a little extra) and hard constraints (penalize heavily if you cross a threshold that breaks the experience).
The three constraints
| Constraint | What it limits | What happens if exceeded | Budget |
|---|---|---|---|
| Space | Model weights, state history, planning artifacts | Out-of-memory crash; cannot fit the agent | Device RAM (typically 4-12 GB, shared with game) |
| Time | Agent planning per frame | Visible jank, stuttering, dropped frames | 16.67ms per frame at 60 Hz (~8ms at 120 Hz) |
| Energy | CPU/NPU compute cycles | Battery drains faster; phone overheats | Battery capacity (~15-20 Wh, shared with game + OS) |
Active game rendering competes for the same hardware budget. The agent must leave enough render headroom for the game itself to draw its frame. On a 60 Hz display, the frame budget at 60fps is approximately 16ms — the operating system asks the renderer for a new frame every 16.67ms, and missing that deadline causes the display to repeat the previous frame, which users perceive as jank (Android Developers — Slow rendering). On 120 Hz displays, that budget shrinks to approximately 8ms.
The solution is a constraint optimization function that uses soft constraints: lightly penalize overuse in one dimension if needed, but heavily penalize time overruns because exceeding the frame budget directly degrades the user experience. An agent that uses slightly too much memory is tolerable; one that causes stuttering is not.
Where NPUs fit in
Neural Processing Units (NPUs) are dedicated AI chips inside modern phones — Apple's Neural Engine (ANE) and Google's Tensor chips. They run neural network inference more efficiently than the CPU or GPU, using less power per operation. That directly addresses the energy constraint: an agent that routes its model evaluation through the NPU instead of the GPU leaves more render headroom and drains the battery more slowly. Core ML automatically routes inference to the ANE, GPU, and CPU in the most efficient configuration (Apple Core ML documentation), and ML Kit supports on-device inference on Android via Google Edge TPU (Google ML Kit). This on-device pattern mirrors the broader shift toward edge physical AI that's bringing inference closer to the user across both consumer and industrial contexts.
Gaze estimation models
A newer capability: on-device convolutional neural networks can track a player's eye gaze and feed it into the agent's decision loop. Gaze data tells the agent where the player is looking — and crucially, where they're struggling to look. If a player with low vision keeps scanning a crossword grid without landing on a cell, that's a signal the agent can act on. This is input the cloud-based approach can't easily get, because it requires continuous, real-time camera/gaze data that can't be uploaded at 60 Hz.
How can on-device AI agents improve game accessibility?
This is where the real breakthrough lies. Traditional accessibility in games follows a static model: an easy mode toggle, a font-size slider, a color-blind filter. Those are fixed menus applied before the game starts. An on-device AI agent can make accessibility dynamic — sensing the player's needs in real time and adjusting the game as they play.
The WCAG foundation
The approach grounds itself in WCAG 2.2, the current international web accessibility standard, which evaluates interactions across four pillars: content must be perceivable, operable, understandable, and robust (W3C WCAG 2.2). This standard was designed for web content, but its principles apply directly to game interfaces.
WCAG 3.0, currently in draft, is evolving the conformance model from a binary pass/fail (A/AA/AAA) to a graded Bronze, Silver, and Gold scoring system. This lets accessibility be measured on a scale rather than a checkbox — a critical shift that matches the agentic approach, where accessibility is a continuously adjustable dial, not a one-time pass (W3C WCAG 3.0 draft).
Three things a local agent can do that a static menu can't
1. Detect and fix focus traps live. When a player using a keyboard or switch device tabs through a game dialogue and hits a broken loop — focus cycling endlessly between two elements with no exit — the agent can monitor the focus path, detect the loop, and inject an exit route in real time, on-device. No cloud round-trip. No static menu can do this because the problem only appears during live interaction.
2. Dynamically resize targets. A player with limited dexterity or "fat fingers" will mistap tiny grid cells. Instead of shipping a fixed large-target mode, the agent measures the interface on the fly, catches targets that are too small for the current player's tap accuracy, and dynamically resizes the controls. The agent rewrites the layout to adapt to the human, not the other way around.
3. Tune the difficulty/accessibility dial continuously. Three core parameters that an agent can adjust in real time:
| Parameter | What it controls | Who it helps |
|---|---|---|
| Input tolerance | How forgiving physical interactions are (bigger hit zones, longer press windows) | Players with motor challenges |
| Step granularity | How many sub-steps a complex task is broken into | Players with cognitive barriers |
| Time pressure | How much time pressure the game applies | Players with processing-speed needs |
The agent dynamically calculates these dials as they rise and fall together, balancing against the player's live context. A player who starts strong but fatigues gets more input tolerance as the session progresses — automatically, without stopping to open a settings menu.
What do you need to build a local agentic game system?
The perceive-predict-act loop in practice
A constraint satisfaction agent solving a crossword puzzle demonstrates the pattern:
- Perceive the grid: read the cell letters, the across/down clues, and the current fill state.
- Build a constraint graph: each cell has constraints (it must fit both the across and down word). Each word has constraints (it must match the clue and fit the letter limit).
- Evaluate candidates: the agent uses natural language reasoning to propose words that fit the constraints.
- Backtrack efficiently: when the constraint graph gets too crowded (too many conflicts), the agent uses natural language from the graph to backtrack — finding the wrong word, removing it, and trying the next candidate.
- Act: fill in the correct word. Loop back to perceive the new state.
This architecture, where an agent OS manages hardware constraints, model inference, and a shared state representation, is particularly well-suited to mobile game scenarios where the agent must coexist with the game engine for limited compute. The general perceptron comes from its design — the same agent can reason about crossword puzzles or shooters, adapting the game to the human. This is the same principle behind perception agents in other on-device contexts: the agent observes the environment, makes a decision, and acts — all within the constraints of the device.
The model landscape
| Model type | Example | Role in a local agent | Status |
|---|---|---|---|
| Generalist agent | Google DeepMind's SIMA (Scalable Instructable Multiworld Agent) | Follows natural-language instructions to play across multiple 3D game worlds | Research stage; trained on 9+ games across 8 studios (DeepMind SIMA) |
| Sample-efficient RL | EfficientZero V2 | Pre-trained model that learns games with minimal data | State-of-the-art, published ICML 2024 (PMLR) |
| On-device inference | Core ML / ML Kit / TensorFlow Lite / ONNX Runtime | Runs the model locally on the phone's NPU/GPU | Production-ready on iOS and Android |
| Gaze estimation | On-device CNN | Tracks eye movement for accessibility signals | Maturing; requires NPU for real-time use |
| Foundational game AI | Finite state machines (Pac-Man, 1980) | Hard-coded behavior switching — still the backbone of most game AI | Established since the 1980s; each Pac-Man ghost used a distinct deterministic algorithm with just 2KB of memory |
Pac-Man's ghost AI, created by Toru Iwatani at Namco in 1980, used finite state machines with chase, scatter, and frightened modes — a form of symbolic AI where each ghost had a distinct personality algorithm (Blinky chased directly, Pinky ambushed, Inky moved unpredictably, Clyde seemed random). That 1980s approach is still the backbone of most game AI today. Local agentic theory is the next evolution: instead of a fixed state machine, a reasoning agent adapts its behavior dynamically at runtime.
What still needs to be built for local agents to reach their potential?
Five gaps remain before local agents can handle complex, real-time games:
- Speed — Plans and decisions must complete within a 16ms frame budget to prevent stuttering. Current models are close on simple games but not on complex ones.
- Prediction — Models need to simulate the game state to see what a layout change will do before making it. This is a world-model problem.
- Long-term memory — The real magic will be an agent that learns one specific person's habits and needs over time. Persistent on-device memory for agents is still nascent.
- Shared game state language — One agent should work across multiple games without being rebuilt from scratch for each release. A shared protocol for game state representation doesn't exist yet — the same gap that single-agent vs multi-agent architecture faces in other agent domains.
- Hardware + benchmarks — Faster NPU chips need to be paired with real benchmarks that prove the agent makes the game better, not just different.
What this means for you
If you're building a game or app with AI features, the on-device agentic approach means:
- For game developers: Start prototyping on-device agent loops with Core ML or ML Kit. Even a simple perceive-predict-act agent can improve accessibility without a settings menu. Budget for the NPU and leave render headroom.
- For accessibility advocates: Dynamic accessibility is a fundamentally better model than static toggles. The WCAG 3.0 Bronze/Silver/Gold shift mirrors this evolution — push for its adoption in gaming the same way.
- For AI engineers: The constraint graph (space × time × energy) is your design canvas. Soft-constrain space and energy; hard-constrain time. Gaze estimation via on-device CNN is an emerging input channel that cloud-based systems can't replicate. The free local AI agent stack with Gemma 4 is a good starting point for prototyping.
- For product leaders: Local agents create a privacy story (data never leaves the device) and a personalization story (the agent learns the individual) that cloud-based competitors can't easily match. Both are differentiators in 2026.
FAQ
Q: What is local agentic theory for mobile games?
A: Local agentic theory is the approach of running AI agents — systems that perceive, predict, decide, and act — entirely on a mobile device's own hardware (NPU, GPU, CPU) rather than in the cloud. The agent runs a continuous loop within the device's frame budget (about 16.67ms at 60 Hz), eliminating network latency and enabling real-time adaptation, including dynamic accessibility adjustments.
Q: How is an agentic system different from reinforcement learning?
A: Reinforcement learning trains a model by running thousands of iterations that change the model's internal weights (like AlphaGo learning Go through self-play). An agentic system reasons over the game's current state at runtime using in-context learning via a language model — there's no weight-changing training loop. The agent adapts dynamically to new situations using tool calls and reasoning, not pre-trained reward optimization.
Q: Can on-device AI agents work without an internet connection?
A: Yes. On-device models run entirely on the phone's own chips (Apple's Neural Engine or Google's Tensor NPU) using frameworks like Core ML and ML Kit. No cloud round-trip is needed, which means the agent works in subway tunnels, on airplanes, or anywhere with no signal — as long as the model fits within the device's memory and energy budget.
Q: What is the 16ms frame budget and why does it matter for AI agents?
A: At a 60 Hz refresh rate, the operating system requests a new frame approximately every 16.67 milliseconds — that's the time budget for all rendering and compute within one frame. If an AI agent's planning step takes longer than this, the frame is dropped and the user perceives stuttering (jank). On 120 Hz displays, the budget shrinks to roughly 8ms, making the constraint even tighter (Android Developers).
Q: How can AI agents improve accessibility in mobile games?
A: Instead of static accessibility menus (easy mode, large text), an on-device agent monitors the player in real time using gaze tracking, tap-pattern analysis, and processing-speed signals. It can detect focus traps (keyboard loops) and inject exit routes live, dynamically resize too-small touch targets based on actual tap accuracy, and continuously adjust input tolerance, step granularity, and time pressure as the player's needs change during play.
Q: What is WCAG 3.0 and how does it relate to agentic accessibility?
A: WCAG 3.0 is the next version of the international web accessibility standard, currently in draft. It replaces the binary A/AA/AAA pass/fail model with a graded Bronze, Silver, and Gold scoring system that measures accessibility on a scale (W3C draft). This mirrors the agentic approach, where accessibility is a continuously adjustable dial rather than a one-time checkbox — the agent tunes the game to the player in real time, which WCAG 3.0's graded model can capture.
Discussion
0 comments