You can run a fully autonomous AI agent on your own machine right now, for free, that edits files, runs commands, browses the web, and never sends a single byte to a cloud API. The breakthrough that made this practical wasn't a new model launch — it was a quiet weight update to Google's Gemma 4 on July 15, 2026 that fixed the tool-calling bugs making local agents unreliable. Combined with Hermes Agent (an open-source agent framework by Nous Research) and Ollama (a local model runtime), the full stack costs zero dollars per token and runs entirely on your hardware.
- Gemma 4's July 15 update fixed tool-calling bugs and added Flash Attention 4 (25–70% faster prompts)
- Full stack: Gemma 4 (model) + Ollama (runtime) + Hermes Agent (agent framework) — all open source
- Minimum hardware: 8 GB RAM for the E4B model; 16+ GB recommended for the 26B MoE
- Total API cost: $0/month — your computer does the inference
- Setup time: ~15 minutes on a clean machine
What changed on July 15, 2026 — and why it matters for local agents
Google shipped a stealth update to Gemma 4 on July 15, 2026, announced via the official Google Gemma account on X (source) and independently corroborated by The Decoder (source). No version bump, no launch event — just new weights on Hugging Face. But the changes inside those weights are what made local agents go from "experimental" to "reliable."
The update delivered four improvements:
| Improvement | What it does | Impact |
|---|---|---|
| Flash Attention 4 | Speeds up prompt processing on Nvidia Hopper GPUs (H100 and later) | 25–70% faster prompt processing, up to 31% faster time-to-first-token |
| Tool-calling bug fixes | Fixes failures and malformed outputs during agentic workflows | Up to +10.1% accuracy on Tau2 Telecom benchmark for 31B |
| Truncated response fix | Reduces cases where the model cuts answers short | Fewer incomplete agent outputs |
| Vision token budget | max_soft_tokens raised from 280 to 1,120 |
Supports images up to 2.51 megapixels for sharper OCR |
The tool-calling fix is the one that matters most. Tool calling is what separates a chatbot from an agent — without it, a model can only talk. With it, a model can open files, search the web, write to a calendar, and execute commands. Before this update, Gemma 4 would sometimes call the wrong tool, send broken data, or stall out completely. Google's published benchmarks for the 31B variant show across-the-board improvements (source):
- BFCL: 74.2% (+0.4)
- TB2: 25.8% (+4.5)
- Tau2 Retail: 77.6% (+3.1)
- Tau2 Airline: 84.0% (+2.0)
- Tau2 Telecom: 62.7% (+10.1)
A 10.1% jump in the telecommunications tool-use scenario is the difference between an agent you can trust to run on its own and one you have to babysit. TechTimes confirmed that if you downloaded Gemma 4 before July 16, 2026, the weights in your local repository are materially different — you must re-pull to get the improvements (source).
What is the full free local AI agent stack?
The stack has three layers, each open source and free:
Gemma 4 is the brain. It's Google DeepMind's open-weight model family, released April 2, 2026 under the Apache 2.0 license — a major shift from the restrictive custom license used by previous Gemma generations (source). The family includes five sizes: E2B, E4B, 12B, 26B A4B (Mixture-of-Experts), and 31B (dense). All accept text and image input; E2B, E4B, and 12B also accept audio. The 26B and 31B support a 256K context window; the smaller models support 128K.
Ollama is the runtime. It downloads and serves quantized models on your machine, exposing an OpenAI-compatible API at http://127.0.0.1:11434/v1 that any agent framework can connect to (source). No GPU required — it runs on CPU, though a GPU makes it dramatically faster.
Hermes Agent is the hands. Built by Nous Research and MIT-licensed, it's a self-improving AI agent framework that runs in your terminal, a desktop app, and on messaging platforms (Telegram, Discord, Slack, WhatsApp, Signal) (source). It has persistent memory across sessions, creates reusable skills from experience, and ships with 70+ built-in skills. Hermes works with any LLM provider — including a local Ollama instance — so you can swap models without changing your workflow.
Which Gemma 4 model should you run locally?
The right model depends on your hardware. Here's the decision table, with memory requirements from Google's official model card (source):
| Model | Ollama tag | Disk size | Min RAM (Q4) | Context | Tool calling | Best for |
|---|---|---|---|---|---|---|
| E2B | gemma4:e2b |
~7.2 GB | 3 GB | 128K | Yes | Phones, Raspberry Pi, low-end laptops |
| E4B (default) | gemma4 |
~9.6 GB | 5 GB | 128K | Yes | Quick local testing, 16 GB Macs |
| 12B | gemma4:12b |
~13 GB | 7 GB | 256K | Yes | Laptops with 16 GB RAM, good balance |
| 26B MoE | gemma4:26b |
~18 GB | 15 GB | 256K | Yes | Consumer GPU (24 GB VRAM) — best quality/speed tradeoff |
| 31B Dense | gemma4:31b |
~20 GB | 17 GB | 256K | Yes | Best quality — strong reasoning and tool use |
For most people setting up a local agent for the first time, the 26B MoE is the sweet spot. It only activates 3.8B parameters per token (out of 25.2B total), so it runs at near-4B speeds while delivering near-30B quality (source). It ranks #6 on the Arena AI open-model text leaderboard with an Elo of 1441 — surrounded by models 10–30x its size (source).
If you have a laptop with 16 GB RAM and no dedicated GPU, start with gemma4:12b or gemma4:e4b. You can always pull a larger model later with a single command. If you're deciding whether to invest in a GPU for this, our GPU buying guide for local AI breaks down the economics.
How to set up the full stack (step by step)
This setup takes about 15 minutes on a clean machine. It works on Linux, macOS, and Windows (via WSL2).
Step 1: Install Ollama
On Linux or macOS, run the official installer:
curl -fsSL https://ollama.com/install.sh | sh
On Windows, download the installer from ollama.com/download and run it. Ollama installs as a background service automatically.
Verify the installation:
ollama --version
You should see a version number. If you get a "command not found" error, make sure Ollama's executable is in your system path (source).
Step 2: Pull the Gemma 4 model
Download the model weights to your machine. For the best quality/speed balance on a machine with 24 GB VRAM:
ollama pull gemma4:26b
For a laptop with 16 GB RAM:
ollama pull gemma4:12b
For a quick test on any machine:
ollama pull gemma4
This downloads the default E4B variant (~9.6 GB). The download is one-time — Ollama caches the model on disk. Verify it pulled successfully:
ollama list
You should see gemma4:26b (or your chosen tag) in the list.
Critical: re-pull if you downloaded before July 16, 2026. The stealth update shipped new weights under the same tag name. Run ollama pull gemma4:26b again to get the tool-calling fixes — Ollama downloads only what changed, similar to how Docker handles image layers (source).
Step 3: Install Hermes Agent
On Linux, macOS, or WSL2:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
This installs uv, Python 3.11, clones the repository, and sets up everything automatically. No sudo needed (source).
Alternatively, Ollama can handle the entire setup for you with a single command:
ollama launch hermes
Ollama installs Hermes, helps you pick a model, configures the Ollama provider (pointing Hermes at http://127.0.0.1:11434/v1), and optionally connects a messaging platform (source).
Step 4: Connect Hermes to your local model
If you used ollama launch hermes, you're done — skip to Step 5. If you installed Hermes manually, run the setup wizard:
hermes setup
Choose Quick setup, then:
- Select More providers...
- Select Custom endpoint (enter URL manually)
- Set the API base URL to:
http://127.0.0.1:11434/v1 - Leave the API key blank (not required for local Ollama)
- Hermes auto-detects your downloaded models — confirm the Gemma 4 tag you pulled
Important context window setting: Hermes requires at least 64,000 tokens of context or it rejects the model at startup. Start your Ollama model with at least 64K context (source):
ollama run gemma4:26b -c 65536
Or set it permanently via a custom Modelfile:
FROM gemma4:26b
PARAMETER num_ctx 65536
Save as Modelfile, then run ollama create gemma4-64k -f Modelfile and use gemma4-64k as your model name in Hermes.
Step 5: Start your agent
hermes
You now have a full working agent running on your own hardware. No cloud, no account required for the model, nobody else sees what you're working on. Test it by asking it to do something that requires tools — "create a file called notes.txt and write three ideas for a blog post in it" — and watch it call the file-write tool.
Step 6 (optional): Connect a messaging platform
Hermes can run on Telegram, Discord, Slack, WhatsApp, Signal, and Email — all from a single gateway process. This means you can talk to your local agent from your phone while it runs on your computer at home:
hermes gateway setup
hermes gateway
Follow the interactive wizard to connect your preferred platform (source).
Local vs cloud: why running local actually matters
The appeal of local AI isn't just cost — it's control, privacy, and reliability. Here's the honest comparison:
| Factor | Cloud API (e.g., GPT, Claude) | Local (Gemma 4 + Ollama) |
|---|---|---|
| Cost per token | $0.15–$15 per million tokens | $0 (your electricity only) |
| Privacy | Prompts and files travel over the internet | Nothing leaves your machine |
| Rate limits | Can be rate-limited mid-task | No limits — your hardware is the only ceiling |
| Internet dependency | Stops working if connection drops | Keeps running offline |
| Model intelligence | Higher (frontier models like GPT-5.6, Claude Fable 5) | Lower, but improving fast — Gemma 4 31B ranks #3 on Arena AI open leaderboard |
| Setup effort | Zero — just an API key | ~15 minutes one-time setup |
| Speed | Fast (optimized inference) | Depends on hardware; GPU recommended |
The cost difference compounds in agent loops. An agent that thinks, acts, checks its own work, and loops again will make dozens of API calls per task. On a paid model, that usage climbs fast. On Gemma 4 running locally, it costs nothing extra because your own computer is doing the work. For more on why API pricing is falling and local is becoming viable, see our analysis of the 2026 AI model pricing war.
The hybrid approach: best of both worlds
You don't have to choose all-local or all-cloud. The smartest setup for most people is a hybrid pipeline: use a stronger cloud model for planning and complex reasoning, then hand the output to Gemma 4 running locally for execution and repetitive work.
Here's a practical example. Say you want to write a detailed product description:
- Send your outline to a frontier cloud model (Claude, GPT) — it writes the structure and key arguments
- Hand that outline to Gemma 4 running locally — it writes the full draft, iterates on it, and formats the output
- Total cost: one small cloud API call for planning + zero cost for the bulk writing
This pattern works for any task with a "think hard once, then execute many times" shape: research, content production, code scaffolding, data processing. The expensive reasoning happens once in the cloud; the repetitive execution happens free on your machine. If you want to explore other free AI tool setups, our guide to running Claude Code for free covers a similar hybrid pattern.
What can you actually do with a local agent?
The combination of Gemma 4's fixed tool calling and Hermes Agent's framework opens up real, practical workflows — not toys:
- Automated content production: Research a topic, outline an article, draft it, and save it to a file — all locally, all free. Hermes's skill system means once you've built a content workflow, you can reuse it with a single command.
- Code assistance: Navigate a codebase, suggest changes, run tests, and explain how different parts work. Gemma 4 handles the reasoning while Hermes manages file operations and command execution.
- Scheduled automation: Hermes supports cron-style scheduling. Your local agent can monitor a calendar, prepare summaries before meetings, draft routine responses, and track action items — all in the background on your machine.
- Research assistant: An always-on agent that monitors topics you care about, collects information from the web, and summarizes findings when you check in.
- File and system management: Organize files, clean up directories, run maintenance scripts, and manage your workspace — the kind of repetitive system admin work that eats time.
The tooling around local models keeps improving independently of Gemma 4 itself. Other frameworks have shipped updates that make tool calls more reliable, cut down on duplicate loops (where an agent repeats the same action), and add self-healing tool calling where broken calls get automatically corrected instead of failing outright. All of that stacks on top of what Gemma 4 already fixed. For a deeper look at building agents that last, our agent architecture survival guide covers the structural decisions that matter.
How fast is local inference — and can you go fully local?
Speed depends entirely on your hardware. On CPU only, a 9B model gives roughly 10 tokens/second on a modern 8-core processor. A 31B model on CPU is slower — about 2–5 tokens/second, meaning each response takes 30–120 seconds (source). A GPU changes this dramatically.
For CPU-only setups, Hermes recommends widening the API timeout since local inference is slower:
# ~/.hermes/.env
HERMES_API_TIMEOUT=1800 # 30 minutes — generous for slow local models
Can Gemma 4 replace a frontier cloud model completely? Not yet. The 31B dense model achieves an Arena AI Elo of 1452 — impressive for its size, but still below frontier models like GPT-5.6 or Claude Fable 5 in raw reasoning capability. For tasks that require deep multi-step reasoning or the absolute best code generation, a cloud model still wins. But for the 80% of agent work that is repetitive execution — file operations, web browsing, formatting, drafting, scheduling — Gemma 4 is now good enough, and the price is impossible to beat. If you're exploring other free AI options, our Kimi K3 free setup guide covers another zero-cost path.
What this means for you
If you've been paying per token for AI agent work that a free local model could handle, the math just changed. The July 2026 Gemma 4 update closed the reliability gap that made local agents frustrating to use. The setup takes 15 minutes. The ongoing cost is zero. And the hybrid pattern means you can keep using frontier cloud models for the hard thinking while offloading everything else to your own machine.
The people getting ahead right now are the ones setting this up today, not waiting for it to get easier. Every week you put this off is a week you're still paying token costs on tasks a free local model could already be handling.
FAQ
Q: Is running a local AI agent really free? A: Yes. Gemma 4 is Apache 2.0 licensed, Ollama is free, and Hermes Agent is MIT licensed. The only cost is your computer's electricity. There are no API keys, no subscriptions, and no per-token charges. Your machine does all the inference work locally.
Q: Do I need a GPU to run a local AI agent? A: No, but it helps significantly. Ollama runs on CPU-only machines — a 9B model gives roughly 10 tokens/second on a modern 8-core CPU. A GPU with 8+ GB VRAM dramatically improves speed. For the 26B MoE or 31B dense models, a GPU with 24 GB VRAM is recommended. The E2B and E4B variants run on laptops and even Raspberry Pi without a GPU.
Q: What is tool calling and why does the Gemma 4 fix matter? A: Tool calling (also called function calling) is what lets an AI model trigger external actions — opening files, searching the web, running commands. Without reliable tool calling, a model is just a chatbot. The July 2026 Gemma 4 update fixed bugs that caused tool-calling failures, with accuracy improvements up to 10.1% on certain benchmarks. This is what made local agents go from experimental to reliable.
Q: Can I use a different model instead of Gemma 4? A: Yes. Hermes Agent works with any Ollama-served model, as well as cloud providers like OpenRouter, OpenAI, and Anthropic. Other popular local models include Qwen 3.5 and Carnice 35B A3B (specifically tuned for tool calling). The key requirement is that the model supports tool calling for the full agent experience.
Q: Will my data stay private with a local setup? A: Yes. With a local Ollama + Hermes setup, no prompts, files, or conversation data leave your machine. This is a major advantage over cloud APIs where your data travels over the internet to a third-party server on every request. The only exception is if you connect Hermes to a messaging platform (Telegram, Discord, etc.) — those messages travel through the platform's servers, but the model inference still happens locally.
Q: How is this different from just using ChatGPT or Claude? A: ChatGPT and Claude are cloud-hosted chat assistants. A local AI agent setup gives you three things they can't: zero per-token cost (important for high-volume agent loops), complete data privacy (nothing leaves your machine), and full autonomy (no rate limits, no internet dependency). The tradeoff is lower raw intelligence — Gemma 4 is strong but not at frontier-model levels. The hybrid approach gets you the best of both.
Discussion
0 comments