Gemma 4 QAT: How to Run Google's 1GB Local AI Model on Any Device (2026)

Google's Gemma 4 QAT models shrink a genuinely capable, instruction-following AI down to roughly 1GB of memory — small enough to run on a budget phone, fully offline, with no cloud subscription and no data leaving your device. Released on June 5, 2026, these quantization-aware-training checkpoints cut memory requirements across the entire Gemma 4 family by about 72% while preserving quality close to the full-precision originals. For a small business, that means a private AI assistant for lead follow-up, email triage, content drafts, and customer FAQs that costs zero per-query dollars to operate.

• Gemma 4 E2B mobile QAT format: ~1GB memory footprint
• QAT checkpoints ship for all 5 sizes: E2B, E4B, 12B, 26B-A4B, 31B
• Memory reduction vs. full-precision (BF16): ~72% across the board
• Runs offline via llama.cpp, Ollama, LM Studio, vLLM, MLX, and LiteRT-LM
• Apache 2.0 license — free for commercial use
• Caution: Google published quality claims without benchmark numbers — validate on your own tasks

---

What Is Gemma 4 QAT and why does it matter?

Gemma 4 QAT is a set of model checkpoints from Google DeepMind that use Quantization-Aware Training (QAT) — a technique that simulates low-bit compression during training, so the model learns to hold up at 4-bit precision instead of being squeezed afterward. This matters because it solves the biggest problem in local AI: the trade-off between model size and model quality. Ordinarily, shrinking a model (quantizing it) after training degrades its output. QAT trains the model to expect the squeeze from the start.

The result, per Google's official announcement, is that QAT yields "even higher overall quality compared to standard [Post-Training Quantization] baselines" while reducing memory by approximately 72%. The smallest model, Gemma 4 E2B, drops to about 1GB in the mobile-optimized format — and the text-only version (without Per-Layer Embeddings) comes in at under 1GB.

For context on what Gemma 4 is and its model family: Gemma 4 is Google's fourth generation of open-weight models, all released under Apache 2.0, supporting text, image, and (on the smaller sizes) audio input. Five sizes span phones through workstations. (Source: Google Gemma 4 model card) If you're interested in building a full open-source AI agent stack on top of a local model, see our guide to building a free open-source AI agent with Hermes Agent and Kimi K3.

How does quantization-aware training actually work?

Quantization-Aware Training works by inserting simulated quantization operations into the model during the training process itself. On every forward pass, the weights are rounded to their target precision (4-bit, 2-bit) and the model's loss is computed against those rounded weights. During backpropagation, the gradients flow through the quantization using a Straight-Through Estimator (STE) — a mathematical trick that treats the non-differentiable rounding function as an identity operation, so gradient descent keeps working even though the rounding itself has no derivative. (Source: 7minAI, 2026-06-06)

The key distinction from Post-Training Quantization (PTQ):

Feature	PTQ (post-training)	QAT (quantization-aware)
When applied	After training, once	During every training step
Quality at 4-bit	Degrades noticeably	Maintains close to original
Cost	Minutes, no retraining	High — requires training infrastructure
Who does it	Anyone, at home	The lab with the GPUs (Google, in this case)

The practical insight: QAT isn't free for the entity doing it — it requires a full training or fine-tuning pass. That's why it comes from Google rather than being a one-line conversion you run yourself. But the benefit flows to you: you download a 4-bit file that behaves closer to the full model than any community-generated quant would. (Source: learnaivisually.com)

What are all the Gemma 4 QAT model sizes and how much memory do they need?

Google released QAT checkpoints for all five Gemma 4 sizes, each targeting a different hardware tier. Here's how much memory each one needs, based on the Unsloth GGUF conversions and Google's own figures:

Model	QAT (int4) GGUF size	Original (BF16)	Memory reduction	Context	Modalities
E2B	~2.62 GB	~9.31 GB	71.86%	128K	Text, Image, Audio
E4B	~4.22 GB	~15.1 GB	72.05%	128K	Text, Image, Audio
12B	~6.72 GB	~23.8 GB	71.76%	256K	Text, Image, Audio
26B-A4B (MoE)	~14.2 GB	~50.5 GB	71.88%	256K	Text, Image
31B	~17.3 GB	~61.4 GB	71.82%	256K	Text, Image

Sources: Unsloth Gemma 4 QAT documentation, Google model card. The E2B mobile-specic format further drops the footprint to ~1GB by applying targeted 2-bit quantization to token-generation layers while keeping core reasoning layers at higher precision, plus embedding and KV cache optimization. (Source: Google announcement)

Note: The "E" in E2B and E4B stands for "effective" parameters. E2B has ~2.3B effective parameters (5.1B total including embeddings), and E4B has ~4.5B effective (8B total). Per-Layer Embeddings (PLE) make a portion of those parameters efficiently loadable on CPU rather than occupying GPU/accelerator memory. (Source: Google model card)

How do you run Gemma 4 QAT locally in 2026?

You can run Gemma 4 QAT locally using any of the popular local AI tools. The process is the same across all of them: install the tool, download the model, and start chatting. Below are the three most common paths.

Method 1: Ollama (easiest for beginners)

Ollama is the simplest way to run local models on macOS, Linux, or Windows. No coding required.

1. Install Ollama from ollama.com. On macOS, download the installer. On Linux: curl -fsSL https://ollama.com/install.sh | sh.
2. Pull and run the model:

   ollama run gemma4
   

   This downloads the default Gemma 4 QAT quant automatically and starts a chat session. (Source: Ollama library — gemma4)
3. To try a different size, specify it: ollama run gemma4:12b or ollama run gemma4:31b. Check available tags on the Ollama library page.

Ollama handles GGUF quantization selection automatically — it picks a format that fits your available system memory.

Method 2: LM Studio (GUI-based, best for desktop)

LM Studio is a free desktop app with a chat interface, model browser, and hardware compatibility badges.

1. Download LM Studio from lmstudio.ai.
2. Open the app and search for "gemma-4" in the model browser. (Source: LM Studio Gemma 4 page)
3. LM Studio shows which quants fit your hardware. Select a QAT Q4_0 GGUF and hit Download.
4. Start chatting in the built-in interface. No terminal needed.

Method 3: llama.cpp (lightweight, best for edge devices)

llama.cpp is the underlying engine that Ollama and many other tools are built on. It runs on CPU, GPU, or both, and is the most configurable option — ideal for phones, Raspberry Pi, or low-resource devices.

1. Install llama.cpp from the official GitHub repo (build from source, or use a prebuilt binary).
2. Download the QAT GGUF from Hugging Face. Google's official Q4_0 collection is at huggingface.co/collections/google/gemma-4-qat-q4-0. The Unsloth community collection also offers UD-Q4_K_XL GGUFs that may hold quality better than raw Q4_0: huggingface.co/collections/unsloth/gemma-4-qat.
3. Run the model:

   ./llama-cli \
     -hf unsloth/gemma-4-E2B-it-qat-GGUF:UD-Q4_K_XL \
     --temp 1.0 --top-p 0.95 --top-k 64
   

For E2B on a phone, use Google's LiteRT-LM runtime, which is specifically optimized for edge deployment. (Source: Google announcement)

Which Gemma 4 QAT model should you pick?

Match the model size to your available memory, not to your ambitions. A larger model at heavier compression usually outperforms a smaller model at higher precision, so prefer sizing up with a lower quant when you can. (Source: atomic.chat guide)

Your hardware	Recommended model	QAT memory	What you get
Phone, Raspberry Pi, 2GB RAM	E2B (mobile format)	~1 GB	Text, image, audio; offline chat; drafts
Budget laptop, 4-8GB RAM	E4B or E2B (Q4_0)	3-5 GB	Stronger reasoning; longer context
8-16GB GPU/Mac	12B	~7 GB	Audio input; 256K context; best quality-per-GB
16-24GB GPU	26B-A4B (MoE)	~14 GB	Fast as a 4B model, near-31B quality
24GB+ pro GPU	31B	~18 GB	Dense flagship; best raw capability

The 26B-A4B is a Mixture-of-Experts model — 25.2B total parameters but only 3.8B active per token — which means it runs almost as fast as a 4B model while approaching the quality of the full 31B dense variant. If you have the memory for it, it's the best speed/quality trade-off in the family. (Source: Google model card)

What can you actually do with a 1GB local AI model?

Running a model locally isn't a parlor trick — it solves real business problems without per-query costs or privacy concerns. The key advantage over cloud AI: your data never leaves your device, works offline, and costs nothing per use. (Source: Google Gemma documentation)

Here are the practical use cases that a Gemma 4 QAT model running in 1-7GB of RAM handles well:

For small business owners:

• Draft customer follow-up messages for leads who went quiet (lead re-engagement)
• Summarize long client emails into 2-3 bullet points before deciding whether to reply
• Turn rough voice note transcripts into clean task lists
• Write onboarding messages for new community members or newsletter subscribers
• Clean up coaching call notes into shareable summaries

For developers:

• Generate boilerplate code and function stubs offline
• Explain unfamiliar code snippets without sending them through an API
• Draft commit messages and pull request descriptions from diffs
• Build on-device assistants that work in air-gapped environments

For the privacy-conscious:

• Process contracts, internal notes, or customer data without sending anything to a third-party server
• Work on a plane, in a remote area, or anywhere with no connectivity
• Avoid vendor lock-in — these are Apache 2.0 open weights you download and own

The honest caveat: Gemma 4 QAT models are excellent at instruction following, summarization, and light reasoning, but they are not frontier models. For complex multi-step reasoning, large-context analysis, or tasks requiring the absolute best quality, you'll still want a large cloud model. The value proposition is that for the 80% of day-to-day AI tasks — drafting, summarizing, sorting, formatting — a locally running model is free, private, and fast.

How does Gemma 4 QAT compare to other local AI models?

Gemma 4 QAT enters a crowded local AI space. Here's how it stacks up against other popular open-weight models for local deployment:

Model family	Sizes	Min memory (4-bit)	License	Notable strength
Gemma 4 QAT	E2B, E4B, 12B, 26B-A4B, 31B	~1 GB (E2B mobile)	Apache 2.0	Official QAT from Google; lowest memory per quality tier
Qwen 3	Multiple up to 235B-A22B	~2-3 GB (small quants)	Apache 2.0	Strong multilingual; versatile sizes
Llama 4	Scout 109B, Maverick 400B	Needs multi-GPU even quantized	Llama Community License	High quality; very high memory demand
Kimi K3	2.8T (16/896 experts active)	Not designed for consumer hardware	Modified MIT	Frontier-scale open model; huge context
DeepSeek V4	Multiple sizes	~4 GB+ (small quants)	MIT	Strong coding/reasoning; efficient

Gemma 4 QAT's unique edge is that Google did the expensive QAT training pass and shipped the result, so you get 4-bit weights that behave like larger ones without retraining anything yourself. The mobile-optimized E2B format setting the bar at ~1GB has no direct competitor at this writing — most other model families at 4-bit sit at 2-4GB minimum even in their smallest configurations. For a deeper dive into another powerful open-weight model that takes a different approach — a 2.8T-parameter Mixture-of-Experts architecture — check our Kimi K3 open-source model guide.

What this means for you

If you're a small business owner, Gemma 4 QAT finally makes running a private AI assistant a zero-cost infrastructure play. A 1GB model running on your phone or an old laptop can handle customer FAQ drafting, email triage, and lead follow-up without a cloud bill or a privacy worry. Start with Ollama — the ollama run gemma4 command works in minutes.

If you're a developer, the QAT checkpoints are a drop-in upgrade: if you're already running community 4-bit GGUFs of Gemma 4, swap in the official QAT Q4_0 to get better quality at the same memory budget. Use the Unsloth UD-Q4_K_XL GGUFs if you want the community-optimized variant that pushes accuracy higher than raw Q4_0.

If you're an enterprise architect, the mobile format is the one to watch: a 1GB multimodal model is small enough to ship inside a phone app and run fully offline, opening up on-device AI features — speech recognition, real-time translation, offline chat — without backend infrastructure.

FAQ

Q: Is Gemma 4 QAT free to use commercially? A: Yes. Gemma 4 ships under the Apache 2.0 license, which permits commercial use, modification, and redistribution. There are no royalty requirements or custom restrictions beyond the standard Gemma terms of use, which are minimal compared to many alternative open-weight licenses.

Q: How much RAM do I need to run Gemma 4 E2B QAT? A: About 1GB for the mobile-optimized format, or roughly 3GB for the standard Q4_0 GGUF. The mobile format keeps reasoning layers at higher precision while compressing token-generation layers to 2-bit and optimizing the KV cache and embeddings. You need additional headroom for the operating system and the inference tool itself, so plan for at least 2-4GB of free system memory. (Source: Google announcement, Unsloth documentation)

Q: Do Gemma 4 QAT models support offline use? A: Yes. When you download a Gemma 4 QAT model and run it via Ollama, LM Studio, llama.cpp, or LiteRT-LM, all inference happens on your device. No internet connection is required after the initial download. Your data never leaves your device. (Source: Google Gemma documentation)

Q: What's the difference between QAT and regular quantization (PTQ)? A: Post-Training Quantization (PTQ) rounds the model weights to lower precision after training is complete — it's fast but degrades quality. Quantization-Aware Training (QAT) simulates the quantization rounding during training itself, so the model learns to operate at low precision. Google's QAT checkpoints yield higher quality than standard PTQ baselines at the same 4-bit file size. (Source: Google announcement)

Q: Did Google publish benchmark scores for the QAT checkpoints? A: No. Google's announcement states that QAT yields "even higher overall quality compared to standard PTQ baselines" but did not include specific benchmark scores. On the previous generation (Gemma 3 QAT), the Q4_0 perplexity drop was cut by 54% compared to PTQ — a promising precedent, but "higher quality than PTQ" is a relative claim, not an absolute one. Validate the QAT checkpoint against your own task before assuming it matches the full model. (Source: MarkTechPost, 2026-06-05)

Q: Can Gemma 4 QAT accept image and audio input locally? A: Yes, on the E2B, E4B, and 12B models, which support text, image, and audio input. The 26B-A4B and 31B models support text and image but not audio. When running locally with a mmproj (multimodal projector) file — for example with llama.cpp — you can attach images and ask questions about them. (Source: Hugging Face model card)

Q: Where are the Gemma 4 QAT model weights hosted? A: Google's official Q4_0 QAT weights are at huggingface.co/collections/google/gemma-4-qat-q4-0 and the mobile-optimalized weights at huggingface.co/collections/google/gemma-4-qat-mobile. Community-improved GGUFs from Unsloth are at huggingface.co/collections/unsloth/gemma-4-qat.

---

Sources

1. Google — "Gemma 4 with quantization-aware training" (The Keyword, Jun 05, 2026) — blog.google
2. Google AI for Developers — Gemma 4 model card — ai.google.dev
3. Google AI for Developers — Gemma overview — ai.google.dev
4. Unsloth — Gemma 4 QAT documentation (2026-06-05) — unsloth.ai
5. MarkTechPost — "Google DeepMind Releases Gemma 4 QAT Checkpoints" (Jun 05, 2026) — marktechpost.com
6. 7minAI — "Gemma 4 QAT: Google's quantization-aware checkpoints" (Jun 06, 2026) — 7minai.com
7. learnaivisually — "Google ships Gemma 4 QAT checkpoints" — learnaivisually.com
8. Ollama Library — gemma4 — ollama.com/library/gemma4
9. LM Studio — Gemma 4 — lmstudio.ai

---

Updates & Corrections

• 2026-08-01 — Initial publish. All facts verified against Google's official announcement (Jun 05, 2026), the Gemma 4 model card, and Unsloth's QAT documentation. Pricing: N/A (free open weights). Note: Google did not publish QAT benchmark scores — quality claim is stated, not numbered.

---