Verdict: A 28.9 million-parameter language model runs on an $8 ESP32-S3 microcontroller at roughly 9.5 tokens per second, fully offline, thanks to a memory tiering technique borrowed from Google's Gemma models called Per-Layer Embeddings. It is a genuine engineering breakthrough — but the model only writes simple children's stories. It cannot answer questions, follow instructions, or write code. The architecture is the story, not the output. For builders exploring edge AI, this is the most important microcontroller LLM demo of 2026 and a blueprint for putting small models on cents-per-unit hardware.
TL;DR: Last verified 2026-07-31
- A developer (known as slvDev) fit a 28.9M-parameter model on an ESP32-S3 — roughly 100× larger than the previous best (260K params) on the same class of chip.
- The trick: park 25M of the 28.9M parameters in flash memory as a lookup table; only ~450 bytes are read per token. The tiny compute core stays in 512KB of SRAM.
- Speed: ~9.5 tokens/sec end-to-end. Model file: 14.9MB at 4-bit quantization.
- Trained on TinyStories (Microsoft Research), so it generates short stories — not a chatbot.
- Hardware: ESP32-S3-WROOM-1 N16R8 (16MB flash, 8MB PSRAM). Boards with only 4–8MB flash will not fit the model.
How does a 28.9M-parameter model fit on a chip with 512KB of RAM?
A standard approach fails immediately: the ESP32-S3 has only 512KB of SRAM, and cramming 28.9 million parameters into that space is physically impossible. The previous largest language model on this class of microcontroller topped out at 260,000 parameters — roughly 110× smaller.
The workaround, borrowed from Google's Gemma 3n models, is called Per-Layer Embeddings. The key insight is that most parameters in a language model do not actually compute anything — they sit in an embedding table that the model reads from, like looking up words in a dictionary. If most of your parameters are only ever looked up, they do not need fast memory. You can leave that table parked in slow, cheap flash storage and only pull the few rows the current token needs.
Here is the memory layout:
| Memory tier | Size | Speed | What lives here |
|---|---|---|---|
| SRAM (on-chip) | 512 KB | Fastest | The "thinking" core — attention heads and feed-forward layers used on every token |
| PSRAM (external) | 8 MB | Medium | The output head and working memory |
| Flash (external) | 16 MB | Slow but huge | The 25M-parameter embedding lookup table — only ~6 rows (~450 bytes) read per token |
Because the model has 6 layers, it pulls about 6 rows from the flash table per token — roughly 450 bytes. That is a tiny read compared to the 15MB model file sitting in flash. The dense compute core (attention + feed-forward) fits comfortably in the 512KB of SRAM. The flash is treated as a read-only database, not as working memory.
This is the same pattern Google built into Gemma 3n for mobile devices: per-layer embedding tables that can be loaded on-demand from flash storage rather than kept in RAM. The ESP32-S3 port simply applies that idea to a microcontroller's far more constrained memory map.
What hardware do you need to run a local LLM on the ESP32-S3?
You need the ESP32-S3-WROOM-1 N16R8 variant specifically. The "N16R8" designation means 16MB of flash and 8MB of PSRAM — the 16MB flash is non-negotiable because the trained model file occupies about 14.9MB. Boards with 4MB or 8MB of flash will not fit it.
| Spec | Required value | Why |
|---|---|---|
| Chip | ESP32-S3 (dual-core Xtensa LX7, 240 MHz) | The S3 variant has AI vector instructions and enough I/O |
| Flash | 16 MB (N16) | Model file is ~14.9MB at 4-bit quantization |
| PSRAM | 8 MB (R8, Octal SPI) | Output head + working memory |
| SRAM | 512 KB (on-chip) | Dense compute core |
| Connectivity | Not required for inference | Wi-Fi/BLE onboard but unused during generation |
| Price | ~$8–$10 (dev board) | Varies by vendor; some list closer to $10 |
The ESP32-S3 is widely available from multiple vendors. Espressif's official datasheet confirms 512KB of on-chip SRAM and support for up to 16MB of external flash via Quad SPI. The N16R8 module variant is the one to look for — boards with less flash simply cannot hold the model.
What is the model trained on, and what can it actually do?
The model is trained on TinyStories, a synthetic dataset created by Ronen Eldan and Yuanzhi Li at Microsoft Research. TinyStories consists of short stories written using vocabulary a typical 3-to-4-year-old would understand, generated by GPT-3.5 and GPT-4. It was designed to test how small a language model can be while still producing coherent English. The paper showed that models below 10 million parameters — even with just a single transformer block — can generate fluent, grammatically correct stories when trained on this dataset.
That training choice is the source of both the demo's success and its limitations:
What it can do:
- Generate coherent short stories at ~9.5 tokens per second
- Maintain basic grammar and narrative consistency across paragraphs
- Respond to simple prompts with story-like text
- Run entirely offline with no server, no Wi-Fi, no cloud API
What it cannot do:
- Answer factual questions (it does not know what a galaxy is)
- Follow instructions or act as a chatbot
- Write code
- Stay on topic — even when prompted with "Star Wars," it drifts back to a story about a little girl, because that is the distribution it was trained on
The model always steers toward story text regardless of the prompt. This is expected behavior for a TinyStories-trained model: if you train on stories, you get stories. If you trained a 28M-parameter model on a general dataset, the output would be gibberish — the model is too small for general-purpose language understanding. TinyStories is specifically designed to make tiny models look competent within a narrow domain.
How was the inference engine built?
The inference code runs in plain C on the bare metal — no operating system, no Python interpreter, no runtime. This approach traces directly to Andrej Karpathy's llama2.c project, which demonstrated that you could train a small Llama 2-architecture model and run inference on it with just a few hundred lines of portable C code. Karpathy's project compiled a 15M-parameter TinyStories model to a single run.c file and achieved ~110 tokens/sec on an M1 MacBook Air.
The ESP32-S3 project is built on the same blueprint: train in PyTorch using the Llama 2 architecture, export the weights to a flat binary, and run inference in C. The difference is the deployment target — instead of a laptop CPU, the inference runs on a 240 MHz microcontroller with a fraction of the memory.
How do you build and flash this yourself?
The project's GitHub repository includes firmware, training code, and a hardware setup guide. Here is the general workflow:
Get the right board. Verify your ESP32-S3 has the N16R8 configuration (16MB flash + 8MB PSRAM). This is the single most common point of failure — many popular ESP32-S3 boards have only 4MB or 8MB of flash.
Install the ESP-IDF toolchain. Espressif's official SDK compiles the C firmware. You need ESP-IDF v5.x or later with the Xtensa toolchain configured for the ESP32-S3 target.
Train the model (optional). The training code lives in the repository's
src/andexperiments/directories. Training a TinyStories model at this scale takes about 25–30 minutes on a modern laptop (the video author reported ~30 minutes on a MacBook). If you skip training, you can use the pre-trained model binary.Export and quantize. The model is exported at 4-bit quantization, bringing the file size to ~14.9MB. The quantization and ablation tooling is in the repository.
Build the firmware. Compile the C inference engine targeting the ESP32-S3. The firmware reader README in
firmware/esp32_llm/README.mdcovers wiring and flashing steps.Flash the board. Connect via USB and use
idf.py flashto write both the firmware and the model binary to the ESP32-S3's flash partition. Changing the prompt requires reflashing the board — the current implementation only supports one pre-flashed prompt at a time.Verify. Once flashed, the model starts generating text immediately and writes each word to a small attached screen. You should see roughly 9 tokens per second.
One important limitation: each prompt must be compiled into the firmware before flashing. You cannot type a new prompt at runtime and get a different response — the prompt is baked into the flash image. This is a proof-of-concept limitation, not a fundamental architectural constraint, but it means the device currently generates from a single fixed seed per flash cycle.
How does this compare to other edge AI approaches?
| Approach | Hardware | Model size | Speed | Capability | Cost |
|---|---|---|---|---|---|
| ESP32-S3 + Per-Layer Embeddings | ESP32-S3 N16R8 | 28.9M params | ~9.5 tok/s | Story generation only | ~$8 |
| llama2.c (laptop) | M1 MacBook Air | 15M params | ~110 tok/s | Story generation | ~$999+ |
| Raspberry Pi + llama.cpp | Raspberry Pi 5 | 1B–3B params | 2–8 tok/s | General chatbot (limited) | ~$80 |
| Cloud API (ChatGPT, Claude) | Any device | 100B+ params | 50+ tok/s | Full general assistant | $20/mo+ |
The ESP32-S3 result is not competing with any of these on capability — it is competing on cost and power. For less than $10 in hardware,with no ongoing subscription and no network dependency, you get a device that generates coherent text. That matters for applications where connectivity is unavailable, latency must be sub-second, or the BOM cost must stay in the cents-per-unit range.
What this means for you
If you are building IoT devices, educational kits, or any product where a small text-generation feature adds value and must work offline, this is your proof that the ceiling just moved. The Per-Layer Embeddings pattern is not limited to story generation — it is a memory-tier architecture that could, in principle, support any small model with a large embedding table. The open question is whether useful non-story models can be trained at this scale.
For makers and hobbyists: this is one of the most accessible edge AI projects available. The total cost is under $15 (board + small display), the code is open source (MIT license), and the build process is well-documented. If you have ever wanted to hold a language model in your hand and watch it think without a network cable, this is the project. If you are also exploring free, local AI tools that work offline — like Meetily's free local meeting transcription or the Handy offline dictation app — this ESP32 project is the hardware counterpart to that same privacy-first philosophy.
For product teams: reassess what "generative feature, no server" can mean at your BOM cost. A device that generates text locally — even simple text — at $8 of silicon with no subscription and no data leaving the device opens product categories that cloud-dependent AI cannot reach. This is especially relevant if you are already thinking about how to run free AI coding agents locally or exploring small, efficient agent models like Ling 3.0 Flash — the Per-Layer Embeddings technique could eventually push similar small-model architectures onto even cheaper silicon.
FAQ
Q: Can an $8 microcontroller really run a language model? A: Yes. An ESP32-S3 with 16MB flash and 8MB PSRAM (the N16R8 variant, ~$8–$10) runs a 28.9M-parameter model at ~9.5 tokens per second, fully offline. The trick is Per-Layer Embeddings: 25M of the 28.9M parameters live in flash as a lookup table, and only ~450 bytes are read per token.
Q: What is Per-Layer Embeddings? A: Per-Layer Embeddings is a memory optimization technique from Google's Gemma 3n models. Instead of keeping all embedding parameters in fast RAM, the model stores separate per-layer embedding tables in flash storage and loads only the rows needed for the current token. On the ESP32-S3, this means the 25M-parameter table stays in flash while the small compute core fits in 512KB of SRAM.
Q: Can the ESP32-S3 model answer questions or write code? A: No. The model is trained on TinyStories, a dataset of simple children's stories. It generates coherent story text but cannot follow instructions, answer factual questions, or write code. A 28M-parameter model trained on general data would produce gibberish — TinyStories is what makes the small model appear competent within its narrow domain.
Q: Which ESP32-S3 board do I need to run this? A: You need the ESP32-S3-WROOM-1 N16R8 variant — 16MB of flash and 8MB of PSRAM. Boards with 4MB or 8MB of flash will not fit the 14.9MB model file. Look for "N16R8" in the product name to confirm the correct configuration.
Q: How fast is the ESP32-S3 language model? A: The project reports about 9.5 tokens per second end-to-end (including I/O) and 9.7 tokens per second of pure compute. For comparison, Karpathy's llama2.c running the same class of model on an M1 MacBook Air achieves about 110 tokens per second.
Q: Is this practical for real products? A: Not yet as a general assistant, but yes for specific use cases. Privacy-first devices, educational kits, and any product where a simple text feature must work offline at minimal BOM cost could benefit. The technique (flash-resident embedding tables) is the transferable idea — the story-writing output is just the proof of concept.
Discussion
0 comments