Verdict: Yes, you really can run a roughly 744-billion-parameter Mixture-of-Experts model locally on consumer hardware today — the open-source engine Colibrì (Apache 2.0, GitHub - JustVugg/colibri) does it by keeping only the dense layers of GLM-5.2 resident in roughly 25 GB of RAM and streaming the 19,456 routed experts on demand from an NVMe SSD. The catch is throughput: on the developer's own 12-core, 25 GB machine you get ~0.05–0.1 tokens/second cold — far below what interactive chat needs. The real value today is offline, private, frontier-quality reasoning on hardware you already own, especially on a workstation with 64+ GB of RAM and a fast internal NVMe drive, where speeds climb toward 1 token/second and beyond.
Last verified: 2026-07-30 · Primary model: GLM-5.2 by Z.ai · Engine: Colibrì v1.1+ (repo) · Minimum RAM: ~25 GB (int4) · Disk: ~370 GB of NVMe · Cold-start speed (dev box): ~0.05–0.1 tok/s · Fastest community run: 6.00 tok/s on a 251 GiB host with 6× RTX 5090
Why Is Running a 744B Model Locally Even Possible?
It is possible because GLM-5.2 is a Mixture-of-Experts (MoE) model, not a dense transformer. Only about ~5.4% of its parameters participate in producing any given token. The GLM-5 technical report and the Hugging Face model card confirm GLM-5.2 carries roughly 743–744 billion total parameters (some sources round to 744B; vLLM recipes list ~743B total / 39B active) but activates only ~39–40 billion per token. The dense portion — attention layers, shared experts, and embeddings — is roughly 17 billion parameters, which at 4-bit (int4) quantization fits in about 9.9 GB of RAM.
That is the architectural loophole Colibrì exploits. A traditional inference engine has to load every weight into fast memory before doing any work. Colibrì instead treats VRAM, system RAM, and NVMe storage as one managed memory hierarchy: the dense essentials stay resident in RAM, and each routed expert (~19 MB at int4) is pulled from disk only when the router selects it for a token, then cached for reuse.
How Does Colibrì Stream a Giant Model From Disk Without Losing Quality?
Colibrì preserves output quality through three deliberate design choices, all of which are documented in the engine's repository README:
Faithful forward pass. The engine is validated to produce token-exact output against a reference transformer implementation. The default policy never silently changes model precision or router semantics — if you under-budget memory, the engine goes slower, it does not "lose quality" by silently dropping precision or routing.
Native Multi-Token Prediction (MTP). GLM-5.2 ships with a native MTP head that GLM-5.2's release notes describe as "extended from 3 to 5 draft tokens" relative to GLM-5.1, "increasing the acceptance length by up to 20%" for speculative decoding (zai-org/GLM-5 GitHub). Colibrì uses this to guess several tokens ahead in a single forward pass, which vLLM's MTP docs confirm is model-based speculative decoding — no separate draft model required.
MLA / DeepSeek Sparse Attention. GLM-5.2 uses Multi-head Latent Attention (MLA), introduced in DeepSeek-V2 and refined in DeepSeek-V3, which compresses the key-value cache into a small latent vector. Independent analysis of a DeepSeek-V3 implementation (jd-opensource/xllm) measures the KV-cache compression at roughly 114× over standard multi-head attention. A tutorialq.com explainer calls MLA roughly "10× smaller than GQA while matching MHA quality." Compressed attention is central to why the in-memory footprint stays small enough for a laptop.
Engine size: Colibrì is a single C file (
c/glm.c) plus small headers — no BLAS, no Python at runtime, and no GPU required by default. An optional CUDA tier exists for pinning hot experts in VRAM.
What Hardware Do You Actually Need to Run a 744B Model Locally?
The minimum bar is low; the realistic bar for usable speed is much higher. The README's back-of-envelope predictions make the gradient explicit:
| Machine config | Expected speed | Notes (primary source) |
|---|---|---|
| Dev box — WSL2 VHDX, ~1 GB/s, 25 GB RAM | ~0.05–0.1 tok/s cold | "proven baseline" (Colibrì README) |
| Native Linux, PCIe4 NVMe (~3–5 GB/s random), 32 GB | ~0.5–1 tok/s | Colibrì README |
| 251 GiB host + 6× RTX 5090, full expert residency | 6.00 tok/s decode | Real measured run, Colibrì README |
Independent testing reports the same gradient externally. Earlier benchmarks on a MacBook with an M2 Max chip, 32 GB RAM, and the model on an external ~1 GB/s Samsung SSD took roughly 2 minutes to produce the first token and crawled at about a tenth of a token per second — with over 80% of the time spent waiting on the disk. The same model on a workstation with 64 GB of RAM, an RTX 5090, and the model on a fast internal drive hit about 0.8 tok/s after roughly 17 seconds to first token. (Those two measurements are widely circulated in Colibrì coverage; treat as vendor/community-reported.)
The non-obvious finding from those tests: the GPU is not the bottleneck on consumer hardware — memory and storage are. On the faster workstation the RTX 5090 stayed largely idle; what mattered was having enough RAM to keep more of the model resident and a fast enough NVMe drive to serve cold experts quickly. Even Colibrì's published numbers put the fastest configs at things like a 6-GPU rig or a dual-Xeon server with a full terabyte of RAM — setups that still beat a single RTX 5090 by a wide margin because more of the expert set can stay resident.
How Do You Set Up and Run Colibrì — Step by Step?
These steps mirror the "Download the model" and "Try it" sections of the official repository. They assume you already have GLM-5.2's int4 weights downloaded locally — Colibrì does not bundle the weights.
Check disk space first. Expect ~370 GB of expert weights at int4, plus room for the dense resident set and a working space. Free up or attach an NVMe drive. Colibrì's
coli plan --jsoncommand reads only safetensors headers and reports the exact dense/expert footprint for your model before you load anything.Clone the repo and build the engine. The runtime path is intentionally flat — a single C file plus small headers. From the repository root,
makebuilds the engine andmake checkruns dependency-free C and Python tests.Convert the model to int4 (one-time). Use the converter the repo provides, pointing
--modelat the safetensors weights and--outat an ext4/NVMe destination with ~400 GB free. Python is only used by the one-time converter; the runtime is pure C.Inspect the storage plan.
COLI_MODEL=/your/path ./coli planreports the exact dense/expert footprint, runtime RAM reserve, safe expert-cache cap, and bounded VRAM hot tier. Add--gpu 0,1 --ram 128 --vram 48 --jsonto plan a multi-GPU tier.Start a chat.
COLI_MODEL=/your/path ./coli chatstarts the interactive runner. RAM budget, expert cache, and MTP are detected automatically. The first prompt is a cold start — it will be slow because nothing is cached yet.Record routing, then pin hot experts into spare RAM.
STATS=stats.txt ./coli chatrecords which experts your workload actually routes to. ThenPIN=stats.txt PIN_GB=20 ./coli chatpromotes the hottest experts into a persistent RAM store. The more you run the same kind of prompt, the warmer this cache becomes.Optional: serve an OpenAI-compatible API.
./coli serve --model /your/pathexposes an OpenAI-shaped endpoint../coli webruns the API plus a browser dashboard with live routing visualisation.
Dual-SSD tip. If you have a second NVMe drive, Colibrì supports a mirror: set COLI_MODEL=/fast/glm52_i4 COLI_MODEL_MIRROR=/second/glm52_i4. Each expert is dispatched to one drive by a deterministic hash, so aggregate read bandwidth is roughly the sum of both drives — a 9 GB/s + 3 GB/s pair reads experts ~33% faster than the fast drive alone (per the README).
How Slow Is Too Slow? Where Colibrì Actually Pays Off
Colibrì is a proof of concept that also runs, not a production replacement for a hosted frontier API. A reasonable way to decide if it fits your workflow:
| Use case | Colibrì vs cloud API | Why |
|---|---|---|
| Privacy-sensitive prompts (legal, medical, confidential code) | Colibrì wins | Nothing leaves your machine; weights are downloaded once and read locally. |
| Long offline research or background analysis | Colibrì wins | You pay latency once and let the cache warm; speed matters less than capability + privacy. |
| Interactive chat at "feels instant" speeds on a 25 GB laptop | Cloud wins | Cold-start at ~0.05–0.1 tok/s is far below the 30–80 tok/s you get from a hosted frontier endpoint. |
| Bulk generation on a 64+ GB workstation with internal NVMe | Tied / Colibrì competitive | Speeds climb toward 0.5–1+ tok/s, and you run it free indefinitely. |
| Production inference under concurrency | Cloud wins | Colibrì's --max-queue runs rate-limited but its real strength is offline single-stream work. |
The economics invert in Colibrì's favour once you have fast local storage and enough RAM to keep a meaningful fraction of the expert set resident. The benchmark that landed the project on Hacker News (covered at news.ycombinator.com) generated genuine interest precisely because frontier-class local inference stopped being gated behind datacenter GPUs.
How Does Colibrì Compare With llama.cpp and vLLM?
Colibrì chose a different design point from the two dominant local-inference stacks:
| Tool | Architecture | Best for | More info |
|---|---|---|---|
| Colibrì | Pure-C, disk-streamed MoE experts, optional CUDA VRAM tier | Arbitrarily large MoE models on bounded RAM (~25 GB) | github.com/JustVugg/colibri |
| llama.cpp | C++/CUDA, loads full model in memory | Models up to ~70B on high-end consumer hardware | github.com/ggml-org/llama.cpp |
| vLLM | Python+CUDA, tensor parallel + paged attention + MTP speculative decoding | Multi-GPU serving of large MoE models with high throughput | docs.vllm.ai |
If your goal is "run a 744B MoE on 25 GB of RAM", Colibrì is the only option in that table; llama.cpp would need to fit the whole model. If your goal is "serve GLM-5.2 at 70+ tok/s on eight H100s", vLLM's GLM-5.2 recipe — which lists ~743B total / 39B active and demonstrates 5-token MTP on AMD MI300X and 8× B200 — is the production path.
What This Means for You
- For small-business owners and builders: Colibrì makes a frontier-class open-weight model realistically reachable on hardware you may already own, but it is not a turnkey replacement for an API. Treat it as capability research, an offline privacy tool, and a serious option once you have a workstation with 64+ GB RAM and an internal NVMe drive. If you want to start smaller and ship faster, look at our guide to running a local coding agent with Gemma 4 and Ollama — the MoE-disk trick only shines when you specifically need frontier-class reasoning, not for everyday local AI.
- For developers and AI teams: Colibrì reframes the local-LLM memory constraint around disk bandwidth instead of VRAM. The two most useful takeaways are the architecture pattern (stream only the routed expert a token needs, cache aggressively, learn the routing pattern) and the engineering trade-off (faithfulness over speed, single C file with zero external dependencies, optional GPU tier for the hottest experts).
- For anyone studying model portability: The wider trend Colibrì belongs to — pushing AI inference to ever-smaller devices via MoE sparsity, MLA KV-cache compression, and aggressive quantization — shows up across our guide to tiny AI models on edge devices. The same thinking underlies Poolside's 118B open-weight Laguna S 2.1 coding model and the broader open-weight movement: capability is leaking out of the datacenter if you know how to cache and stream it.
FAQ
Q: Can you really run a 744B parameter model on a 25 GB laptop?
A: Yes — with caveats. The Colibrì README explicitly demonstrates a 12-core, 25 GB laptop running GLM-5.2 by streaming experts from disk. The catch is speed: cold starts are around 0.05–0.1 tokens per second on a dev box. It produces correct, frontier-quality answers without a GPU; it simply does so slowly on that hardware.
Q: Is Colibrì free and open source?
A: Yes. Colibrì is licensed under Apache 2.0 on GitHub. GLM-5.2 weights are released by Z.ai under the MIT license on Hugging Face.
Q: Why is the GPU not the bottleneck on consumer hardware?
A: Because the bottleneck is memory and disk bandwidth, not compute. Even with int4 quantization, ~744 billion parameters take roughly 370 GB on disk, and only a small fraction of that fits in a typical consumer GPU's VRAM. The engine must keep fetching experts from disk; if the disk is slow, you spend most of your wall-clock time waiting, not computing. Fast internal NVMe and more system RAM (not a bigger GPU) are what move the needle.
Q: How much disk space do I need?
A: Roughly 370 GB for the int4 expert weights (GLM-5.2 has 19,456 routed experts at ~19 MB each), plus headroom for the resident dense set and working space. If you don't have the space internally, an external SSD works but is slower, which directly hurts throughput.
Q: Does Colibrì work on Windows and macOS?
A: Yes. The README documents a Windows 11 native build with optional CUDA DLL, and the developer built and tested the engine on a 12-core laptop. Linux with native PCIe4 NVMe is the configuration the README predicts fastest raw expert reads from (~3–5 GB/s random), so a dual-boot Linux partition pays off here if you have one available.
Q: What is MoE and why does it make this possible?
A: A Mixture-of-Experts model splits its parameters across thousands of specialist sub-networks ("experts"). A router selects only a small subset to activate per token. GLM-5.2 activates roughly ~39B of its ~743–744B parameters per token — about 5.4%. That means the dense, always-active part is small enough to fit in RAM, and only the selected experts need to be loaded from disk at any instant. A dense 744B model would not be feasible to run this way.
Q: Does the int4 quantization hurt quality?
A: The Colibrì forward pass is validated to be token-exact against the full-precision reference implementation, and the engine's default policy never silently changes precision or routing. The same per-token experts fire whether the answer came from a VRAM-resident expert or a disk-streamed one. Where you may notice differences in practice is in highly technical domains where the right experts are repeatedly being loaded cold from disk.
Discussion
0 comments