Most "how to run local AI" guides hand you ollama run llama3 and call it a day. That is the right first command — and the wrong place to stop. The builders who actually run open models in production treat local AI as a five-layer stack: hardware, model, inference engine, gateway, and interface, plus the harness that ties them together. Each layer is a real choice with real trade-offs, and choosing deliberately is what separates a working setup from a weekend toy.
This article lays the stack out layer by layer, names the live choices at each layer in 2026, and ends with the workflow serious builders actually use — the one nobody on YouTube puts in the title: use closed APIs to pay for the open infrastructure you keep. You front the work on Claude or Codex tokens, then move the recovered patterns, prompts, and config onto machines you control. The models listed here are open-weight, not always open-source — that distinction matters and is treated below.
Last verified: 2026-07-26 · Five layers: hardware, model, engine, gateway, interface · Open-weight frontier has caught up to closed within months, not years · Apple silicon's unified memory is the cheap on-ramp · Pricing, model versions, and quant names change fast — re-check before you deploy.
What is the local AI stack — and why think in layers?
A local AI stack is the set of components you assemble to run a model on hardware you control instead of calling a hosted API. Thinking in layers matters because each piece can be swapped independently — you do not throw away your model library when you change inference engine, and you do not change engine when you only need a different chat UI.
The five layers, from bottom to top:
- Hardware — the physical machine running the model: GPU, CPU, RAM, storage.
- Model — a file (or set of files) holding trained weights, usually a few gigabytes to a few hundred.
- Inference engine — the software that turns those weights into tokens: llama.cpp, vLLM, Ollama, LM Studio, MLX.
- Gateway — the API translation layer that exposes many models through one OpenAI-shaped interface: LiteLLM is the canonical choice.
- Interface + harness — what humans and agents actually touch: Open WebUI for chat, agents like Claude Code or Codex for code, n8n or ComfyUI for pipelines.
Each layer is a separate problem. Pick the layer you want to change without rewriting the rest.
Layer 1 — Hardware: how much machine do you actually need?
RAM is the gating constraint for local AI, not raw compute. A model has to fit in memory before it can run, and the rule of thumb is unforgiving: at full 16-bit precision, a model needs roughly twice its parameter count in gigabytes of RAM. A 70-billion-parameter model is about 140 GB at FP16. (tonisagrista.com GGUF guide; ggufloader.github.io memory table)
Quantization roughly cuts that in proportion to bit depth. At Q4_K_M (4-bit, the most common sweet spot), the same 70B model needs about 40 GB of RAM. Numbers per parameter published independently on PromptQuorum and the vucense GGUF guide:
| Format | Bytes / param | 7B model | 70B model |
|---|---|---|---|
| FP16 (no quantization) | ~2.0 | ~14 GB | ~140 GB |
| Q8_0 (8-bit) | ~1.0 | ~7.7 GB | ~70 GB |
| Q6_K (6-bit) | ~0.81 | ~5.7 GB | ~57 GB |
| Q5_K_M (5-bit) | ~0.69 | ~4.8 GB | ~48 GB |
| Q4_K_M (4-bit, recommended default) | ~0.55 | ~4.5 GB | ~40 GB |
| Q3_K_S (3-bit) | ~0.42 | ~3.3 GB | ~30 GB |
Apple silicon is the cheapest on-ramp in 2026
This is why Apple silicon has quietly become the default entry machine for local AI. On a discrete-GPU PC, the model must fit inside the GPU's VRAM — typically 12–24 GB on consumer cards — and spilling past that into system RAM costs you 10–50× in speed (PCIe 4.0 x16 caps around 32 GB/s). On Apple silicon, the CPU and GPU share one unified memory pool. A 64 GB Mac effectively gives the GPU ~56–60 GB of usable model memory. (ai-on-mac.com; PromptQuorum memory guide)
Practical fit at common Mac sizes, taking macOS overhead (~8 GB) into account:
| Mac unified memory | Comfortable model size (Q4_K_M) |
|---|---|
| 16 GB | 7B–8B (tight) |
| 36 GB | 13B comfortably, 34B tight |
| 64 GB | 34B Q5, 70B Q3 tight |
| 128 GB | 70B Q5 comfortably |
You cannot upgrade Apple memory after purchase. The honest rule from PromptQuorum's guide: buy the maximum memory you can afford at purchase time — memory is 5–10% of the machine cost up front; replacing the whole Mac later is 100%.
When you need a real GPU
For multi-user serving, ambient agents that run around the clock, or any model above ~70B at decent quantization, you move to NVIDIA — a single H100 has 80 GB of HBM at >3 TB/s, and vLLM's tensor parallelism spreads one model across several. The discrete-GPU tax is real, but you pay it because unified memory does not scale to multi-user concurrency.
Layer 2 — Model: which open-weight model should you actually run?
Model choice is the most volatile layer in 2026. The closed→open gap has collapsed from a year to months, and most of the leading open-weight releases this year came from Chinese labs. (interconnects.ai Kimi K3 analysis)
Open-weight vs open-source — read the model card
Two different things. Open-source means the training code and data are available. Open-weight means the trained weights are downloadable under a license — usually MIT, Apache 2.0, or a modified-MIT — but you cannot reproduce them yourself. Almost everything called "open-source AI" in 2026 is actually open-weight. Always check the model card on Hugging Face for the exact license terms; some are non-commercial, some have regional restrictions.
The live landscape (verified July 2026)
| Model family | Vendor | Notable release | Why builders reach for it |
|---|---|---|---|
| Qwen | Alibaba Cloud | Qwen3.6 series, multiple sizes (huggingface.co/Qwen) | The most-downloaded open-weight line on Hugging Face. Code, vision, math, audio specializations. Permissive licensing on most variants. |
| Kimi K3 | Moonshot AI | 2.8T MoE, 1M context, weights via Modified MIT (kimi.com/blog/kimi-k3; interconnects.ai) | Single largest open-weight release to date. 16/896 experts active per token. Always-on reasoning mode. API at $3/M input, $15/M output. Self-hosting needs ~1.4 TB at 4-bit — supemost users will use the API, not the weights. |
| DeepSeek V4 | DeepSeek | V4-Pro 1.6T / V4-Flash 284B, MIT license, April 24 2026 (morphllm.com) | 80.6% on SWE-bench Verified at $0.87/M output — the strongest open-weight coding result shipped. Per token, V4-Pro is roughly 29× cheaper than Claude Opus 4.8 for comparable work. |
| GLM (e.g. GLM-5 series) | Z.ai | Strong coding at Opus-tier for one-fifth the price | Increasingly cited as the price/value coding pick in 2026 benchmarks. |
| MiniMax | MiniMax | Frontier MoE, multimodal | Among the most-cited alternative Chinese frontier labs alongside DeepSeek. |
| Gemma | Lightweight line, Apache 2.0 | The easy default when you want a small, locally-runnable model from a US lab. See our Gemma local coding assistant guide. | |
| Llama 4 | Meta | Scout, Maverick | The non-Chinese open-weight option; sized for a range of machines. |
A common pre-2026 mental model was "US labs lead; China is six months behind." After Kimi K3 and DeepSeek V4 that is no longer obviously true. Some independent benchmarks now put the gap inside 3–5 months, and on the price-for-quality axis the open-weight frontier is often cheaper by an order of magnitude. (interconnects.ai)
For a deeper take on the open-weight vs closed-weight distinction for your stack, see our Open-Weight vs Closed-Weight AI: The 2026 Era. For an in-depth Qwen breakdown — including the 2.4-trillion-parameter claim — see Qwen 3 Explained: A Builder's Guide.
Layer 3 — Inference engine: who actually turns weights into tokens?
This is the layer people skip over and then wonder why their setup is slow. If the model is the file, the inference engine is the kernel that runs it. Almost all local inference in 2026 runs on one of these — and they are layered, not always competing:
llama.cpp — the foundation
llama.cpp is the pure C/C++ engine underneath almost everything labeled "run local LLM." MIT licensed, 85,000+ GitHub stars, runs GGUF-format quantized models on CUDA, ROCm, Metal, CPUs, embedded, and even Raspberry Pi. When you run a model "locally" through almost any popular tool, llama.cpp is probably doing the actual math. The complete local LLM inference tooling survey on dev.to puts it bluntly: "almost everything in the local LLM inference stack is fully open source under permissive licenses."
Ollama — the right first install for solo developers
Ollama wraps llama.cpp (and, on Apple silicon since v0.19 in March 2026, the MLX backend) in a Go binary with a model registry, automatic GPU detection, and an OpenAI-compatible REST API at localhost:11434. ollama run qwen3:8b pulls a quantized model and starts inference in under five minutes with zero config. On Apple silicon, Ollama now uses MLX natively, which is the fastest path on Macs. The whole agentic ecosystem (Cursor, Continue, Aider, Open WebUI, LangChain, LlamaIndex) targets Ollama's API by default. (dev.to guide; Ollama docs)
For the practical "Ollama + a model on my laptop" walkthrough at the simplest end, see our How to Run Local AI on Your Computer in 2026: The No-Hype Guide.
vLLM — production serving under concurrency
vLLM is the production standard when more than one user hits the same model. Its PagedAttention algorithm treats the GPU KV cache like virtual-memory pages, which buys you 16–20× Ollama's throughput at peak concurrency. The gap collapses to near-zero at one user — a single developer querying sequentially sees no benefit and feels the slower cold start. vLLM is Apache 2.0, OpenAI-compatible, and the right answer for internal AI platforms, multi-GPU datacenter deployments, and any workload where throughput under concurrency is the constraint. (vLLM distributed serving docs; vLLM config docs)
KV cache: the part everyone forgets
The weights are not the only thing in memory at inference time. As the conversation grows, the model keeps a KV cache — compressed representations of past tokens that let attention work without re-reading the whole prompt. KV cache size scales with context length, not with model size. A long context window can double a model's footprint. Quantizing the KV cache separately (q8_0 or q4_0 KV) is a separate lever from quantizing weights — supported in both llama.cpp (--cache-type-k q8_0) and recent Ollama builds. If you are running long-context or document-heavy workloads, KV cache matters as much as weight quantization.
How do you pick between engines?
| Use case | Engine |
|---|---|
| Solo developer, prototyping, agents on your laptop | Ollama |
| Pushing every tok/sec out of one Mac | MLX (via Ollama-native or mlx-lm directly) |
| Multi-user internal AI platform | vLLM |
| Need exotic samplers, EXL2, custom quant formats | llama.cpp directly or SGLang |
| Vision-language model serving on Ampere GPUs | LMDeploy |
| Zero-config desktop GUI, fully auditable code | Jan (open-source LM Studio alternative) |
SGLang, LMDeploy, Jan, GPT4All, and TensorRT-LLM all exist and are good — covered in detail in the dev.to July 2026 inference tools survey. LM Studio is the lone commonly-used proprietary tool in the stack; Jan is its open-source alternative.
Layer 4 — Gateway: one API, many models
Once you have more than one model in play — local Qwen for sensitive work, an Anthropic API when you need frontier reasoning, a self-hosted vLLM for the team — you do not want to write three different SDKs. The gateway layer collapses that complexity into one OpenAI-shaped interface.
LiteLLM is the canonical 2026 choice. Open-source AI gateway calling 100+ providers through one OpenAI-compatible API. Use it as a Python SDK in your code, or as a standalone proxy server your whole team points at. The proxy adds virtual keys, per-user spend tracking, rate limits, caching, fallbacks, guardrails, and an admin dashboard. Benchmarked at 8 ms P95 latency at 1k RPS. Mature adopters include Stripe, Anthropic, and many internal AI platforms. (BerriAI/litellm on GitHub; LiteLLM proxy docs)
The hybrid pattern: let one gateway route to both local and closed
The reflexive "100% local or 0% local" framing is wrong. The pragmatic pattern is to put LiteLLM in front of both a self-hosted vLLM/Ollama endpoint and the Anthropic/OpenAI APIs, then route by task. Cheap, high-volume, sensitive, or ambient work goes to local. Frontier-thinking, complex agent loops, and bursty hard problems go to closed APIs. The gateway is what keeps the choice invisible to your application code.
Layer 5 — Interface + harness: what humans and agents actually touch
This is the layer the user experiences. Three categories worth knowing in 2026:
- Chat UI — Open WebUI is the standard open-source frontend for local models. Drop-in replacement for the ChatGPT interface pointed at Ollama or vLLM. Multi-user, supports documents, integrates with the model registry. It is the defacto open front-end for local AI deployments.
- Agent harnesses — Claude Code (Anthropic), OpenAI Codex, Gemini Code Assist. These are closed-source harnesses with closed-source models behind them. The trick serious builders use: use them as scaffolding for the open infrastructure you keep (more on this below).
- Pipelines / orchestration — n8n for workflow automation, ComfyUI for image generation pipelines, pi.dev for ambient-agent scaffolds. ComfyUI is increasingly the standard for any visual AI pipeline beyond text. These let you compose local models into larger workflows without writing a runtime from scratch.
The Tailscale angle — making your local stack reachable
Tailscale is the boring piece nobody mentions that makes local AI useful. The Personal plan is free, supports up to 6 users and unlimited devices, and gives you a WireGuard-based mesh. Install Tailscale, your laptop and your home server are now on one private network, and your phone can reach your local Open WebUI anywhere. You do not expose ports to the internet; you do not fight NAT. For a one-person local AI stack or a small team, Tailscale plus an always-on Mac mini running Ollama is the cleanest path to "my own private ChatGPT."
The hybrid workflow nobody puts in the title
Here is the part most local-AI tutorials skip: the practical way to actually build and maintain a local stack is not to swear off closed APIs. It is the opposite — use them to do the heavy lifting up front, and migrate what survives onto open infrastructure.
- Build with closed. Front the work on Claude Code, Codex, or Gemini Code Assist. The frontier models are better at understanding your intent, writing the initial configs, and surviving the broken setups every local stack starts with.
- Capture the artifacts. Prompts, model choices, gateway config, infra-as-code, evaluation harnesses — these are what matters. They are model-agnostic.
- Migrate to open. Take the recovered patterns and run them against Qwen, Kimi, DeepSeek, or Gemma served locally via Ollama/vLLM through LiteLLM. Your application code does not change because the gateway already speaks OpenAI.
- Verify before trusting. Independent testing on Kimi K3 found ~51% hallucination on some raw-accuracy probes; the model recovered to higher scores with verification in the loop. Any frontier model — open or closed — needs evaluation on your data before going to production. Use the captured evals to gate the migration. (codersera.com Kimi K3 guide)
- Keep the closed on standby for the work local cannot do well enough yet. Your gateway routes automatically — you do not make a binary decision.
This is the real "best of both worlds"; the closed model is your scaffolding and your fallback, the open infrastructure is what you own. It works because the gateway already abstracts away which model serves the request.
For how to compare candidate models head-to-head before migrating, see How to Test Frontier AI Models Side by Side in 2026. For running an autonomous agent locally without paying per-token forever, see How to Run Hermes Agent on Free Hunyuan 3 (2026).
What this means for you
If you are a developer wondering where to start: do not start with the biggest model. Start with the stack — buy a 64 GB Mac if you can, install Ollama, run qwen3:8b, and put Open WebUI in front of it. That is your Layer 1–3 and your Layer 5 interface working end-to-end in an afternoon. When that feels tight, add LiteLLM as the gateway and route the hard tasks to a closed API while keeping the cheap and sensitive work local. Cheapest on-ramp, fastest to a working setup, and the gateway means every future choice is reversible.
If you are a team: skip the personal-experiment step. Put vLLM behind LiteLLM on real GPUs, route through LiteLLM to your closed vendor of choice, and serve Open WebUI to your users. Start with Qwen for general work, add DeepSeek V4 or Kimi K3 once you have evals that prove the migration.
If you already run closed-only: the model layer is now good enough that you should at least mirror your most frequent workflows on local infra for vendor-risk insurance. The brief July 2026 US delisting of Anthropic's Fable/Mythos models — restored days later — is a recent reminder that vendor dependency has tail risk that gets cheaper to hedge every quarter. (interconnects.ai)
Layer-by-layer starters (one command each)
- Hardware: 64 GB unified-memory Mac, or any NVIDIA GPU with ≥16 GB VRAM, or a cloud A100/H100 if you are serving a team.
- Model (after install):
ollama pull qwen3:8b— one of the most-downloaded open-weight models on Hugging Face today. - Engine:
ollama serve(the daemon behindollama run), orvllm serve <model>for production. Both expose an OpenAI-compatible endpoint. - Gateway:
docker run -p 4000:4000 ghcr.io/berriai/litellm:main-latestwith a config.yaml listing your endpoints. Point your apps atlocalhost:4000. - Interface:
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main→ openhttp://localhost:3000. - Reachability: install Tailscale on every device you want on the network and they share one private subnet, no ports opened.
FAQ
Q: What is the difference between open-source AI and open-weight AI? A: Open-source means the training code and data are publicly available, so you could in principle reproduce the model yourself. Open-weight means the trained weights are downloadable under a license — usually MIT or Apache 2.0 — but you cannot reproduce them without the training pipeline. Almost everything marketed as "open-source AI" in 2026 is actually open-weight. Always check the model card for the exact license.
Q: How much RAM do I need to run a 7B model locally? A: About 5 GB at Q4_K_M (4-bit quantization, the recommended default), ~8 GB at Q8_0, and ~14 GB uncompressed at FP16. Add ~8 GB for macOS/Windows overhead. A 16 GB machine runs a 7B model comfortably; a 64 GB machine runs a 34B model at Q5 or a 70B at Q3 tight. (PromptQuorum)
Q: Ollama or vLLM — which one should I use? A: Ollama for one user, prototyping, agents on a laptop, and any path where setup time is the bottleneck. vLLM for production once more than one request is in flight at a time — its PagedAttention algorithm gives 16–20× Ollama's throughput at peak concurrency. The gap goes to zero at one user.
Q: Is Kimi K3 really self-hostable? A: The weights are scheduled to be downloadable under Modified MIT (released to Hugging Face around July 27, 2026) — so in principle yes. In practice K3's 2.8T parameters at 4-bit are ~1.4 TB of weights; Moonshot recommends a 64+ accelerator supernode for serious inference. Almost everyone will use K3 via the API at $3 / M input and $15 / M output, and self-host smaller open-weight siblings instead. (kimi.com/blog/kimi-k3; interconnects.ai)
Q: Do I lose quality by running models quantized at Q4? A: Minimal — Q4_K_M typically loses 1–3% on standard benchmarks versus FP16, while using about 30% of the memory. The loss is more noticeable on coding and reasoning than chat. If you have the memory budget, Q5_K_M or Q6_K are safer; if you do not, prefer a smaller model at Q6 over a bigger one at Q3 — quality-from-precision beats quality-from-parameter-count once you compress hard enough. (vucense.com)
Q: What is LiteLLM for and do I need it? A: LiteLLM is an open-source AI gateway that exposes 100+ model providers (OpenAI, Anthropic, OpenAI-compatible local servers like Ollama and vLLM, Bedrock, Azure) through one OpenAI-shaped API. You need it once you have more than one model — local or closed — because it lets your application code speak one protocol while you swap engines and providers underneath. It also adds spend tracking, virtual keys, fallbacks, and caching.
Q: Why do people say local AI has "vendor risk" — what does that actually mean? A: A closed API can change price, throttle rate limits, change its terms of service, briefly get delisted by regulators, or be discontinued by the vendor — and you have no fallback. With an open-weight model on local hardware you control, none of those things can pull the model away from you. The upfront cost is hardware; the hedge is permanence.
Q: Can I really "use closed APIs to build open infrastructure I keep"? A: Yes — and increasingly, this is the standard workflow. The closed frontier is good enough at understanding intent that you spend Claude/Codex tokens writing your prompts, configs, evals, and infra-as-code. Those artifacts are model-agnostic — once the setup works, you point it at a local model served through LiteLLM and the closed vendor is no longer in the loop until you specifically need it. The closed model is scaffolding; the open infrastructure is the building you keep.
Discussion
0 comments