An on-device AI personal assistant is a locally-run language model that reads everything on your Mac — files, messages, email, notes, screenshots — builds a private knowledge base, and then proactively offers to do your work when you wake up. The category went mainstream in 2026 because Apple Silicon got fast enough, small open models got smart enough, and the cost of running cloud APIs became a real business concern — so the winning products run 80-90% of their compute on your machine and only call a frontier model for the hard parts.
This guide explains how the architecture actually works, what the leading free open-source tool (Sentient OS) does, how to set it up, and where the approach still falls short — so you can decide whether proactive local AI belongs in your workflow today.
Last verified: 2026-08-03 · Primary tool covered: Sentient OS (free, open-source, macOS) · Runs on: Apple Silicon Macs with 8 GB+ RAM · Cost: $0 in model usage if you BYO ChatGPT/Claude subscription for the frontier layer
What Is an On-Device AI Personal Assistant?
An on-device AI personal assistant is an AI agent that runs a language model directly on your hardware (your Mac, phone, or laptop) instead of sending every query to a cloud API. The key distinction from a chatbot like ChatGPT Desktop or Claude Desktop is that the model's primary job is to continuously read and understand your data, not to respond only when you type a prompt.
Sentient OS — the free open-source app that reached the top of r/macapps in July 2026 after building a 2,500-person waitlist — is the reference product for this category. Per the official docs at sentient-os.ai, it runs a custom fork of LiteRT-LM hosting Google's Gemma 4 E4B model (~4.5 B effective parameters) during overnight processing. The model decodes local app databases (WhatsApp, iMessage, Apple Notes) and generates markdown summaries of each item, which are stored as folders of plain markdown you control.
That knowledge base is then made available to your regular AI subscription (ChatGPT, Claude, or a local model of your choice) so that when you ask a question, the model already has the full context of your life without you having to manually paste anything in. This is the same memory-layer principle we covered in the three-layer memory blueprint every AI agent OS needs in 2026 — but Sentient OS goes one step further by automating the ingestion step overnight instead of waiting for you to manually upload context. The same pattern shows up in coding agents that learn from past sessions, as we wrote in our Claude Code agent memory guide; the difference here is Sentient applies it to your entire digital life, not just a codebase.
Why Is This Suddenly Possible in 2026?
Three things needed to converge to make on-device proactive AI viable: the model, the math, and the hardware.
1. Small models became genuinely competent. Google released Gemma 4 in April 2026 under Apache 2.0, with the E4B tier (~4.5 B effective parameters, 128K context window) explicitly designed for mobile and laptop deployment. The official Gemma 4 technical report notes the small models support multimodal understanding (image, video, audio) natively — which is what allows a model that small to actually understand screenshots and memes, not just text.
2. Inference math got clever. Sentient OS and similar local agents use a technique called KV cache reuse — because every file that goes through the overnight loop receives the same prompt ("summarize, classify, is this junk, is this sensitive?"), the model can reuse its key-value cache between files instead of recomputing it. Combined with flash attention and speculative decoding, that drops overnight inference cost from "impossible" to "your Mac sleeps through it."
3. Apple Silicon unified memory removed the bottleneck. An M-series Mac with 8 GB of unified RAM can hold a quantized Gemma 4 E4B at INT4 (~3 GB of VRAM, leaving 5 GB for the OS), and runs inference directly on the Neural Engine or GPU using the same memory pool the CPU uses. That is why the official Gemma 4 on-device deployment guide lists the E4B at Q4_K_M as the pick for 8 GB Apple Silicon machines.
How the Architecture Works: The Three-Layer Split
A modern on-device proactive assistant uses a hybrid design that splits work between a small local model and a frontier cloud model. Sentient OS openly describes this split on its site: the on-device model "runs 90% of Sentient's compute" for summarization, filtering, and knowledge-base generation; the frontier model (your ChatGPT Plus subscription, your Claude subscription, or OpenRouter / LM Studio) handles complex reasoning and executes multi-step tasks through a coding-agent CLI like OpenAI's Codex CLI — the same pattern we wrote about in our guide to using OpenAI Codex CLI for free in 2026.
| Layer | What runs | Where | Cost | Why |
|---|---|---|---|---|
| 1. On-device triage | Gemma 4 E4B (custom LiteRT-LM fork) | Your Mac (overnight, lid shut) | $0 (your electricity) | Continually summarize and classify everything you generate |
| 2. Knowledge base | Plain markdown folders | Your Mac, zero-access encrypted | $0 | Persistent, human-readable, editable memory of your life |
| 3. Frontier compute | Your ChatGPT / Claude / OpenRouter subscription | Cloud (your account, not the startup's servers) | Already paid | Reasoning, planning, and executing multi-step actions (computer use) |
The marginal cost per user is ~$0 for the startup because the heavy compute either runs on your battery/electricity (layer 1) or on a subscription you were already paying for (layer 3). The startup itself runs no inference servers — that is the whole moat.
How to Install Sentient OS (Step by Step)
Sentient OS is free and open-source. The official installer is a Homebrew cask.
Step 1 — Install via Homebrew
Open Terminal on an Apple Silicon Mac (mac 12.7+ recommended) and run:
brew install --cask sentient-os-labs/tap/sentient-os
You can also download the DMG directly from sentient-os.ai if you prefer a GUI install.
Step 2 — Grant Full Disk Access
MacOS requires you to explicitly authorize the app to read your files. Open System Settings > Privacy & Security > Full Disk Access and enable the toggle for Sentient OS. The app will prompt you to do this on first launch; it cannot read your WhatsApp database or iMessage archive without it. This is the same permission any local backup tool requires.
Step 3 — Connect your frontier model
You need an existing subscription to ChatGPT Plus ($20/month, the most common route per OpenAI's Codex CLI documentation) or a Claude plan. The app sets up a locally-installed Codex CLI that uses your subscription's API quota. You can also point it at an OpenRouter key or a local model via LM Studio. None of your raw files ever leave your Mac — only the small markdown summaries are passed as context when needed. The official page verifies this: "no accounts; raw data never leaves."
Step 4 — Leave your Mac plugged in, lid shut, overnight
Sentient runs its daily processing loop at 3:00 AM. The app uses thermal and power gates to avoid overheating your machine, and it only runs while the lid is shut and the Mac is plugged in — so you do not lose daytime performance. When you open the lid in the morning, you get "Morning Cards" — proactive suggestions for what it noticed overnight and can act on if you tap them.
Step 5 — Review your knowledge base
The KB lives in a folder of plain markdown under your user directory. Open it in any editor to see what the model knows about you. You can delete entries manually if you want the model to forget something — a level of control you do not get with a cloud assistant like Google Gemini Advanced that stores your context on Google servers.
On-Device vs Cloud AI Assistants: A Practical Comparison
The advantage is not raw intelligence — frontier cloud models (GPT-5.6, Claude 4.6 Opus) are still smarter than Gemma 4 E4B. The advantage is context, control, and cost.
| Dimension | On-Device (Sentient OS, Jan.ai, LM Studio) | Cloud (ChatGPT Desktop, Claude Desktop, Gemini Advanced) |
|---|---|---|
| Personal context | Reads WhatsApp, iMessage, all your local files automatically | Requires you to manually upload or paste context per message |
| Proactivity | Wakes overnight, prepares task suggestions for the morning | Reactive — only responds when you prompt |
| Privacy | Raw files never leave your Mac; KB is yours to edit and delete | Uploaded to vendor cloud per their data policy |
| Cost | $0 in model usage (uses your Mac + your existing subscription) | $20+/month subscription, billed regardless of usage volume |
| Model capability | Local model is smaller (4-12 B), suited for triage/summarization | Frontier models are smarter on multi-step reasoning |
| Walled-garden access | Can decode WhatsApp/iMessage databases locally | Apple only sees iMessage; Google only sees Gemini/Gmail |
| Internet required | No — works fully offline once your models are downloaded | Yes — every query requires a round trip |
Why Apple and Google Struggle to Build This
A question worth asking: if proactive personal AI is so clearly valuable, why is a tiny open-source team (Sentient-OS-Labs on GitHub) building it instead of Apple Intelligence or Google Gemini?
The answer is the walled garden. Apple Intelligence only reads iMessage, Apple Mail, Apple Notes, and Apple Files — it cannot reverse-engineer a WhatsApp SQLite database because that is a Meta-owned app. Google Gemini Advanced can search your Gmail and Drive (per Google's own description), but it cannot read your iMessage archive because that is an Apple-owned database. Neither company can legally reach into the other's application data without an explicit API agreement.
A third-party open-source app does not face that constraint. Sentient OS reads WhatsApp by directly parsing its local SQLite database on your machine (using "typedstream decoding and protobuf walks," per the official site). The data never leaves your Mac, so Meta and Apple's data-sharing rules do not apply — you are reading your own file from your own disk.
Apple Intelligence also has a business-model constraint: it lacks incentive to ship local inference aggressively because it does not earn per-token revenue. Google has the same issue from the opposite direction — it would rather have you use Gemini Advanced's cloud API (metered) than run a model you downloaded once and use forever. This is exactly the strategic gap that defines what we call the five levels of AI building in 2026 — the level where a small team can outmaneuver the giants is "own a layer the giants can't or won't ship."
What This Means for You
If you do knowledge work, customer support, or any task that involves juggling dozens of open tabs, threads, and files, a proactive local AI assistant does two things a cloud chat cannot:
- It removes the context-paste step. No more pasting your relevant Slack thread, Notion page, and email into ChatGPT before asking a question. The model has already read all of them.
- It works while you sleep. Open your laptop in the morning to a draft reply to an email you did not have time to answer last night, a reminder that a client asked for an invoice three days ago and you forgot, or a note that two files have the same customer's name spelled two different ways and you might want to reconcile them.
The trade-off is you must be comfortable granting Full Disk Access to a third-party app and accepting that the local model is not as smart as GPT-5.6. For privacy-sensitive knowledge work — legal documents, client emails, financial files — the no-cloud-leak property is the whole point. This is also why the broader "large action models vs plain LLMs" debate (which we covered separately) keeps coming back to architecture: a smart model without a local context layer is just a chatbot waiting for you to type.
How to Pick Your Stack If You Do Not Want Sentient OS
If you want the architecture without committing to one tool, you can build the three-layer split yourself with off-the-shelf open-source parts:
- Local model runtime: LM Studio (GUI-native, MLX-optimized for Apple Silicon) or Ollama (CLI, the gold standard for Linux/Windows) — both are free, both support Gemma 4 E4B out of the box.
- Knowledge base storage: Obsidian (plain markdown vault, free for personal use) or just a folder of markdown files in your home directory.
- Coding-agent / computer-use layer: OpenAI Codex CLI (free, open-source, Apache 2.0) or Claw Code by Anthropic — both can read your KB folder as context and execute multi-step actions against the frontier model you authenticate with.
The advantage of a self-built stack is control and portability across machines. The disadvantage is you lose Sentient OS's purpose-built inference optimizations (KV cache reuse across repeated prompts, thermal and battery gating, and the database decoders for WhatsApp/iMessage) — you have to build or skip those yourself. For a deeper dive on building a free local agent stack with off-the-shelf open-source parts, see our guide to building a free AI agent team with Hermes Agent and OpenCode in 2026.
FAQ
Q: What is the difference between on-device AI and a locally-installed chatbot? A: A locally-installed chatbot (Jan.ai, LM Studio running as a chat server) responds to prompts you type. An on-device personal assistant (Sentient OS) continually reads your data on its own schedule and proactively surfaces suggestions, not just answering when asked. The differentiator is proactivity — that requires background processing, database decoders for apps like WhatsApp, and a persistent local KB.
Q: Do I need a ChatGPT or Claude subscription to use Sentient OS? A: No, but you lose the "actually do the work" layer. Sentient OS's local Gemma 4 E4B model handles triage, scoring, and the knowledge base on its own — that part is fully local and needs no subscription. The computer-use execution layer (the agent that clicks around your apps to actually finish tasks) routes through a frontier model. You can run your own local model via LM Studio or Ollama as the frontier layer, but for complex multi-step tasks you will want a frontier-class model.
Q: Does on-device LLM inference drain my Mac battery or wear out the hardware? A: This is why the 3 AM overnight loop matters — Sentient OS only runs when your Mac is plugged in and the lid is shut, and it uses thermal and AC gates to throttle itself. You do not lose daytime battery or performance. For users who close their laptops overnight, the trade-off is just a bit of electricity (roughly negligible on modern Apple Silicon). The official HN post from the founder notes the model "trades speed for not cooking the device."
Q: Is Sentient OS really private if it reads all my files? A: The privacy model: the app has Full Disk Access locally (required for reading WhatsApp/iMessage databases), but raw data never leaves your Mac. Only small markdown summaries are passed as context to your frontier model when you actually ask a question. Sentient OS explicitly states it operates with "no accounts, no user data, no readable user data, zero-access encrypted ciphertext only." You can audit the code on GitHub at Sentient-OS-Labs/sentient-os.
Q: What hardware do I need to run an on-device AI personal assistant on a Mac? A: For Sentient OS running Gemma 4 E4B at INT4 quantization, an Apple Silicon Mac with 8 GB of unified memory is the minimum. The official Gemma 4 hardware recommendations put the E4B at Q4_K_M at 3 GB VRAM, leaving 5 GB for the OS on an 8 GB machine. For users who want to also run a larger local frontier model (Gemma 4 12B or 26B), 16-32 GB of unified memory is recommended.
Q: Can I run on-device proactive AI on Windows or Linux, not just Mac? A: As of this writing Sentient OS is macOS-only (the database decoders target Apple's iMessage/Notes formats specifically and the inference engine is MLX-optimized for Apple Silicon). On Windows or Linux, you can build a similar stack with Ollama + Codex CLI running a Gemma 4 E4B GGUF file, but you lose the WhatsApp/iMessage decoders and have to provide your own data ingestion layer.
Q: Is Sentient OS the same as the Peter Thiel-backed "Sentient" crypto project? A: No — they are two completely different organizations with no shared investors or founders. The crypto "Sentient" (Sentient Foundation, announced in July 2024) builds open-source AI models with blockchain-based contributor rewards and raised an $85M seed from Founders Fund. "Sentient OS" (Sentient-OS-Labs, founded July 2026 per its GitHub history) is a free open-source macOS app. Do not confuse the two when researching.
Discussion
0 comments