Last verified: 2026-08-04 · Core Tech:
cua-driver(formerly Kua, MIT) · Best for: Parallel desktop automation · Supported OS: macOS, Windows, Linux · Setup: ~5 minutes Volatile facts: model pricing, cua-driver version and platform support change often — last checked August 2026.
Why "Invisible" Agents Change the Game
Most AI "computer use" demos share a fatal flaw: Foreground Dependency. When you tell an agent to click a button, it takes over your mouse. If you touch your trackpad, the agent crashes or misclicks. You are effectively held hostage by your own automation.
Hermes Agent solves this through a "No-Foreground" contract. By using native process-level event injection instead of raw HID (Human Interface Device) emulation, the agent can:
- Act on occluded windows: It can click buttons in an app that is buried under other windows.
- Synthesize events without cursor warp: It posts clicks directly to the application's event queue. Your real OS cursor stays exactly where you left it.
- Bypass AppNap and Focus: It keeps application accessibility trees alive even when the app is "sleeping" or out of focus.
Your active window does not jump to the front, and your virtual desktop or macOS Space does not switch. You can keep typing in your editor while the agent clicks around in a browser in another window. A faint tinted overlay cursor shows you where the agent is working; your real cursor stays put.
| Feature | Traditional Computer Use | Hermes Background Mode |
|---|---|---|
| Cursor Control | Hijacks real mouse | Independent virtual cursor |
| Multitasking | Impossible (Screen lock) | Full parallel use |
| Targeting | Visual/Pixel only | Accessibility Tree + Vision |
| Model Lock-in | Usually locked to Claude | Any Vision Model (local/cloud) |
| Security | Minimal/Foreground only | Built-in destructive blocks |
How Background Computer Use Works (The Tech)
The secret sauce is the cua-driver (Computer Use Agent Driver), an open-source MIT-licensed background driver from trycua/cua. Hermes speaks the Model Context Protocol (MCP) over standard I/O to it. The driver acts as a hardware abstraction layer that translates high-level AI commands ("Click the Save button") into native OS calls that do not require window focus.
| Platform | Accessibility tree | Input dispatch | Key behaviour |
|---|---|---|---|
| macOS | AX (private SkyLight SPIs) | SLPSPostEventRecordTo — pid-scoped, no cursor warp |
Events posted directly to target processes; no HID event tap |
| Windows | UIAutomation | SendInput + PostMessage — no focus steal |
Works without elevation for non-UAC windows |
| Linux | AT-SPI (X11 + Wayland) | XTest (X11) / virtual-keyboard (Wayland) | XWayland bridge needed on pure Wayland |
The mechanism is the same everywhere: route synthetic input events to specific process IDs instead of the shared system cursor. Each action is scoped to a window and process. The agent captures a screenshot with a set-of-marks overlay numbering every clickable element, decides what to click, sends the event, and captures again to verify.
That overlay is also what makes it affordable. Instead of sending a massive high-res screenshot for every move, the agent works from a semantic map of the screen, and Hermes prunes old screenshots from context. A 20-action session on a 1568×900 display costs roughly 30K tokens of screenshot context rather than 600K — the ~95% compression figure you see quoted for Set-of-Mark capture [1][2].
How to Set Up Hermes Computer Use
If you already have Hermes Agent installed, enabling background computer use takes one command. If you do not, start with the one-line installer.
Step 1 — install the driver
hermes computer-use install
This fetches and runs the upstream cua-driver installer — install.sh on macOS and Linux, install.ps1 on Windows. Confirm it landed:
hermes computer-use status
You can also enable it interactively: run hermes tools, pick "Computer Use (macOS/Windows/Linux)", and the same upstream installer runs.
Step 2 — grant platform permissions
| Platform | What to grant |
|---|---|
| macOS | System Settings → Privacy & Security → Accessibility + Screen Recording → allow your terminal (or the Hermes desktop app). |
| Windows | Nothing at install time. If driving over SSH rather than RDP or console, you need the autostart pattern for the Session 0 ↔ Session 1+ proxy. |
| Linux | DISPLAY set for X11, or XDG_SESSION_TYPE=wayland for Wayland. Pure Wayland needs an XWayland bridge for screen capture. AT-SPI must be enabled (default on GNOME, KDE, Xfce). |
Step 3 — start a session with computer use enabled
hermes -t computer_use chat
Or add computer_use to your enabled toolsets in ~/.hermes/config.yaml so it is always available.
Step 4 — verify with a simple prompt
Give the agent something small: "open my notes app and type 'hello world'", or for a safer first test, "take a screenshot of my desktop and tell me what windows are open." You should see the tinted agent cursor move while your real cursor stays put.
Step 5 — run diagnostics if anything fails
hermes computer-use doctor
This runs cua-driver's structured health check and prints a per-check matrix: a missing permission, a stale binary, a display-server problem. It is the first thing to run when anything is off.
Which models work with it?
Hermes Computer Use works with any tool-capable model that can process screenshots. It does not lock you into Anthropic's native computer-use schema the way some integrations do, and that is one of its biggest advantages.
| Provider | Vision | Works | Notes |
|---|---|---|---|
| Anthropic (Claude Sonnet/Opus 3+) | Yes | Yes | Best overall; set-of-marks plus raw coordinates |
| OpenAI (GPT-4+, GPT-5) | Yes | Yes | Multi-part tool messages supported |
| Google (Gemini 2+) | Yes | Yes | Tool calling and vision both supported |
| OpenRouter (any vision model) | Yes | Yes | Route to whichever model fits the task |
| Local vLLM / LM Studio / Ollama | Yes | Yes | If the model supports multi-part tool content |
| Text-only models | No | Degraded | Use mode="ax" for accessibility-tree-only operation |
The practical implication: run routine clicking and typing through a free or cheap model and save premium models for reasoning-heavy steps. With a free-tier OpenRouter model or a local model on your own hardware, the running cost is zero.
5 High-Leverage Background Workflows for 2026
You don't need to watch your agent work. Here are five ways to use Hermes Agent in the background today:
- Continuous Content Audits: While you write in one window, the agent can crawl your Obsidian or Notion vault, checking for broken links or outdated stats.
- Asynchronous CRM Cleanup: Have an agent go through your browser-based CRM in the background, verifying LinkedIn profiles and updating lead statuses while you're on a sales call.
- Automated "File Triage": Set an agent to monitor your Downloads folder. When a file arrives, it opens it, reads the content, and moves it to the correct project folder—all without a single window popping to the front.
- Member Onboarding Checks: For communities, an agent can check new Discord or Teams members against a payment database and assign the correct roles silently.
- Secure Local Automation: Because Hermes Agent is model-agnostic, you can run a local vision model (like Qwen 2-VL) to handle sensitive financial or HR data without the screenshots ever leaving your machine.
Beyond those, the cross-app shuffle is where this earns its keep:
- Email triage and drafting: "Read my inbox, identify the three emails that need replies today, draft the replies, save them to Drafts."
- Note-taking and journaling: "Open Notes, capture a journal entry about today's decisions, tag it with the right project."
- File organisation: "Move every PDF on my Desktop into the correct project folder based on filename context, and rename them consistently."
- Cross-app workflows: "Take this morning's meeting notes, summarise the action items, paste them into the project tracker, create follow-up tasks." That is the multi-step shuffle that used to eat 30 minutes a day.
For firing off tasks from your phone while the agent keeps working on the machine at home, Hermes connects to Telegram, Discord and other platforms — you approve actions remotely and it carries on. See our guide on accessing your AI agent dashboard remotely with Tailscale for the networking layer.
Can you run several agents at once?
Yes. Each Hermes run gets its own session ID and its own agent cursor on screen, so agents work concurrently without tripping over each other — one sorting files while another writes notes. Screenshot pruning keeps long multi-action sessions from bogging the machine down.
To extend this into full orchestration, see our guide to running an AI agent operating system and the walkthrough of building a free AI agent team with Hermes Agent and OpenCode.
What safety guardrails are there?
Two permission modes:
| Mode | How it works | Best for |
|---|---|---|
| Manual / smart approvals (default) | Any potentially destructive action — deleting a file, running a shell command — stops and asks before executing. You see what the agent wants to do and can approve, deny, or stop mid-task. | Daily work; everything except throwaway VMs |
| YOLO / unrestricted | One explicit risk acceptance, no runtime prompts. The unrestricted daemon is private to that session and ends with it. | Disposable VMs only; never your real machine |
In the default mode Hermes also hard-blocks dangerous patterns: it will not type passwords, will not execute shell payloads matching its dangerous-pattern lists, and blocks emptying the trash or logging you out. Prompt-injection protection is layered on top — the agent is instructed to ignore instructions hidden inside screenshots or web pages.
YOLO mode explicitly does not protect against prompt injection or unintended input. The official documentation recommends it only inside a disposable VM whose accounts and data you would accept losing entirely.
How does it compare to other tools?
| Tool | Background mode | Free / open source | Cross-platform | Any model | Permission guardrails |
|---|---|---|---|---|---|
| Hermes Computer Use | Yes | Yes (MIT) | macOS, Windows, Linux | Yes | Yes — destructive actions require approval |
| Anthropic Claude Computer Use | No (foreground) | Paid API | macOS only | Claude only | Limited |
| OpenAI Operator | No (foreground, web only) | Paid | Web only | GPT only | Yes |
| Manus | No | Paid | Web only | Proprietary | Yes |
| pyautogui / AutoHotkey | No — steals cursor | Free / open source | Cross-platform | No AI model | No |
Hermes is the only entry that combines all four: quiet background operation, no cost, cross-platform, and no single-provider lock-in.
What are the limitations?
- Background mode is slower than foreground. Accessibility-routed events take roughly 5–20 ms on macOS, 3–10 ms on Windows, 5–15 ms on Linux, against direct HID posting. Irrelevant at agent speed, unsuitable for real-time speed-runs.
- Long chains are less reliable. Scope tasks to 3–5 steps for clean output. Very long unattended chains hit stale element references and unexpected dialogs.
- macOS private SPI risk. Apple can change SkyLight's symbol surface in any OS update. Hermes warns when the installed cua-driver is older than its tested baseline; if you need reproducibility across a macOS bump, pin the version with
HERMES_CUA_DRIVER_VERSION. - Windows SSH sessions. SSH runs in Session 0, which has no interactive desktop. Drive from RDP or console, or set up the autostart pattern.
- Linux Wayland. Pure Wayland without XWayland produces empty captures. Use an X11 session or make sure the XWayland bridge is active.
- Token cost on premium models. Screenshots are not free. ~30K tokens per 20-action session adds up on Claude Opus or GPT-5. Route routine work to free or local models.
What This Means for You
The shift from foreground to background agents is the transition from "AI as a tool" to "AI as a coworker." You no longer have to choose between doing the work yourself or watching a robot do it for you. By leveraging Hermes Agent's Sidekick features, you can keep a small status sprite on your screen that tells you exactly what your "invisible" agent is accomplishing in the background.
If a meaningful part of your day goes on repetitive desktop work — sorting files, drafting routine email, moving data between apps, organising notes — this is a free second pair of hands that never gets bored. Start with a free model to validate the workflow, give the agent goals rather than micro-tasks, and approve its actions while you do the work that needs your brain.
For the fuller picture, see our guide to building a self-improving AI agent operating system, the three-layer memory blueprint, and the agent OS that ships any AI tool on demand.
FAQ
Q: Does it move my actual mouse? A: No. Hermes uses a virtual event synthesis engine. You can move your mouse freely while the agent clicks in another app. Your keyboard focus does not change and your virtual desktops do not switch.
Q: Is Hermes Computer Use free? A: Yes. Hermes Agent and cua-driver are both MIT-licensed open source. You only pay for model API calls, and free OpenRouter models or a local model bring that to zero.
Q: Which models support this? A: Any model with vision capabilities. This includes Claude 3.5 Sonnet, GPT-4o, Gemini 1.5/3.1, and local models like Qwen 2-VL via Ollama or OpenRouter. Text-only models degrade to accessibility-tree-only operation.
Q: Does it work on Windows and Linux? A: Yes, on macOS, Windows and Linux. Windows needs no special permissions for console sessions; Linux needs a reachable display server (X11, or Wayland with XWayland).
Q: How long does setup take?
A: About five minutes: one command to install the driver, then OS-level permissions (Accessibility and Screen Recording on macOS). Run hermes computer-use doctor if anything misbehaves.
Q: Which model should I start with? A: A free or cheap vision model — a free OpenRouter model or a local one via Ollama — to validate the workflow. Graduate to Claude Sonnet or GPT-5 for reasoning-heavy steps.
Q: Is it safe to let an AI control my computer? A: Hermes includes "destructive action" blocks. Any command involving deleting files, emptying the trash, or typing passwords is hard-blocked at the tool level or requires manual human approval. YOLO mode disables approvals and should only be used in a disposable VM.
Q: Can it be tricked by prompt injection? A: The system prompt is hardened to ignore any instructions found inside the applications it sees. It only follows the instructions you provide in the chat. That protection does not apply in YOLO mode.
Q: Can I run more than one agent at the same time? A: Yes. Each run gets its own session ID and its own agent cursor. Multiple agents work concurrently without interfering.
Every claim here is traced to a primary source, dated, and listed under Sources. Research and drafting are AI-assisted; editing, verification and publication are human decisions, and a person is accountable for what appears on this page. How we work →
Discussion
0 comments