Verdict: Hermes Agent's Computer Use feature is the most practical background desktop automation tool available in 2026. It lets an AI agent click, type, scroll, and drag across your real applications without moving your cursor, stealing keyboard focus, or switching virtual desktops. You and the agent work on the same machine at the same time. It is free, open source (MIT), works on macOS, Windows, and Linux, and runs with any vision-capable model — Claude, GPT, Gemini, or a local open model. Setup takes one command and about five minutes.
What Is Hermes Computer Use?
Hermes Computer Use is a feature of Hermes Agent — an open-source personal AI agent built by Nous Research — that gives the agent the ability to drive your real desktop. It can click buttons, type text, scroll pages, drag elements, and read what is on your screen, all through the operating system's native accessibility APIs.
The key differentiator is that everything happens in the background. Your real mouse cursor does not move. Your active window does not jump to the front. Your virtual desktop or macOS Space does not switch. You can keep typing in your editor while the agent clicks around in a browser in another window. This is fundamentally different from older automation tools like pyautogui or AutoHotkey, which take over your mouse and keyboard.
Under the hood, Hermes speaks the Model Context Protocol (MCP) over standard I/O to cua-driver, an open-source background computer-use driver licensed under MIT. The driver uses each platform's native accessibility and input stack:
| Platform | Accessibility tree | Input dispatch | Key behavior |
|---|---|---|---|
| macOS | AX (private SkyLight SPIs) | SLPSPostEventRecordTo — pid-scoped, no cursor warp |
Events posted directly to target processes |
| Windows | UIAutomation | SendInput + PostMessage — no focus steal |
Works without elevation for non-UAC windows |
| Linux | AT-SPI (X11 + Wayland) | XTest (X11) / virtual-keyboard (Wayland) | XWayland bridge needed on pure Wayland |
The result on every platform is the same: the agent reads the accessibility tree of any visible window and posts synthesized input events without bringing windows to the front or moving the real cursor (Hermes Computer Use docs, cua.ai no-foreground contract).
How Does Background Computer Use Work?
It works by routing synthetic input events to specific process IDs instead of the shared system cursor. On macOS, cua-driver uses private SkyLight SPIs (SLPSPostEventRecordTo) to post events directly to the target application — there is no HID event tap and no cursor warp. On Windows, it uses SendInput combined with PostMessage to dispatch input without focus theft. On Linux, it uses the XTest extension for X11 or a virtual-keyboard interface for Wayland compositors.
Each action the agent takes — a click, a type, a scroll — is scoped to a specific window and process. The agent captures a screenshot (with a set-of-marks overlay showing every clickable element numbered), decides what to click, sends the event, then captures again to verify. A faint tinted overlay cursor appears on screen to show where the agent is acting, but your real OS cursor remains wherever you left it (Hermes docs, trycua/cua on GitHub).
A 20-action session on a 1568×900 display typically costs about 30K tokens of screenshot context, not 600K, because Hermes prunes old screenshots from context and estimates each image at ~1,500 tokens (Hermes Computer Use docs).
How to Set Up Hermes Computer Use (Step by Step)
Setting it up takes about five minutes. The prerequisite is that Hermes Agent itself is already installed — if you do not have it yet, run the one-line installer.
Step 1: Install the computer-use driver
Open your terminal and run:
hermes computer-use install
This fetches and runs the upstream cua-driver installer — install.sh on macOS and Linux, install.ps1 on Windows. You can verify the install with:
hermes computer-use status
Alternatively, you can enable it interactively: run hermes tools, pick "Computer Use (macOS/Windows/Linux)", and the setup runs the same upstream installer (Hermes docs).
Step 2: Grant platform permissions
After install, grant the OS-level permissions the driver needs:
| Platform | What to grant |
|---|---|
| macOS | System Settings → Privacy & Security → Accessibility + Screen Recording → allow your terminal (or the Hermes desktop app). Run hermes computer-use doctor to see which permission is missing. |
| Windows | No permissions needed at install time. If driving over SSH (not RDP or console), you need the autostart pattern for the Session 0 ↔ Session 1+ proxy (cua.ai/docs). |
| Linux | Ensure DISPLAY is set for X11 sessions, or XDG_SESSION_TYPE=wayland for Wayland. Pure Wayland needs an XWayland bridge for screen capture. AT-SPI must be enabled (default on GNOME, KDE, Xfce). |
Step 3: Start a session with computer use enabled
hermes -t computer_use chat
Or add computer_use to your enabled toolsets in ~/.hermes/config.yaml so it is always available.
Step 4: Verify with a simple prompt
Give the agent a simple instruction like "open my notes app and type 'hello world'" — or, for a safer test, "take a screenshot of my desktop and tell me what windows are open." You should see the tinted agent cursor move while your real cursor stays put.
Step 5: Run diagnostics if anything fails
hermes computer-use doctor
This runs cua-driver's structured health check and prints a per-check matrix. It tells you exactly what is wrong — a missing permission, a stale binary, a display server problem. It is the first thing to run when anything is off (Hermes docs).
Which Models Work With Hermes Computer Use?
Hermes Computer Use works with any tool-capable model that can process screenshots. It does not lock you into Anthropic's native computer-use schema the way some integrations do. This is one of its biggest advantages.
| Provider | Vision? | Works? | Notes |
|---|---|---|---|
| Anthropic (Claude Sonnet/Opus 3+) | Yes | Yes | Best overall; set-of-marks + raw coordinates |
| OpenAI (GPT-4+, GPT-5) | Yes | Yes | Multi-part tool messages supported |
| Google (Gemini 2+) | Yes | Yes | Tool-calling and vision both supported |
| OpenRouter (any vision model) | Yes | Yes | Route to whichever model fits the task |
| Local vLLM / LM Studio / Ollama | Yes | Yes | If the model supports multi-part tool content |
| Text-only models | No | Degraded | Use mode="ax" for accessibility-tree-only operation |
Source: Hermes Computer Use provider compatibility
The practical implication: you can run routine clicking and typing tasks through a free or cheap model and save premium models for reasoning-heavy steps. This keeps your total cost near zero if you use a free-tier model from OpenRouter or a local model on your own hardware.
What Safety Guardrails Does Hermes Computer Use Have?
Hermes Computer Use has built-in guardrails designed to keep you in control. The system operates in two permission modes:
| Mode | How it works | Best for |
|---|---|---|
| Manual/smart approvals (default) | Any potentially destructive action — deleting a file, running a shell command — stops and asks for your approval before executing. You see what the agent wants to do and can approve, deny, or stop mid-task. | Daily work; everything except throwaway VMs |
| YOLO / unrestricted | One explicit risk acceptance; no runtime approval prompts. The unrestricted daemon is private to that session and ends when the session closes. | Disposable VMs only; never on your real machine |
In the default mode, Hermes also hard-blocks dangerous patterns: it will not type passwords, it will not execute shell payloads matching dangerous pattern lists, and it blocks actions like emptying the trash or logging you out. Prompt injection protections are layered in — the agent is instructed not to follow sneaky instructions hidden inside screenshots or web pages (Hermes docs).
YOLO mode explicitly does not protect against prompt injection or unintended input. The official documentation recommends using it only inside a disposable VM with accounts and data whose full compromise you accept.
What Can You Actually Do With It?
The practical use cases for background computer use fall into a few categories — all of which can run while you keep working on something else on the same machine:
- Email triage and drafting: "Read my inbox, identify the three emails that need replies today, draft the replies, and save them to my Drafts folder."
- Note-taking and journaling: "Open Notes, capture a journal entry about today's decisions, and tag it with the right project."
- File organization: "Move every PDF on my Desktop into the correct project folder based on filename context, and rename them consistently."
- Knowledge-management maintenance: If you use a tool like Obsidian as a memory layer, point Hermes at your vault to organize and link notes while you focus on content. For more on this pattern, see our guide to building an AI agent OS with a three-layer memory blueprint.
- Cross-app workflows: "Take this morning's meeting notes, summarize the action items, paste them into the project tracker, and create follow-up tasks." This is exactly the kind of multi-step, cross-app shuffle that used to eat 30 minutes of your day.
For remote work — firing off tasks from your phone while your agent keeps working on your machine back home — Hermes connects to messaging platforms like Telegram and Discord. You can approve actions from your phone and let the agent carry on. See our guide on accessing your AI agent dashboard remotely with Tailscale for the networking layer.
Can You Run Multiple Agents at Once?
Yes. Each Hermes run gets its own session ID and its own agent cursor on screen. You can have multiple agents working concurrently without them tripping over each other — one sorting files in the background while another writes notes. Hermes is also smart about screenshot management: it prunes old screenshots from context, so even long multi-action sessions do not slow your machine to a crawl.
This multi-session capability is part of what makes Hermes a genuine agent platform, not a single-shot toy. If you want to extend this further into a full multi-agent orchestration setup, see our guide to running an AI agent operating system and our walkthrough of building a free AI agent team with Hermes Agent and OpenCode.
How Does Hermes Computer Use Compare to Other Tools?
| Tool | Background mode | Free / open source | Cross-platform | Any model | Permission guardrails |
|---|---|---|---|---|---|
| Hermes Computer Use | Yes | Yes (MIT) | macOS, Windows, Linux | Yes | Yes — destructive actions require approval |
| Anthropic Claude Computer Use | No (foreground) | Paid API | macOS only | Claude only | Limited |
| OpenAI Operator | No (foreground, web only) | Paid | Web only | GPT only | Yes |
| Manus | No | Paid | Web only | Proprietary | Yes |
| pyautogui / AutoHotkey | No — steals cursor | Free / open source | Cross-platform | No AI model | No |
Hermes is the only tool in the table that combines all four: runs quietly in the background, costs nothing, works cross-platform, and is not locked to a single model provider. Sources: Hermes docs, trycua/cua on GitHub, Nous Research.
What Are the Limitations?
- Background mode is slower than foreground. Accessibility-routed events take ~5–20 ms on macOS, ~3–10 ms on Windows, ~5–15 ms on Linux versus direct HID posting. Not noticeable for agent-speed clicking, but not suitable for real-time speed-runs (Hermes docs).
- Complex multi-step tasks are less reliable. Scope tasks to 3–5 steps for clean output. Very long unattended chains can hit stale element references or unexpected dialogs.
- macOS private SPI risk. Apple can change SkyLight's symbol surface in any OS update. Hermes warns when the installed cua-driver is older than its tested baseline, but if you need reproducibility across a macOS bump, pin the driver version with the
HERMES_CUA_DRIVER_VERSIONenvironment variable. - Windows SSH sessions. SSH runs in Session 0, which has no interactive desktop. Drive from an RDP or console session, or set up the autostart pattern.
- Linux Wayland. Pure Wayland without XWayland will produce empty captures. Use an X11 session or ensure the XWayland bridge is active.
- Token cost on premium models. Screenshots are not free. A 20-action session is ~30K tokens, which adds up on Claude Opus or GPT-5. Route routine work to free or local models and save premium tokens for reasoning-heavy steps.
What This Means for You
If you spend a meaningful part of your day on repetitive desktop tasks — sorting files, drafting routine emails, moving data between apps, organizing notes — Hermes Computer Use gives you a free second pair of hands that never gets bored and never needs a break. The five-minute setup and any-model flexibility mean there is no barrier to trying it today. Start with a free model for routine work, give the agent goals (not just tiny tasks), and approve its actions while you keep doing the work that actually requires your brain.
For a deeper journey into building a full agent operating system around Hermes — with persistent memory, scheduled automation, and multi-platform access — see our guide to building a self-improving AI agent operating system and our walkthrough of the agent OS that ships any AI tool on demand.
FAQ
Q: Is Hermes Computer Use free?
A: Yes. Hermes Agent and cua-driver are both MIT-licensed open source. You only pay for model API calls, and you can use free models from OpenRouter or run a local model on your own hardware at zero cost.
Q: Does Hermes Computer Use work on Windows and Linux?
A: Yes. As of 2026, it works on macOS, Windows, and Linux. Windows needs no special permissions for console sessions; Linux requires a reachable display server (X11 or Wayland with XWayland). Source: Hermes Computer Use docs.
Q: How long does setup take?
A: About five minutes: one command to install the driver, then grant OS-level permissions (Accessibility + Screen Recording on macOS). Run hermes computer-use doctor if anything is not working.
Q: Will the agent move my cursor or disrupt my work?
A: No. Background mode means your real cursor does not move, your keyboard focus does not change, and your virtual desktops do not switch. You and the agent co-work on the same machine.
Q: Which model should I start with?
A: Start with a free or cheap model — a free OpenRouter vision model or a local model via Ollama — to validate your workflow. Then graduate to Claude Sonnet or GPT-5 for reasoning-heavy steps. This keeps your cost near zero for routine tasks.
Q: Can the agent delete files without my permission?
A: No. In the default permission mode, destructive actions require explicit approval before execution. Hermes also hard-blocks dangerous patterns like password typing, emptying trash, and dangerous shell payloads. YOLO mode disables approvals but should only be used in a disposable VM.
Q: Can I run more than one agent at the same time?
A: Yes. Each Hermes run gets its own session ID and its own agent cursor on screen. Multiple agents can work concurrently without interfering with each other.
Discussion
0 comments