<!-- Markdown mirror. Canonical: https://empryo.com/docs/tools/computer-use -->

# Computer use

> The agent reads and drives real UI — our browser, your Chrome or the machine — one accessibility-first protocol with leases, policy switches and a mote cursor.

## Overview

Computer use is the agent operating a real interface instead of a file: a page in
Empryo's browser, a tab in your own Chrome, an app window, the whole display. One
contract covers all of them — a `SurfaceDriver` per backend, one coordinator in
front, the same tools on every surface.

It is **off by default**. Turn it on with `computerUse: true`, or from
**Settings → Computer use** on the desktop and `/computer` in the TUI.

| Surface | What it can drive |
|---------|-------------------|
| Desktop app | Empryo's browser panel, your Chrome over CDP, apps and displays through the native helper |
| TUI | The same, minus the panel — `/computer` lists targets, permissions and live sessions |
| Headless | Browser lanes and the native helper; every `ask` decision denies with a reason (nobody is there to answer) |

## Read the tree, not the pixels

A screenshot costs ~1 500 tokens and still leaves the model guessing coordinates.
`computer_observe` returns the accessibility tree as indexed text instead:

```
[1] heading   "Form"                    (enabled)  actions: click  level="1"
[3] textbox   "Name"   value="Eru"      (focusable, enabled, editable)  actions: set_value, focus
[4] checkbox  "I agree"                 (focusable, enabled, checked)   actions: toggle, focus
[7] button    "Send"                    (focusable, enabled)  actions: click, focus
```

Five rules make it cheap, and they are enforced, not suggested:

- **Diff by default.** A second observation returns only what changed (`~` changed,
  `+` added, `-` removed). No change is an answer: *"act before observing again."*
- **Indices die on write.** Every batch that mutates the target invalidates them;
  using a stale index is refused with the latest snapshot id, never silently retried.
- **Batch, then observe once.** `computer_batch` runs a planned sequence under one
  lease and returns a single diffed observation.
- **Screenshots are opt-in** (`mode:"screenshot"` / `"both"`), for visual tasks or
  when the tree came back empty.
- **Per-app priors.** The first observation of a target carries a short operating
  note for that kind of app (Chrome's omnibox, Finder's list, a terminal's textarea)
  — once per incarnation, not per turn.

## The tools

| Tool | What it does |
|------|--------------|
| `computer_js` | The `cua` REPL: bind a target, batch actions, get the resulting state — one call |
| `computer_targets` | What can be driven right now: displays, apps and windows, browser tabs |
| `computer_observe` | The indexed AX text for one target, diffed; optional screenshot |
| `computer_act` | One action against the latest observation, then observe |
| `computer_batch` | Many actions in order, one lease, one trailing observation |
| `computer_tab` | Open, close or show a tab in the external-Chrome profile |
| `computer_workflow` | List, run or record a taught workflow |
| `computer_permissions` | What the OS has granted, and exactly how to grant the rest |

Actions are semantic first — `set_value`, `toggle`, `focus`, a named `perform`
action off the element — with coordinates as the fallback, because a role-and-name
click survives a layout change and a coordinate does not.

## One tool, written as code

`computer_js` is the fast lane, and it is on by default. The model writes
JavaScript against a persistent `cua` session; binding a target prints that
target's accessibility state into the same tool result, so binding and reading
are one round trip instead of two.

```js
let tab = await cua.createBrowserTab("empryo", "http://localhost:5173");
await tab.setValue(3, "Eru");
await tab.click(8);
return await tab.getAXState();
```

Three browsers, and only the last needs anything installed:

- `"empryo"` — the browser inside Empryo. Default, nothing signed in.
- `"chrome"` — real Chrome on a profile Empryo manages, separate from yours.
- `"extension"` — your own signed-in profile, once you install the Empryo
  extension. Listed as unavailable until it answers, never assumed.

Installing that extension: `bun run ext:pack` builds the Web Store zip, and
`chrome://extensions` → Developer mode → **Load unpacked** on
`packages/tools/native/chrome-extension` installs it now. While it drives, the
toolbar badge lights, the page carries an "Empryo is driving this tab" strip, and
Chrome's own bar reads "Empryo started debugging this browser".
`Cmd/Ctrl+Shift+X` stops everything without asking the app.

Actions **queue** and run as one batch the moment the code reads state. That is
not an optimisation: element indices belong to the observation the model read, so
a sequence that shares one snapshot is the only sequence that can be validated.
The policy broker pre-flights the whole batch, which means one approval for a
group of steps rather than one per click.

This is the shape OpenAI ships in Codex as `cua_repl`, running on our own
runtime: no ChatGPT desktop app, no bundled plugin, no extension. The same tool
is published by `empryo mcp computer-use`, so Claude Desktop or Codex can drive
your machine through Empryo's coordinator, leases and policy.

## Who is driving

Control is legible in under a second, and the cue's scope says what is reachable:

- **Panel ring** — the agent is inside one Empryo browser tab.
- **Window cue** — it is driving your Chrome; it survives our window losing focus.
- **Screen cue** — it is driving the machine.

The **mote cursor** — Empryo's organism, glowing — moves to wherever the last action
landed, in the panel for embedded tabs and as a capture-excluded overlay window for
everything else. Turn it off with `computerUseSettings.cursor: false`.

Taking over is a first-class move, not an interruption: touch the mouse or keyboard
and the session pauses, the lease's epoch is fenced, and the agent's next action is
refused with *"control was taken over."* `/computer stop` and **Stop all** release
every lease and drop held input immediately.

## Policy

Capability switches are independent of what the model claims it needs. Each is
`true` (allow), `false` (deny) or `"ask"` (a human decides; headless denies with a
reason).

| Switch | Default | Gates |
|--------|---------|-------|
| `nativeInput` | `true` | Clicks and keys against the desktop |
| `uploads` | `ask` | File inputs and file choosers |
| `downloads` | `ask` | Accepting a download into the profile's directory |
| `clipboardRead` | `ask` | Reading your clipboard |
| `credentialEntry` | `false` | Typing into password and credential fields |
| `arbitraryExecution` | `ask` | `eval`, a REPL, raw CDP |
| `consequential` | `ask` | Send, pay, delete, publish, grant |

A batch is pre-flighted as a whole: the approval names every step before any of
them runs, and answering it can be remembered for the session or for good, per
app/origin × effect. `computerUseApps` decides which applications may be driven at
all — a default plus per-platform allow/deny.

### The word list, and what sits behind it

Consequential detection starts as 25 literal words (`send`, `pay`, `delete`, `publish`, `grant`…). That is a floor, not a model of the space: on the recorded corpus it catches **6 of 23** controls a user would want to approve, and misses *Empty Trash*, *Deploy*, *Revoke access*, *Terminate instance*, *Drop table*.

With [TypeSafe](/docs/tools/typesafe) enabled and `typesafe.computerUseGuard` on, a classifier answers the same question for the controls the word list did not recognise — one batched call before the batch runs, ~300 ms, fractions of a cent. Two properties make it safe to leave on:

- **Union only.** The word list's verdict always stands. The classifier can add an approval, never remove one, so its failure mode is friction and never a missing guard.
- **Shadow by default.** `computerUseSettings.consequenceMode` starts at `"shadow"`: the verdict is recorded, nothing is gated, and the numbers accrue on your real controls before anyone pays an approval for them. `"apply"` turns it into a gate.

Measured end to end through the broker (`fixtures/consequence-ab.json`): word list alone 6/23 with 0 false gates, word list ∪ classifier 23/23 with 0 new false gates. Read the caveat in that file before quoting the number — 20 of the 23 positives were written by hand, so it is a demonstration of coverage, not an estimate of production recall. A second, deliberately hard corpus (`consequence-hard-negatives.json`) is where precision is actually tested: there the word list scores 11 true / 11 false, the classifier 14 true / 6 false.

Secure fields never reach it: a password, passphrase, OTP or API-key control is dropped before the question is built.

Two guards are not switches because they should never be a decision:

- **Secure fields.** With `secureFieldGuard` on (default), a screenshot is never
  even captured while a password field has focus, and typed secrets never reach the
  trace.
- **Redaction.** The trace keeps actions, costs and outcomes; typed text and AX
  text only with `verboseTrace`.

## The pilot — jev drives, the model watches

A classifier cannot write prose, so the received wisdom is that driving a UI
needs a frontier model. Most steps do not need prose. They need one choice from
a short list, and our observation protocol already produces both lists: the
elements on screen and the actions each one supports. Turn them into two
questions — *which kind of move* and *which element* — and the step is a
classification.

With `typesafe.pilot` on, that is exactly what happens: no screenshot, no plan,
~360 ms and a fraction of a cent per step. Recorded run
(`fixtures/pilot-ab.json`), a sign-up form in real Chrome, goal *"name is Eru,
Email is eru@empryo.dev, agree to the terms, then create the account"*:

| step | move | element | confidence |
|------|------|---------|-----------|
| 1 | `set_value` | textbox "Name" | 0.96 |
| 2 | `set_value` | textbox "Email" | 0.89 |
| 3 | `click_item` | checkbox "I agree to the terms" | 0.98 |
| 4 | `click_item` | button "Create account" | 0.97 |
| 5 | `done` | — | 0.71 |

The page's own output read `created:Eru:eru@empryo.dev:true`. Five decisions,
6 545 input tokens, **$0.000275**, 1.8 s wall. "Cancel" sat beside "Create
account" the whole time and was never chosen.

What it cannot do is invent a value: text comes out of the goal literally, and
a field that needs composing ends the run and hands the step back. That limit
is load-bearing — the first run of this task stalled because the extractor
truncated the email at the dot, and the classifier was *right* to keep trying to
fix the field.

Three rules keep it honest:

- **It only chooses from what it was given.** An invented index is rejected
  before dispatch, never sent.
- **It stops rather than guesses.** Below the confidence floor, on a repeated
  no-op, or on "nothing here helps" — the run ends and reports.
- **A cheap planner does not get cheap permissions.** Every step goes through
  the coordinator: lease, policy broker, consequential-action guard, trace.

## Profiles

The browser the agent drives has a profile, and the default is the safest one.

| Profile | What it is |
|---------|------------|
| `ephemeral` *(default)* | A fresh session, wiped on close. No ambient logins to spend or corrupt. |
| `empryo:<slug>` | A persistent Empryo partition — logins survive between runs. |
| `chrome:<slug>` | Your own Chrome/Chromium binary, launched by Empryo with a **managed** user-data-dir. Sign in once inside it. |
| `chrome:<slug>` with `attach: true` | The browser you **already have open** — your real profile, logins and extensions. |

Chrome refuses to start a second process against a live profile and copying one corrupts it, so the real profile is reached by **attaching**, never launching: quit your browser, start it once with `--remote-debugging-port=9222`, and Empryo connects to that port. Everything the agent does then happens in your own session, which is why the origin policy and the consequential-action guard still apply there.

Your real, signed-in default Chrome profile is never copied or launched directly.
Set them up in **Settings → Computer use → Browser profile**.

## Concurrency

Several agent tabs can drive several targets at once. A lease belongs to an
**actor** (one worker invocation), carries a fencing epoch, and is checked again at
dispatch — not just at acquisition. Native input is a single global lease: only one
actor types on the machine at a time. A ninth browser tab queues rather than
evicting someone's work.

## Config

| Key | Default | Effect |
|-----|---------|--------|
| `computerUse` | `false` | The whole feature, including the provider's own computer tool where the lane has one |
| `computerUseSettings.switches` | see above | Per-capability allow / ask / deny |
| `computerUseSettings.cursor` | `true` | The mote cursor overlay |
| `computerUseSettings.secureFieldGuard` | `true` | No screenshots while a password field has focus |
| `computerUseSettings.verboseTrace` | `false` | Keep typed text and AX text in the trace |
| `computerUseSettings.consequenceMode` | `shadow` | What a registered consequence classifier may do: record, or gate |
| `typesafe.computerUseGuard` | `false` | Ask jev about controls the word list does not know |
| `typesafe.pilot` | `false` | Let jev choose the next UI action instead of the model |
| `typesafe.pilotMinConfidence` | `0.55` | Below this the pilot declines and the model keeps the wheel |
| `computerUseApps` | — | Which apps may be driven, per platform |
| `browser.profile` | `ephemeral` | The active browser profile |
| `browser.profiles` | `{}` | Configured Empryo and Chrome profiles |

## Commands

| Command | What it does |
|---------|--------------|
| `/computer` | Targets, permissions and live sessions |
| `/computer stop` | Stop everything and release input |
| `/computer teach` | Record a workflow by demonstrating it (`/computer teach stop` saves it) |
| `/computer workflows` | List taught workflows |

In headless runs, `--replay <path>` writes a printable HTML timeline of everything
the agent did: statuses, elapsed time, redacted actions, costs and screenshots.

## Next

#### The browser

The other half: snapshot-and-click against your dev server, audits and throttling.

#### Provider tools

Anthropic's and OpenAI's own computer tools, and what they resolve to on your model.