Computer use

Overview

Computer use is the agent operating a real interface instead of a file: a page in Empryo's browser, a tab in your own Chrome, an app window, the whole display. One contract covers all of them — a SurfaceDriver per backend, one coordinator in front, the same tools on every surface.

It is off by default. Turn it on with computerUse: true, or from Settings → Computer use on the desktop and /computer in the TUI.

Read the tree, not the pixels

A screenshot costs ~1 500 tokens and still leaves the model guessing coordinates. computer_observe returns the accessibility tree as indexed text instead:

[1] heading   "Form"                    (enabled)  actions: click  level="1"
[3] textbox   "Name"   value="Eru"      (focusable, enabled, editable)  actions: set_value, focus
[4] checkbox  "I agree"                 (focusable, enabled, checked)   actions: toggle, focus
[7] button    "Send"                    (focusable, enabled)  actions: click, focus

Five rules make it cheap, and they are enforced, not suggested:

• Diff by default. A second observation returns only what changed (~ changed,

+ added, - removed). No change is an answer: *"act before observing again."*

• Indices die on write. Every batch that mutates the target invalidates them;

using a stale index is refused with the latest snapshot id, never silently retried.

• Batch, then observe once. computer_batch runs a planned sequence under one

lease and returns a single diffed observation.

• Screenshots are opt-in (mode:"screenshot" / "both"), for visual tasks or

when the tree came back empty.

• Per-app priors. The first observation of a target carries a short operating

note for that kind of app (Chrome's omnibox, Finder's list, a terminal's textarea) — once per incarnation, not per turn.

The tools

Actions are semantic first — set_value, toggle, focus, a named perform action off the element — with coordinates as the fallback, because a role-and-name click survives a layout change and a coordinate does not.

One tool, written as code

computer_js is the fast lane, and it is on by default. The model writes JavaScript against a persistent cua session; binding a target prints that target's accessibility state into the same tool result, so binding and reading are one round trip instead of two.

let tab = await cua.createBrowserTab("empryo", "http://localhost:5173");
await tab.setValue(3, "Eru");
await tab.click(8);
return await tab.getAXState();

Three browsers, and only the last needs anything installed:

• "empryo" — the browser inside Empryo. Default, nothing signed in.
• "chrome" — real Chrome on a profile Empryo manages, separate from yours.
• "extension" — your own signed-in profile, once you install the Empryo

extension. Listed as unavailable until it answers, never assumed.

Installing that extension: bun run ext:pack builds the Web Store zip, and chrome://extensions → Developer mode → Load unpacked on packages/tools/native/chrome-extension installs it now. While it drives, the toolbar badge lights, the page carries an "Empryo is driving this tab" strip, and Chrome's own bar reads "Empryo started debugging this browser". Cmd/Ctrl+Shift+X stops everything without asking the app.

Actions queue and run as one batch the moment the code reads state. That is not an optimisation: element indices belong to the observation the model read, so a sequence that shares one snapshot is the only sequence that can be validated. The policy broker pre-flights the whole batch, which means one approval for a group of steps rather than one per click.

This is the shape OpenAI ships in Codex as cua_repl, running on our own runtime: no ChatGPT desktop app, no bundled plugin, no extension. The same tool is published by empryo mcp computer-use, so Claude Desktop or Codex can drive your machine through Empryo's coordinator, leases and policy.

Who is driving

Control is legible in under a second, and the cue's scope says what is reachable:

• Panel ring — the agent is inside one Empryo browser tab.
• Window cue — it is driving your Chrome; it survives our window losing focus.
• Screen cue — it is driving the machine.

The mote cursor — Empryo's organism, glowing — moves to wherever the last action landed, in the panel for embedded tabs and as a capture-excluded overlay window for everything else. Turn it off with computerUseSettings.cursor: false.

Taking over is a first-class move, not an interruption: touch the mouse or keyboard and the session pauses, the lease's epoch is fenced, and the agent's next action is refused with *"control was taken over."* /computer stop and Stop all release every lease and drop held input immediately.

Policy

Capability switches are independent of what the model claims it needs. Each is true (allow), false (deny) or "ask" (a human decides; headless denies with a reason).

A batch is pre-flighted as a whole: the approval names every step before any of them runs, and answering it can be remembered for the session or for good, per app/origin × effect. computerUseApps decides which applications may be driven at all — a default plus per-platform allow/deny.

The word list, and what sits behind it

Consequential detection starts as 25 literal words (send, pay, delete, publish, grant…). That is a floor, not a model of the space: on the recorded corpus it catches 6 of 23 controls a user would want to approve, and misses *Empty Trash*, *Deploy*, *Revoke access*, *Terminate instance*, *Drop table*.

With TypeSafe enabled and typesafe.computerUseGuard on, a classifier answers the same question for the controls the word list did not recognise — one batched call before the batch runs, ~300 ms, fractions of a cent. Two properties make it safe to leave on:

• Union only. The word list's verdict always stands. The classifier can add an approval, never remove one, so its failure mode is friction and never a missing guard.
• Shadow by default. computerUseSettings.consequenceMode starts at "shadow": the verdict is recorded, nothing is gated, and the numbers accrue on your real controls before anyone pays an approval for them. "apply" turns it into a gate.

Measured end to end through the broker (fixtures/consequence-ab.json): word list alone 6/23 with 0 false gates, word list ∪ classifier 23/23 with 0 new false gates. Read the caveat in that file before quoting the number — 20 of the 23 positives were written by hand, so it is a demonstration of coverage, not an estimate of production recall. A second, deliberately hard corpus (consequence-hard-negatives.json) is where precision is actually tested: there the word list scores 11 true / 11 false, the classifier 14 true / 6 false.

Secure fields never reach it: a password, passphrase, OTP or API-key control is dropped before the question is built.

Two guards are not switches because they should never be a decision:

• Secure fields. With secureFieldGuard on (default), a screenshot is never

even captured while a password field has focus, and typed secrets never reach the trace.

• Redaction. The trace keeps actions, costs and outcomes; typed text and AX

text only with verboseTrace.

The pilot — jev drives, the model watches

A classifier cannot write prose, so the received wisdom is that driving a UI needs a frontier model. Most steps do not need prose. They need one choice from a short list, and our observation protocol already produces both lists: the elements on screen and the actions each one supports. Turn them into two questions — *which kind of move* and *which element* — and the step is a classification.

With typesafe.pilot on, that is exactly what happens: no screenshot, no plan, ~360 ms and a fraction of a cent per step. Recorded run (fixtures/pilot-ab.json), a sign-up form in real Chrome, goal *"name is Eru, Email is eru@empryo.dev, agree to the terms, then create the account"*:

The page's own output read created:Eru:eru@empryo.dev:true. Five decisions, 6 545 input tokens, $0.000275, 1.8 s wall. "Cancel" sat beside "Create account" the whole time and was never chosen.

What it cannot do is invent a value: text comes out of the goal literally, and a field that needs composing ends the run and hands the step back. That limit is load-bearing — the first run of this task stalled because the extractor truncated the email at the dot, and the classifier was *right* to keep trying to fix the field.

Three rules keep it honest:

• It only chooses from what it was given. An invented index is rejected

before dispatch, never sent.

• It stops rather than guesses. Below the confidence floor, on a repeated

no-op, or on "nothing here helps" — the run ends and reports.

• A cheap planner does not get cheap permissions. Every step goes through

the coordinator: lease, policy broker, consequential-action guard, trace.

Profiles

The browser the agent drives has a profile, and the default is the safest one.

Chrome refuses to start a second process against a live profile and copying one corrupts it, so the real profile is reached by attaching, never launching: quit your browser, start it once with --remote-debugging-port=9222, and Empryo connects to that port. Everything the agent does then happens in your own session, which is why the origin policy and the consequential-action guard still apply there.

Your real, signed-in default Chrome profile is never copied or launched directly. Set them up in Settings → Computer use → Browser profile.

Concurrency

Several agent tabs can drive several targets at once. A lease belongs to an actor (one worker invocation), carries a fencing epoch, and is checked again at dispatch — not just at acquisition. Native input is a single global lease: only one actor types on the machine at a time. A ninth browser tab queues rather than evicting someone's work.

Config

Commands

In headless runs, --replay <path> writes a printable HTML timeline of everything the agent did: statuses, elapsed time, redacted actions, costs and screenshots.

Next