Computer use

OverviewLink to this section

Computer use lets the agent operate an interface instead of a file. It can work in a tab in Empryo's browser, a tab in your own Chrome, another app's window, or the whole screen. Use it for tasks that only exist in a UI: filling a form in an admin panel, checking a settings screen, walking through a sign-up flow.

It's on by default and needs no extra account or app. Your operating system still has to grant screen recording and input access the first time. To turn it off, set "computerUse": false, or use Settings → Computer use on desktop or /computer in the terminal UI.

If something is missing, /computer setup in the terminal UI lists what's still needed and the command that fixes each item.

How the agent sees the screenLink to this section

Screenshots are expensive and still leave the model guessing where to click. So by default the agent reads the screen's accessibility tree, the same structured list of controls that screen readers use:

[1] heading   "Form"                    (enabled)  actions: click  level="1"
[3] textbox   "Name"   value="Eru"      (focusable, enabled, editable)  actions: set_value, focus
[4] checkbox  "I agree"                 (focusable, enabled, checked)   actions: toggle, focus
[7] button    "Send"                    (focusable, enabled)  actions: click, focus

A few rules keep this cheap and reliable:

• Only changes come back. After the first look, the agent sees only what changed (~ changed, + added, - removed).
• Numbers expire after an action. Once the agent changes something, the old element numbers are no longer accepted. It has to look again, so it never clicks the wrong thing on a page that moved.
• Several steps, one look. The agent can plan a sequence of actions, run them together and read the result once.
• Screenshots only when needed, for visual checks or when an app exposes no accessibility tree.

The agent acts by meaning first: set this field's value, toggle this checkbox, press this button. It only falls back to clicking at screen coordinates when there's no better option, because a named button survives a layout change and a coordinate doesn't.

The toolsLink to this section

On Cua Driver windows, computer_act and computer_js also take menu (run a menu path), verify (check up to eight conditions) and frame (move and resize the window).

Scripting with computer_jsLink to this section

computer_js is the fast path and is on by default. The model writes a few lines of JavaScript against a cua object:

let tab = await cua.createBrowserTab("empryo", "http://localhost:5173");
await tab.setValue(3, "Eru");
await tab.click(8);
return await tab.getAXState();

Actions queue up and run together when the script reads the screen again. Because the whole group is known in advance, Empryo can check it against your permissions once and ask you a single question instead of one per click.

There are three browsers to choose from. Only the last needs setup:

• "empryo": the browser built into Empryo. The default, with nothing signed in.
• "chrome": real Chrome on a separate profile that Empryo manages, not your own.
• "extension": your own signed-in Chrome, once you install the Empryo extension.

Connecting your own ChromeLink to this section

To let the agent work in your everyday Chrome, with your logins, you need two things:

1. The bridge, which lets Chrome talk to Empryo. On desktop: Settings → Computer use → Connect Chrome → Register bridge. In the terminal UI: /computer extension install.
2. The extension. Until it's in the Chrome Web Store, you load it yourself: open chrome://extensions, turn on Developer mode, click Load unpacked, and pick the folder Empryo shows you.

/computer extension tells you whether the extension is installed, registered and connected.

While the agent is driving your Chrome, the extension's toolbar badge lights up, the tab title is marked, and Chrome shows its own "Empryo started debugging this browser" bar. Press Ctrl+Shift+X (Cmd+Shift+X on Mac) in Chrome to stop immediately.

Using it from other appsLink to this section

empryo mcp computer-use publishes the same computer-use tools as an MCP server. Claude Desktop, Codex or any other MCP client can then drive your machine through Empryo, with Empryo's permissions and stop controls still in place.

Who decides each clickLink to this section

A full language model doesn't need to choose every click. Many steps are just "pick one of these controls", and a small, fast classifier can do that.

That classifier is jev, the model behind Empryo's TypeSafe feature. With a TypeSafe key, you choose who drives in Settings → Computer use, or with computerUseDrive in config:

Without a TypeSafe key, the model drives, and the settings screen says so.

jev has firm limits:

• It only picks from what's on screen. It can't invent a control, and a made-up choice is rejected before anything happens.
• It stops instead of guessing. If it's not confident enough, repeats an action that did nothing, or sees nothing useful, it hands control back to the model. typesafe.pilotMinConfidence sets that threshold (default 0.55).
• It can't write text. Values must appear literally in the goal, such as "email is eru@empryo.dev". A field that needs composed text goes back to the model.
• Same permissions as the model. Every step jev takes goes through the same checks, approvals and stop controls.

Seeing and stopping the agentLink to this section

You can always tell what the agent is controlling:

• A ring around the browser panel means it's working inside one Empryo browser tab.
• A window cue means it's driving your Chrome. The cue stays visible even when Empryo isn't the front window.
• A screen cue means it's driving the whole machine.

A small animated cursor, the mote, moves to wherever the last action happened. It pulses on a click and shakes when an action fails. Turn it off with computerUseSettings.cursor: false.

Moving your own mouse or typing does not interrupt the agent, so you can keep working next to it. To stop it, use any of these:

• /computer stop in the terminal UI
• Stop all computer use on desktop
• The stop button on a tab the agent is driving
• Ctrl+Shift+X / Cmd+Shift+X in Chrome, when it's driving through the extension

Stopping blocks the agent's next action with "the user stopped this task" and releases any key or mouse button it was holding.

PermissionsLink to this section

Each capability has its own switch, whatever the model says it needs. A switch is true (allow), false (deny) or "ask" (you decide each time; headless runs deny).

Set them under computerUseSettings.switches. When Empryo asks, the question lists every step in the batch before any of them runs. You can remember your answer for the session or permanently, per app or site and per kind of action.

computerUseApps controls which applications may be driven at all: a default of allow or deny, plus per-platform exceptions (macOS bundle id, Windows process name, Linux window class).

Spotting risky actionsLink to this section

Empryo recognizes risky buttons by a fixed list of words such as *send*, *pay*, *delete*, *publish* and *grant*. That list misses plenty, for example *Empty Trash*, *Deploy* or *Drop table*.

With TypeSafe set up, turn on typesafe.computerUseGuard and jev also checks the controls the word list didn't recognize. It can only add an approval prompt, never remove one, so if it's wrong you get an extra question, not a missing safeguard.

It starts in shadow mode (computerUseSettings.consequenceMode: "shadow"): it records what it would have flagged but doesn't prompt. Set "apply" to let it ask for approval.

Password, one-time-code and API-key fields are never sent to it.

Always-on protectionsLink to this section

Two protections aren't switches:

• Secure fields. With secureFieldGuard on (the default), no screenshot is taken while a password field has focus, and typed secrets never reach the log.
• Log redaction. The action log keeps actions, costs and outcomes. Typed text and screen text are kept only if you turn on verboseTrace.

Browser profilesLink to this section

The browser the agent uses has a profile, and the default is the safest one.

Chrome won't run two copies on the same profile, and copying a profile corrupts it. So to use your real profile, Empryo attaches to it instead: quit Chrome, start it once with --remote-debugging-port=9222, and Empryo connects to that port. Everything the agent does then happens in your own session, and the site and risky-action checks still apply.

Empryo never copies or launches your default Chrome profile directly. Set up profiles in Settings → Computer use → Browser profile.

Cua DriverLink to this section

Empryo drives other apps through Cua Driver, trycua's open-source driver, whenever it is installed and ready. Until then, Empryo's built-in helper does the job. Cua Driver is the recommended setup:

• It clicks and types in the background. Your cursor stays put and the window stays where it is.
• Every action reports what happened: confirmed, can't tell, or probably did nothing.
• It runs app menu commands such as File → New through accessibility, so no keyboard shortcut and no focus change.
• It checks conditions on a window (a button exists, a field holds a value, the window has a given frame) over several samples and answers holds, doesn't hold or can't tell. It never guesses yes.
• It moves and resizes windows and reads the result back.
• It launches apps without bringing them to the front, and lists minimized and hidden windows too.
• On macOS the permissions belong to CuaDriver.app, so they survive Empryo updates.

Its windows show up in computer_targets as window:cua:<pid>:<window>. The same permissions, approvals, stop controls and logging apply.

Set it up from EmpryoLink to this section

Each step only shows up when it applies: update appears when Cua Driver reports a newer release, allow when a macOS permission is missing.

• Install runs trycua's official installer for your user account, then starts the driver. It is the same command you could run yourself:

``sh /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" # macOS, Linux irm https://cua.ai/driver/install.ps1 | iex # Windows PowerShell ``

• macOS. Empryo starts the driver through CuaDriver.app, so macOS asks about Accessibility and Screen Recording for CuaDriver, not for your terminal. Allow opens those dialogs.
• Windows. Start registers a logon task and starts it now, so the driver runs in your signed-in desktop session, also when Empryo itself runs over SSH.
• Linux. The driver needs your graphical session. From a desktop terminal Empryo starts it. From a shell with no display, Empryo installs it and tells you to run cua-driver serve in your session. If the driver needs system libraries your distribution lacks (a minimal Debian needs libX11, libXi and libxkbcommon), Empryo names them and offers the apt-get or dnf command; it runs it itself only as root.
• A broken install. If cua-driver is there but cannot start, Empryo reports it as broken and offers to reinstall instead of failing.
• Update installs the newest release and restarts the driver.
• Live preview turns on Cua Driver's picture-in-picture window, which shows the app it is driving in the background. The driver restarts to apply it; run it again to turn it off.

If you have Codex or the ChatGPT desktop app installed, their computer-use plugins also appear under Settings → Computer use → Engines as fallbacks. Empryo doesn't install or ship them, and they stay off unless you turn them on there. /computer lanes prints the same list in the terminal UI.

Several agents at onceLink to this section

Several agent tabs can each drive their own target at the same time. Only one agent at a time can type or click outside the browser, since there's only one keyboard and mouse.

ConfigLink to this section

CommandsLink to this section

In headless runs, --replay <path> writes a printable HTML timeline of everything the agent did: status, timing, redacted actions, costs and screenshots.

NextLink to this section