StackMap
Subscribe
Explore / typesafe-computer-use
awlevin

typesafe-computer-use

macOS computer-use loop at ~$0.0002/step: OCR + accessibility tree read the screen, a TypeSafe classifier picks the next action, code clicks; an LLM writes only free text.

1,288 123 Python MITupdated 1 months ago
agentsdecision-models
View on GitHubDispute this mapping →
Curator's take

Use it when per-step cost and latency matter more than raw capability: cheap, gated, replayable desktop automation with a confidence number per action. Skip it when tasks need real visual reasoning — anything the frontier model infers from pixels must be rebuilt as deterministic state — and remember it drives your real mouse, needs a paid TypeSafe key, and Windows is experimental.

Mapped by ShipWithAI editors · links verified

Continue your stack

What teams reach for next — and why each earns a place beside typesafe-computer-use. Ranked by curator confidence.

pairs wellpairs wellpairs wellalternativealternativealternativepg-jevOllamae2ejev-ultrafastbrowser-usecuatypesafe-computer-use
pairs wellalternativebuilt withpick a node for the why · open it from the panel
Weekly digest
README.md2 min read

typesafe-computer-use

CI MIT license Python 3.12+ macOS, Windows experimental TypeSafe Ruff

typesafe-computer-use (jev for short) drives a Mac toward a goal you type in plain English, for about a fiftieth of a cent per step. It reads the screen deterministically, asks a small classifier which action comes next, and only calls a writing model when a text field genuinely needs free text or the classifier has stopped and the screen needs reading.

One idea runs through it: the classifier picks, code decides facts, and the writer only writes free text. Anything a model would have to work out (a date, whether a field is focused, whether a URL is clean) is computed in code and handed over as state. VISION.md says what it is for in a dozen lines.

clicker "go to techcrunch and take me to the checkout page for the cheapest tickets to their next upcoming event" --act

Beta. This is under heavy development. Expect rough edges, and expect settings and behavior to change between 0.x releases. It drives your real mouse and keyboard, so start with a dry run.

Why a classifier

Frontier-model computer use is capable and expensive: every step ships a screenshot and waits several seconds for a plan. Most steps do not need a plan. They need one choice from a short list, made quickly and cheaply, with a confidence number you can gate on.

TypeSafe sells exactly that: a decision model that answers a Choice over up to 255 options with a full probability distribution and a calibrated confidence, in a few hundred milliseconds, with free output tokens. This project is a computer-use loop built around it.

Measured on the same screenshot and goal, one decision each:

typesafe (jev) Claude Opus 5, bare screenshot multiplier
input tokens 4,882 4,785 same
cost per decision $0.0002 $0.032 155x cheaper
cost per decision, realistic loop with history $0.0002 $0.035 to $0.08 170x to 390x cheaper
cost per 12-step task $0.003 $0.40 to $0.90 130x to 300x cheaper
model latency 0.13 to 0.38 s 5.2 s 14x to 40x faster
end-to-end step, with capture and OCR about 1.5 s about 5.5 s 3.7x faster

The honest caveat: the big model read the event dates off the pixels and compared them unaided. The classifier needed the date parsing described in how a step works. Every piece of reasoning the frontier model does for free has to be rebuilt here as deterministic state.

Install

macOS 14 or newer, Python 3.12 or newer, uv. Windows 10 and 11 are experimental; see Windows.

git clone https://github.com/awlevin/typesafe-computer-use
cd typesafe-computer-use
uv sync
cp .env.example .env     # fill in the keys
variable required purpose
TYPESAFE_API_KEY yes every decision
ANTHROPIC_API_KEY no type_text, writer-proposed URLs, and the final answer
CLICKER_EMAIL no