How computer use differs from an agent with access to files and the terminal, why controlling a mouse needs the cloud and screenshots, and which option already works locally today.

"AI that controls the computer" usually conjures up one image: a model moving the mouse itself, clicking buttons, and doing your routine work inside programs for you.

Systems like that exist. But two very different approaches hide behind that one phrase, and they differ both in reliability and in what a remote server gets to see about you. The difference matters more than it looks at first glance.

Approach one: look at the screen

This is called computer use. The model gets a screenshot, identifies the interface elements in it, and outputs actions: click these coordinates, type this text, scroll. Then the loop repeats — a new screenshot, a new action.

This is how the agentic modes of cloud assistants work, along with open projects like ByteDance's UI-TARS or Microsoft's OmniParser, which breaks a screenshot down into elements for a model.

The approach has a real strength: it works with any program, even one with no API at all. The model sees what a person sees.

There are more weaknesses, and the first is privacy. A screenshot is everything on the screen: an open inbox, a messenger, account details, someone else's personal data. With a cloud solution, every one of those frames goes to the provider's server, and that happens dozens of times in a single task. Local options like UI-TARS Desktop can run their own model, in which case the frames stay with you, but the hardware requirements are higher than usual: it's a multimodal model that also has to parse images.

The second problem is fragility. A click on coordinates breaks from just about anything: a different resolution, a button that shifted, a notification popping up on top. What's frustrating isn't that the action fails — it's that it fires somewhere else, and the model has no way of knowing.

The third is boring but decisive in practice: every step needs a new screenshot, parsing it, and a decision. A task made of twenty actions takes minutes instead of seconds.

Approach two: work through tools

Here the model doesn't look at the screen — it gets direct access to what's behind it: the filesystem, the terminal, applications' own interfaces.

Instead of "find the save button and click it" — "write the file." Instead of "open email and read the latest message" — reaching email through an MCP server. Instead of "click the run-tests button" — running a command in the terminal.

The difference turns out to be fundamental. An action either succeeds or returns a visible error. There's no missing a button, no ambiguity. It runs an order of magnitude faster, and only what's needed for the task ends up in the model's context — not your whole screen.

The cost is that a program has to be reachable through files, commands, or an API. An ancient piece of accounting software with no interface can't be automated this way — through screenshots, theoretically, it could be.

◆

A useful check before picking an approach: can your task be described as "read the files, do the thing, write the result"? If yes, the second path is more reliable. If the task sounds like "click in this window," sometimes there's no way around it.

Which of these actually works locally

Both approaches work, but differently.

Computer use locally needs a multimodal model that understands images, and a decent GPU. Open projects in this space have come a long way, but it's still experimental territory: errors are frequent, setup isn't trivial.

Working through tools locally is already an ordinary scenario, because it doesn't need to understand pictures. The model works with text: file paths, content, command output. Local models handle this fine, as long as there's enough context and the model's size fits the task.

What Doka does

Let's be direct, so as not to set the wrong expectations: Doka doesn't control the mouse or click windows. It works the second way — with files, the terminal, and connected tools.

In practice, this covers most of what people are actually looking for when they search "AI that controls the computer": go through a folder and rename files based on their content, pull together a summary from exports, find and fix a failing test, prepare a document from a template, do it on a schedule with no input from you. The model can be local, in which case neither the files nor screen content go anywhere.

What Doka won't do: click a button in a program with no API, get past a captcha, or operate an interface that only exists as a window.

What not to trust an agent with

Regardless of the approach. A program allowed to take actions on your machine can do the wrong thing — because a phrasing was misread, or the model simply made a mistake.

A few rules worth setting up right away:

start with reversible tasks: copies, drafts, a separate folder instead of your working one;
keep human confirmation on anything that deletes or overwrites;
don't give access to secrets and keys unless the task actually needs them;
check the result on a small sample before running it on the whole folder.

More on limiting access is in the article on connecting an MCP server.

The short version

Controlling the mouse looks flashier and works worse: slower, less reliable, and it sends screen content outward if the model is cloud-based. Access to files and commands looks more boring and solves the same tasks predictably.

If you want to see how this feels on your own tasks, download Doka and start with something reversible — ask it to go through a copy of a documents folder, for instance.

AI That Controls Your Computer: How It Actually Works