An agent that learns an app once

Every agent that drives real software starts from zero every time. This piece is about what happens after the hundredth look at the same app — what the agent keeps, what it refuses to keep, and what it costs to use.

Ivan Podtsebnev · naykip · a builder's log · 24 August 2026

Every agent that drives real software starts from zero every time. Open the same messenger tomorrow and it will look for the search box again, read the same thousand nodes again, ask the same model the same question — “where do I type?” — and pay for the answer again. The work of figuring the app out is done on every run and kept nowhere.

People do not work like that. The first time you use a new chat app you look around; the tenth time your hands know it. kovavue is built so an agent's hands can know it too. This article is about the shape of that knowledge — without the internals, which are the product. It is the second piece about kovavue; the first — “Your agent is paying to look at the screen. Twice.” — was about the cost of one look.

No API. The running app itself.

One thing to say plainly before the diagrams, because it decides everything after them: kovavue does not integrate with applications. It operates them.

There is no Telegram API key, no Slack app, no LinkedIn partnership, no headless browser of its own. The agent's eye reads the application that is already running on your desktop — the same window you have open, signed in as you — and its hand acts on that window's own controls: the real search box, the real composer, the real Send button, with a real keyboard and pointer where the app insists on them. A web page and a native desktop app are the same kind of thing to it: a surface with controls.

This is not a workaround for missing APIs; it is the point. An API is a second door the vendor decided to cut, with its own rules, quotas and blind spots; the window is the door the vendor had to build, because a person has to use it. Whatever a person can do in the app, the agent can learn to do — and nothing more: it has exactly the reach your session has, under your name, and the gate sits on the way out. Skills, below, are knowledge about that window — never tokens, never credentials, never a back channel.

What a “skill” is, concretely

The word is overloaded, so let me say what I mean by it here. A skill is a learned way to get something done in one application, stored so that the next agent (or the same one tomorrow, after its memory is wiped) can replay it without looking around first.

Four things have to be true for that to work, and each one is its own layer:

• The live map is the truth about the window right now — every control with a role, a name, a state, and a handle the agent can act on — read from the running app, not from any API. It is cheap to read and it dies with the window; it is not memory.
• The stream is the map's changes over time: a field got a value, a dialog appeared, the page navigated. An agent watching the stream does not re-read the map to find out what its own action did.
• The app dictionary is the durable part: which controls exist in this app, what they are called, how to recognise them after a re-render. It is keyed by the application and its version — when the app ships a new layout, the dictionary learns the new shape rather than lying about the old one.
• Operation paths are the skills proper: “open a chat with name”, “post text in channel” — a short sequence of steps an agent worked out once and can replay later without reading the screen. A path can fail honestly; when it does, the agent falls back one layer and re-learns.

The fifth box sits under everything. The consent gate holds any action that leaves the machine — send, submit, apply, connect — until a person says yes. It is not a skill and it cannot be learned around: a replayed path that reaches a send stops exactly where a hand-driven one would.

The loop an agent actually runs

The skill layers are only useful if the agent reaches for them in the right order. The expensive failure — measured on myself more than once — is an agent that re-reads the whole map when the dictionary already knew the answer.

Recall first, look only if recall has nothing, act with one verb on one target, verify by what the surface now holds. The last box is the same on every path.

“Verify by state, not by ack” is the line that took longest to learn. A click can return OK and change nothing; a paste can return uncertain and have landed perfectly; a form can accept your submit and reject it thirty seconds later with a banner. The only honest answer is to read the thing you changed — which is why the map has a verb that reads a form the way a person does: the label they see over the box, what the box holds, whether the page requires it, and, for a Yes/No pair, which one is down.

What it costs, measured

Numbers from 23 August 2026, one machine (Ubuntu 24.04, X11/KDE), idle box, the agent driving through kovavue's MCP bridge. “Calls” are round-trips between the agent and the tool; wall time includes the agent's own thinking.

The first Telegram attempt that morning took 110 s and 9 calls. The difference was not the model and not the machine — it was two things the dictionary now knows and did not yet: that the search box is not the composer (so Enter there is navigation, not a send, and the gate should not hold it), and what the composer is called (so the agent does not hunt for it). That is what a skill is worth: the second run paying for the first run's lessons.

The other number worth printing is the one that went down by design. Closing a defect in kovavue requires a speed measurement on a fixed benchmark, and the number must not rise — a fix that costs speed is treated as a new defect, not a trade nobody noticed. Today's releases sit within noise of last week's baseline.

What the agent refuses to keep

Three things deliberately do not go into a skill:

• Live handles. A control's id is born with the window. The dictionary keeps what the control is; the handle is re-bound live every time. A skill that stored handles would work until the first re-render and lie after it.
• The person's data. What a form was filled with — a phone number, a salary expectation, a cover letter — is the human's, lives with the human, and is never part of the app's dictionary. Another agent that inherits the skill for an applicant-tracking system inherits how the form works, not what anyone wrote in it.
• Permission. No path remembers a yes. The gate asks every time, and it asks with the full text of what would be sent and where.

What a second agent gets

Because the dictionary and the paths are files keyed by application and version, they travel. Another agent — or the same model on another machine — that opens the same app starts from the tenth run, not the first: controls already named, paths already learned, the gate already there. The honest caveats are also facts: a path carries the preconditions it was learned under, and a good one says so (one of mine assumed a keyboard shortcut meant “search” in any state; in the composer it meant “insert link”; it now starts by leaving the composer). The shape is there; the registry, signing and sharing of skills across strangers is the part still ahead.

What's coming

kovavue will be available as an MCP server — an eye and a hand that any MCP-speaking agent can drive, with the dictionary, the paths and the gate included. The two things being finished before that: the map as a stream — a window's map arriving on its own from the moment the window appears, so the first look is free — and a cheaper native read for desktop apps, which today cost more than browser tabs. Both are measured against the same benchmark, and the numbers will be here when they are real.

As before: numbers were measured by me on my machine and are reproducible from the MCP bridge; there is no comparison with other browser agents because I will not publish numbers I did not take myself. The diagrams are the shape of the thing, not its internals.

— naykip —