Computer-use AI

AI that can operate computers like humans: reading what is on the screen and acting through the same interface a person uses.

Explanatory overview of the technology category. It describes no specific Samtal product feature, customer or result.

What it is

A computer-use agent receives a goal, observes the screen (typically as screenshots, sometimes together with accessibility data), decides on the next action, and carries it out by moving the pointer, clicking, typing or scrolling. It repeats this loop until the task is complete or it needs a person to step in.

Why it matters

Many business systems have no API, or only a partial one. Older desktop applications, internal tools and third-party portals are often reachable only through their user interface. An agent that works through the interface can automate these systems without integration projects.

How it typically works

1. Perceive: capture the current screen state.
2. Reason: a model interprets the screen against the goal and plans the next step.
3. Act: issue a mouse or keyboard action.
4. Verify: check the result and correct course if something unexpected appears.

Open challenges

• Reliability over long, multi-step tasks.
• Speed compared with direct API calls.
• Security: agents act with a user's permissions, so access control and human approval for sensitive steps matter.
• Auditability: recording what the agent saw and did.

See also RPA vs AI agents and the glossary.