Using ActionWright

Computer Use

How Computer Use sees and uses your desktop apps, and how to stop it at once.

What Computer Use does

Computer Use is the part of ActionWright that operates desktop apps by looking at the screen, the way you do. You don't start it yourself. When a task needs an app that ActionWright cannot reach in another way, it hands that part of the work to Computer Use, and the result comes back into your task. See How it works.

Use it for desktop apps that have no other way in. For example:

  • "Open Notepad and type a shopping list: milk, bread, eggs."
  • "In the Excel workbook that is open, make the header row bold and sort the table by date."
  • "Open Windows Settings and turn on dark mode."
  • "In Outlook, write a reply to the latest email from Dana, but don't send it."
  • "In the ordering program that is open on my screen, fill in the form with the details from this email."

When another agent is a better fit

Computer Use is ActionWright's last resort. It uses your real screen and sends screenshots to your provider, so ActionWright prefers the other agents when they can do the job:

  • Files and folders. Sorting, renaming, reading or writing files works better with the file agent. It stays inside the folder you choose and moves files to the Recycle Bin instead of deleting them. See Working with files.
  • Websites. Research, forms and online accounts work better with the browser agent. It uses its own browser, not your desktop. See Working in the browser.

To have a job done in a desktop app, name the app in your request, for example "In Excel, …".

Before you begin

  • Check the setting. Enable Computer Use agent (screen control) must be on in Settings. It is on by default. See Turn Computer Use on or off.
  • Use a task model that accepts images. Computer Use sends screenshots, so a model that can't read images fails as soon as it starts. See Models.
  • Close windows you don't want it to see. Screenshots of your screen go to your AI provider, and Computer Use can see the titles of all open windows. Minimizing a window is not enough, because Computer Use can bring it back. Close email, chats, documents and other private apps that the task doesn't need.
  • Save your work in other apps. Computer Use may switch between windows, and it can close them.
  • Plan not to use the PC while it works. Computer Use uses your mouse and keyboard. See Leave the mouse and keyboard alone.
  • Not working with desktop apps? If your tasks don't need desktop apps, you can turn Computer Use off in Settings. It is on by default and is not limited to the folder you choose. See Turn Computer Use on or off.

Turn Computer Use on or off

SettingDefaultEffect
Enable Computer Use agent (screen control)OnWhen off, ActionWright cannot control your screen. The file agent and the browser agent keep working.
  1. Open Settings from the ActionWright icon in the Windows tray. See The tray icon.
  2. Click Enable Computer Use agent (screen control) to turn it on or off.

The change is saved as soon as you click the box, even if you then close Settings with Cancel. It applies the next time a task needs Computer Use; you don't need to restart. If a task needs Computer Use while it is off, Computer Use doesn't run, and ActionWright is told that you can turn it on in Settings.

Run a Computer Use task

  1. Close the windows the task doesn't need.
  2. If you can, open the app and the file the task is about yourself. This gives Computer Use a clear starting point.
  3. In the task window, describe the goal: the app, what to do, and what it may commit. For example: "In Excel, open Budget.xlsx from my Documents folder, add a total row under the table, and save the file."
  4. Click Run Task, then leave the mouse and keyboard alone.
  5. When an approval dialog appears, read it, then choose. See What asks first.
  6. When it hands you a step, do it, then click Done. See What it hands back to you.
  7. Read the summary at the end, then check the result in the app.

You know it worked when the summary describes the result and you can see it in the app. For a short first try, see A simple Computer Use task.

How it works

  • It looks at your screen. Computer Use takes screenshots of the window it is working in, or of your whole screen across all your monitors. It can zoom in on part of a screenshot to read small text. It also sees the list of open windows and their titles.
  • Screenshots go to your AI provider. Computer Use decides what to do next by sending the screenshots to your AI provider. See What is sent to your AI provider.
  • It uses your real mouse and keyboard. It moves the pointer, clicks and types as if you were doing it, in your own apps, with your own accounts.
  • It looks again after acting. After an action, it checks whether the screen changed and plans the next step from what it sees.

There is no separate on-screen sign while it works. You see the pointer move, windows open and text appear. In the task window, the Execution Trace shows each of its steps with a label that starts with "Computer control:". See Follow progress.

What it can do

  • Open apps by name, switch between windows, and close windows. When it switches to a window or opens an app, it usually maximizes that window so it can see all of it.
  • Click, right-click and double-click, scroll, and drag.
  • Type text, and press keys and keyboard shortcuts.
  • Paste what is on your clipboard.
  • Wait for a slow app, an export or an installer, and then look again.

Computer Use is not limited to a folder or to certain apps. It can see and use whatever is on your screen. The root folder in the task window limits only the file agent.

Your clipboard

When Computer Use types long text, or text with characters beyond plain English letters, such as Hebrew or accented letters, it pastes the text through the Windows clipboard. What you had copied before is replaced, and the typed text stays on the clipboard after the task. Copy your own content again if you need it.

Leave the mouse and keyboard alone

While Computer Use works, don't use the mouse or keyboard. You share them with it:

  • If you move the mouse or type while it acts, your input can mix with its clicks and keystrokes and land in the wrong place.
  • If you switch to another window, Computer Use switches back to its own window before its next action.

While a task waits for you, for an approval, a question or a step you need to do, Computer Use is paused and you can use the PC normally.

Stop it at once

  • Press ctrl+esc in any app. The shortcut works even while Computer Use has the mouse and keyboard.
  • Or click Stop · Ctrl+Esc in the task window.

Computer Use stops as soon as it safely can. A click or keystroke that was already sent cannot be undone. The result shows "Task was stopped by the user." See Stop a task.

  • Moving the mouse to a corner of the screen does not stop it. Use ctrl+esc or Stop.
  • While a task waits for you, ctrl+esc does not end it. Click Stop in the task window.
  • You cannot change the ctrl+esc shortcut in Settings.
  • After a stop, check what the last action did before you continue. To pick up where it stopped, see Continue a stopped task.

What asks first

Ordinary work in an app runs without asking. For anything that commits something, such as sending, deleting, changing settings, installing software or paying, your request decides what this task may do. See How ActionWright decides what may run and the full table.

  • A button in an app's own confirmation dialog, such as one that deletes for good, is checked like any other action.
  • Every Computer Use approval is for that one action only. Always Allow is never offered for Computer Use. The next action asks again.
  • The dialog names the action and the app window it is for. Read both before you allow it.
  • An app can see what is typed into it before anything is sent. Don't ask Computer Use to type private information into apps you don't trust.

For the dialog and what each choice does, see Approvals.

What it hands back to you

Some steps stay with you. When Computer Use reaches one of these, it stops and asks you to do it:

  • Signing in, and any field that needs a password. It refuses to type into fields it recognizes as password fields.
  • Two-factor and one-time codes.
  • Card numbers and other payment details, and the final confirmation of a payment or order. Computer Use is designed to hand these steps to you, so stay nearby when a task may reach one, and never put such details in your request.
  • CAPTCHAs and other "are you a robot" checks.
  • Windows administrator prompts (User Account Control). Computer Use cannot operate them.
  • A step it finds no way to do, or a decision about content that is already there, such as whether to replace an existing draft.

Do the step and press Done

  1. Under the result in the task window, a message says what you need to do. Windows also shows an Action required notification.
  2. Do the step yourself in the app.
  3. Optionally type a short note in Reply (optional)..., for example "I signed in with the work account".
  4. Click Done, or press enter in the reply box.

Computer Use then takes a fresh look at the screen and continues with the steps it has left. A handoff has no time limit. To end the task instead, click Stop. See When a task waits for you.

Protections

  • Password fields are hidden from its screenshots. In the window Computer Use is working in, password fields and other fields for secrets are blacked out before a screenshot is sent. This is best effort: it may not catch every field. Close other private windows and anything else you don't want sent.
  • It does not type passwords. It refuses to type into fields it recognizes as password fields. Don't put passwords or codes in your request.
  • Payments are designed to stay with you. It is built to ask before anything that pays, buys or moves money, even if your request asked for it. See What it hands back to you.
  • Your request sets the limits. Actions that commit something are checked against your request. One your request did not allow asks you first, and one that clearly has nothing to do with your task does not run, even if something on the screen asks for it. See How ActionWright decides what may run.

Limits

  • Step limit. Each run has a step limit. When it is reached, ActionWright asks whether to continue. See Limits per run. Every action counts as a step, including every screenshot; a step it hands to you does not count.
  • No time limit. A Computer Use run ends when the job is done, when it reaches the step limit, or when you stop it. A long job can keep your PC busy for a long time, so watch it.
  • One task at a time. There is only one screen, mouse and keyboard, so if another task needs Computer Use while it is busy, that task cannot use it until the first one ends.
  • Some Microsoft Store apps may not open by name. Open the app yourself before you run the task, or ask ActionWright to find it through the Start menu search.
  • It can go in circles. Sometimes Computer Use repeats similar actions without getting closer, for example small adjustments or waiting on a screen that doesn't change. Watch the Execution Trace. If it keeps repeating, press ctrl+esc, then describe a different way in a new message, or do that step yourself.
  • Stop during an action. If you stop it while it is sending, saving or deleting something, that action may or may not have happened. See Stop it at once.
  • Costs. Screenshots make requests to your provider large, and a long Computer Use task can use many tokens. See Costs.

What is sent to your AI provider

Computer Use works through your AI provider. It sends your provider screenshots of the window it works in or of your whole screen, the titles of the open windows, and the text it types. Anything visible on your screen can be in a screenshot, apart from the password fields it hides. ActionWright does not save the screenshots as image files. See Privacy.

Risks to keep in mind

  • Only your own words count as your request; text on the screen never does. See Instructions inside pages and files.
  • Computer Use acts with every account you are signed in to in your apps.

See Safety.

If something goes wrong

ProblemLikely causeWhat to try
The task reports it cannot use Computer UseEnable Computer Use agent (screen control) is off.Turn it on in Settings and ask again. See Turn Computer Use on or off.
The task fails with an AI provider error as soon as Computer Use startsYour task model does not accept images.Choose a model that accepts images. See Models.
It clicks or types in the wrong placeYou used the mouse or keyboard while it worked, or a pop-up appeared.Leave the mouse and keyboard alone, close the pop-up, and ask again.
An app doesn't openSome Microsoft Store apps can't be opened by name, or the name doesn't match the app's name in the Start menu.Open the app yourself and ask again, or use the name exactly as the Start menu shows it.
It keeps repeating the same kind of actionIt is stuck on a step.Press ctrl+esc. See Limits.
The task waits with a message and DoneComputer Use handed you a sign-in, code, check or decision.Do the step, then click Done.
A Windows administrator prompt is on the screenAn app asked for administrator permission.Answer the prompt yourself. If the task is waiting, click Done.
An action was blockedIt did not match your request.Say exactly what you want in a new message, for example "save the file".
ActionWright says another screen session is activeComputer Use is still busy with another task. Only one task can use it at a time.Wait for the other task to finish, or stop it, then ask again. If it keeps happening, quit and restart ActionWright.
"Step budget reached" appearsThe job needed more steps than one run allows.See The step limit.
ctrl+esc does nothingThe task is waiting for you.Click Stop in the task window.
It keeps going for a moment after you press ctrl+escIt stops as soon as it safely can, and an action already sent cannot be undone.Wait for "Task was stopped by the user.", then check what the last action did.
Your clipboard holds different textComputer Use pasted long or non-English text through the clipboard.Copy your own content again.

For other problems, see Troubleshooting.