Setup & Installation
Or with the ClawHub CLI, for registry-managed skill folders outside a full OpenClaw workspace:
Version History
Version 1.0.3 - Enforces a new requirement to proactively report automation results after task completion, summarizing actions, findings, screenshots, and files. - Adds an explicit "Report results" step to the workflow and documents it as best practice. - Updates example environment variable setup with new supported models (Qwen 3.5, Doubao Seed 2.0 Lite). - Minor naming update: skill is now "desktop-computer-automation". - Refines and clarifies documentation, especially around closing response loops and model selection.
What This Skill Does
Controls macOS, Windows, and Linux desktops with natural language using Midscene.js. Operates from screenshots, so it works with any visible UI without requiring DOM access or accessibility labels. Automates clicks, typing, window management, and app interactions across any application.
Unlike Selenium or AppleScript, it works with any app on any stack because it reads the screen directly rather than relying on DOM structure or accessibility APIs.
When to use it
- Automating repetitive form entry in a legacy desktop app
- Taking targeted screenshots of specific windows for documentation
- Launching apps and navigating their menus via text commands
- Extracting visible on-screen data from any GUI application
- Running multi-step workflows across different desktop programs
Example Workflow
Here's how your AI assistant might use this skill in practice.
User asks: Open Chrome, search for the weather in New York, and tell me the result
- 1Connect to the desktop with `npx @midscene/computer@1 connect`
- 2Open Chrome using the system launcher (`open -a 'Google Chrome'` on macOS)
- 3Take a screenshot to confirm Chrome is in the foreground
- 4Run `act --prompt "search for weather in New York and read the current conditions from the page"` to perform the search and extract results
- 5Disconnect and report the weather information extracted from the screen
Current weather conditions for New York, with a screenshot showing the search result page
Requirements
Accounts, API keys, or tools you or your AI assistant may need to set up while using this skill.