computer-control
Automates desktop GUI workflows through the computer use API with screenshots. Use when scripting GUI interactions.
How do I install this agent skill?
npx skills add https://github.com/athola/claude-night-market --skill computer-controlIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides a framework for automating desktop GUI workflows using Claude's Computer Use API. It implements security best practices such as sandboxing, human-in-the-loop confirmation, and iteration limits to manage the risks associated with desktop automation.
- Socketpass
No alerts
- Snykwarn
Risk: MEDIUM · 1 issue
What does this agent skill do?
Computer Control Skill
Use Claude's Computer Use API to see and control desktop environments through screenshots and mouse/keyboard actions.
When To Use
- Automating GUI-based workflows that lack CLI alternatives
- Testing web applications through visual interaction
- Filling forms, navigating menus, or interacting with desktop apps
- Building automation pipelines that need visual verification
When NOT To Use
- Tasks achievable through CLI or API (no GUI needed)
- Browser automation better served by Playwright or CDP
Why this stays opt-in. Per docs/inclusive-defaults.md (TRUE-exception category 4), Computer Use takes screenshots and synthesizes keyboard/mouse input: cross-process side effects that must always be explicitly invoked, never default-on.
Architecture
The computer use system has three layers:
- Display Toolkit (
phantom.display) - executes OS-level actions via xdotool/scrot on the real or virtual display - Agent Loop (
phantom.loop) - manages the conversation cycle between Claude API and the display toolkit - CLI (
phantom.cli) - command-line interface for running tasks or checking environment readiness
User Task
|
v
Agent Loop <----> Claude API (beta)
| |
v v
Display Toolkit tool_use responses
| (click, type, screenshot)
v
OS Commands (xdotool, scrot)
|
v
Display (X11 / Xvfb / WSLg)
Quick Start
Check environment
cd plugins/phantom
uv run python -m phantom.cli --check
Run a task
export ANTHROPIC_API_KEY="sk-ant-..."
uv run python -m phantom.cli "Open Firefox and search for Claude AI"
--confirm asks before each action and runs it only on a reply that
starts with y or Y. Any other reply, or a closed stdin, rejects it.
--budget-usd 0.50 stops the run with budget_exceeded once its
estimated API cost passes that many dollars.
Use in Python
from phantom.display import DisplayConfig
from phantom.loop import LoopConfig, LoopHooks, LoopSetup, run_loop
def approve(action: dict) -> bool:
try:
reply = input(f"Run {action.get('action')}? [y/N] ")
except EOFError: # closed stdin is a no
return False
return reply.strip().lower().startswith("y")
result = run_loop(
task="Take a screenshot of the desktop",
api_key="sk-ant-...",
setup=LoopSetup(
loop_config=LoopConfig(
model="claude-sonnet-5",
max_iterations=10,
budget_usd=0.50,
),
display_config=DisplayConfig(width=1920, height=1080),
),
hooks=LoopHooks(confirm_callback=approve),
)
print(f"Done in {result.iterations} iterations")
print(result.final_text)
API Versions
| Model | Tool Version | Beta Flag |
|---|---|---|
| Opus 4.6, Sonnet 4.6, Opus 4.5 | computer_20251124 | computer-use-2025-11-24 |
| Sonnet 4.5, Haiku 4.5, older | computer_20250124 | computer-use-2025-01-24 |
The resolve_tool_version() function handles this mapping
automatically based on the model name.
Available Actions
All versions:
screenshot- capture displayleft_click- click at[x, y]type- type text stringkey- press key combo (e.g.,ctrl+s)mouse_move- move cursor
Enhanced (20250124+):
scroll- scroll with direction and amountleft_click_drag- drag between coordinatesright_click,middle_click,double_click,triple_clickhold_key- hold key for durationwait- pause between actions
Latest (20251124):
zoom- inspect screen region at full resolution
Safety
Computer use carries risks. Follow these guidelines:
- Use a sandbox: Run in Docker or a VM, not your main OS
- Limit access: Do not provide login credentials unless necessary, and never for banking or sensitive services
- Set iteration caps: Always use
max_iterationsto prevent runaway API costs - Human approval: For actions with real-world consequences,
pass
confirm_callback(--confirmon the CLI). It returns True to run the action and False to reject it.on_actiononly observes actions, and its return value is ignored - Close sensitive apps: Claude sees the full screen via screenshots; close anything private before starting
Environment Requirements
Linux (native or WSL2 with WSLg):
sudo apt install xdotool scrot xclip
Headless (Docker/CI):
# Install Xvfb for virtual display
sudo apt install xvfb xdotool scrot xclip
Xvfb :1 -screen 0 1920x1080x24 &
export DISPLAY=:1
Prompting Tips
- Be specific about each step of the task
- Add "After each step, take a screenshot and verify" to catch mistakes early
- Use keyboard shortcuts when UI elements are hard to click
- Provide example screenshots for repeatable workflows
- Set a system prompt with domain-specific instructions
Exit Criteria
-
uv run python -m phantom.cli --checkexits 0 before any task is launched; if it fails, required OS tools (xdotool, scrot, xclip) are installed or Xvfb is started before proceeding -
max_iterationsset on everyrun_loop()call; no task launched without an explicit iteration cap to prevent runaway API costs - Final result includes
result.iterationscount andresult.final_textconfirming the task outcome; emptyfinal_texttreated as failure, not success - Sensitive applications (password managers, banking, private files) closed before task starts; task prompt does not contain raw credentials
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/athola/claude-night-market/computer-control">View computer-control on skillZs</a>