native-devtools-mcp-automation
MCP server for computer use & browser automation - screenshot, OCR, click, type, find_text, Chrome/Electron CDP, template matching on macOS, Windows & Android
How do I install this agent skill?
npx skills add https://github.com/reason-machines/devtools-skills --skill native-devtools-mcp-automationIs this agent skill safe to install?
- Gen Agent Trust Hubwarn
This skill provides computer use and browser automation capabilities through a third-party MCP server. It involves installing and executing code from an unknown GitHub user and a public package registry. The skill requires high-level system permissions and includes tools for arbitrary application launching and JavaScript evaluation, which creates a broad attack surface for indirect prompt injection and dynamic code execution.
- Socketwarn
1 alert: gptSecurity
- Snykwarn
Risk: MEDIUM · 3 issues
What does this agent skill do?
Native Devtools MCP Automation
Skill by ara.so — Devtools Skills collection.
native-devtools-mcp is an MCP server that gives AI agents direct control over native desktop apps, Chrome/Electron browsers, and Android devices. It provides screenshots, OCR, accessibility-first element lookup, input simulation, window management, Chrome DevTools Protocol (CDP), and ADB — all in one local server.
Works with Claude Desktop, Claude Code, Cursor, and any MCP-compatible client.
Platform Support
- macOS: Full support with Accessibility tree dispatch (preferred), screenshots, OCR (Vision), input simulation
- Windows: UI Automation, screenshots, OCR (Windows Media OCR), input simulation
- Android: ADB-based screenshots, uiautomator text lookup, input, app management
- Chrome/Electron: CDP-based DOM automation for web content and Electron apps
Installation
Quick Start (no install)
npx -y native-devtools-mcp
Global Install
npm install -g native-devtools-mcp
Build from Source (Rust)
git clone https://github.com/sh3ll3x3c/native-devtools-mcp
cd native-devtools-mcp
cargo build --release
# Binary: ./target/release/native-devtools-mcp
Setup Wizard
Run the setup wizard to configure permissions and MCP clients:
npx native-devtools-mcp setup
This will:
- Check permissions (Accessibility and Screen Recording on macOS)
- Detect MCP clients (Claude Desktop, Claude Code, Cursor)
- Write the correct configuration
MCP Client Configuration
Claude Desktop (macOS)
Config file: ~/Library/Application Support/Claude/claude_desktop_config.json
{
"mcpServers": {
"native-devtools": {
"command": "/Applications/NativeDevtools.app/Contents/MacOS/native-devtools-mcp"
}
}
}
Claude Desktop (Windows)
Config file: %APPDATA%\Claude\claude_desktop_config.json
{
"mcpServers": {
"native-devtools": {
"command": "C:\\path\\to\\native-devtools-mcp.exe"
}
}
}
Claude Code / Cursor / Other MCP Clients
{
"mcpServers": {
"native-devtools": {
"command": "npx",
"args": ["-y", "native-devtools-mcp"]
}
}
}
Claude Code Auto-Approval
To avoid approving every tool call, add to .claude/settings.local.json:
{
"permissions": {
"allow": ["mcp__native-devtools__*"]
}
}
Three Approaches to Interaction
1. Visual (Universal)
Works with any app — games, Qt apps, custom renderers, anything without an accessible tree.
Key Tools: take_screenshot, find_text, click, type_text, find_image
2. AX Dispatch (macOS - Preferred for Native Apps)
Element-precise automation for AppKit/SwiftUI apps. Doesn't move the cursor or steal focus.
Key Tools: take_ax_snapshot, ax_click, ax_set_value, ax_select
3. CDP (Chrome/Electron)
DOM-level automation for web content and Electron apps.
Key Tools: cdp_connect, cdp_find_elements, cdp_click, cdp_fill, cdp_navigate
Core Tools Reference
Screenshot & OCR
// Take a full screen screenshot
take_screenshot()
// Take a window screenshot
take_screenshot(window_id: number)
// Take a region screenshot
take_screenshot(
x: number,
y: number,
width: number,
height: number
)
// Find text with OCR
find_text(
text: string,
window_id?: number,
x?: number,
y?: number,
width?: number,
height?: number
)
Input Simulation
// Click at global coordinates
click(x: number, y: number)
// Click relative to window
click(x: number, y: number, window_id: number)
// Click relative to screenshot
click(x: number, y: number, screenshot_id: string)
// Double-click
click(x: number, y: number, double_click: true)
// Right-click
click(x: number, y: number, right_click: true)
// Drag
drag(
from_x: number,
from_y: number,
to_x: number,
to_y: number,
window_id?: number
)
// Type text
type_text(text: string)
// Press key
press_key(key: string, modifiers?: string[])
// Examples: "Return", "Tab", "Escape"
// Modifiers: ["command"], ["control", "shift"]
// Scroll
scroll(delta_x: number, delta_y: number)
Window Management
// List all windows
list_windows()
// Focus a window
focus_window(window_id: number)
// Launch an app
launch_app(app_name: string, args?: string[])
// Quit an app
quit_app(app_name: string)
// Record window frames
record_window(
window_id: number,
duration_ms: number,
interval_ms?: number
)
macOS Accessibility Tree (AX Dispatch)
// Take AX snapshot
take_ax_snapshot(
window_id?: number,
include_descriptions?: boolean
)
// Returns tree with element UIDs (a1, a2, a3...)
// Click an AX element
ax_click(uid: string)
// Set value (text fields, sliders)
ax_set_value(uid: string, value: string)
// Select item (menus, lists)
ax_select(uid: string)
// Inspect element details
ax_inspect(uid: string)
AX Dispatch Flow Example:
// 1. Take AX snapshot of System Settings
const windows = await list_windows();
const settingsWindow = windows.find(w => w.app_name === "System Settings");
const snapshot = await take_ax_snapshot(settingsWindow.id);
// 2. Find the element you want (e.g., "a12" is the "Privacy & Security" button)
// The LLM reads the tree structure from the snapshot
// 3. Click it without moving the cursor
await ax_click("a12");
// 4. Take another snapshot to verify
const updatedSnapshot = await take_ax_snapshot(settingsWindow.id);
Template Matching
// Load an image template
load_image(
path: string,
name: string
)
// Find the template in a screenshot
find_image(
template_name: string,
screenshot_id?: string,
window_id?: number,
threshold?: number // 0.0-1.0, default 0.8
)
Template Matching Flow:
// 1. Save a reference image of a button/icon
// (manually crop from a screenshot)
// 2. Load it
await load_image("/path/to/button.png", "submit_button");
// 3. Take a screenshot
const screenshot = await take_screenshot();
// 4. Find the template
const matches = await find_image("submit_button", screenshot.id);
// 5. Click the first match
if (matches.length > 0) {
await click(matches[0].x, matches[0].y);
}
Chrome DevTools Protocol (CDP)
// Connect to Chrome/Electron
cdp_connect(
port: number,
host?: string // default "localhost"
)
// Navigate
cdp_navigate(url: string)
// Find elements (returns UIDs: d1, d2, d3...)
cdp_find_elements(
query: string,
limit?: number
)
// Take DOM snapshot
cdp_take_dom_snapshot()
// Click element
cdp_click(uid: string)
// Hover over element
cdp_hover(uid: string)
// Fill input field
cdp_fill(uid: string, value: string)
// Type into element
cdp_type(uid: string, text: string)
// Press key
cdp_press_key(key: string)
// Examples: "Enter", "Tab", "Escape"
// Wait for condition
cdp_wait_for(
text?: string[],
selector?: string,
timeout_ms?: number
)
// Evaluate JavaScript
cdp_eval(expression: string)
// Handle alert/confirm/prompt
cdp_handle_dialog(accept: boolean, prompt_text?: string)
// Manage tabs
cdp_new_tab(url?: string)
cdp_close_tab(tab_id: string)
cdp_list_tabs()
cdp_switch_tab(tab_id: string)
// Inspect element
cdp_inspect_element(uid: string)
// Get element attributes
cdp_get_attributes(uid: string)
// Screenshot element
cdp_screenshot_element(uid: string)
CDP Flow Example:
// 1. Launch Chrome with remote debugging
await launch_app(
"Google Chrome",
["--remote-debugging-port=9222", "--user-data-dir=/tmp/chrome-profile"]
);
// 2. Connect
await cdp_connect(9222);
// 3. Navigate
await cdp_navigate("https://example.com");
// 4. Find elements
const elements = await cdp_find_elements("search");
// Returns: [{ uid: "d1", tag: "input", text: "", ... }, ...]
// 5. Fill and submit
await cdp_fill("d1", "search query");
await cdp_press_key("Enter");
// 6. Wait for results
await cdp_wait_for(["Results"], null, 5000);
// 7. Take DOM snapshot
const dom = await cdp_take_dom_snapshot();
Android (ADB)
// List connected devices
adb_devices()
// Take screenshot
adb_screenshot(device_id?: string)
// Find text (uiautomator)
adb_find_text(
text: string,
device_id?: string
)
// Tap coordinates
adb_tap(
x: number,
y: number,
device_id?: string
)
// Type text
adb_type(text: string, device_id?: string)
// Press key
adb_press_key(
key: string,
device_id?: string
)
// Examples: "KEYCODE_HOME", "KEYCODE_BACK"
// Swipe
adb_swipe(
from_x: number,
from_y: number,
to_x: number,
to_y: number,
duration_ms?: number,
device_id?: string
)
// Launch app
adb_launch_app(
package: string,
device_id?: string
)
// Stop app
adb_stop_app(
package: string,
device_id?: string
)
Common Patterns
Pattern 1: Visual Navigation with OCR
// 1. Take a screenshot
const screenshot = await take_screenshot();
// 2. Find the target text
const matches = await find_text("Submit");
// 3. Click the first match
if (matches.length > 0) {
await click(matches[0].center_x, matches[0].center_y, screenshot.id);
}
Pattern 2: AX Dispatch on macOS (Preferred)
// 1. List windows
const windows = await list_windows();
const targetWindow = windows.find(w => w.title.includes("Notes"));
// 2. Take AX snapshot
const snapshot = await take_ax_snapshot(targetWindow.id, true);
// 3. Find element by role/label in the tree
// (LLM identifies "a5" is the "New Note" button from the snapshot)
// 4. Click without moving cursor
await ax_click("a5");
// 5. Set value in text field (e.g., "a8")
await ax_set_value("a8", "Meeting notes for 2026-05-18");
Pattern 3: Web Automation with CDP
// 1. Launch Chrome with debugging
await launch_app(
"Google Chrome",
["--remote-debugging-port=9222", "--new-window", "https://github.com/login"]
);
// 2. Connect
await cdp_connect(9222);
// 3. Find login fields
const elements = await cdp_find_elements("login");
// Returns: [{ uid: "d1", tag: "input", ... }, { uid: "d2", tag: "input", type: "password", ... }]
// 4. Fill credentials from env
await cdp_fill("d1", process.env.GITHUB_USERNAME);
await cdp_fill("d2", process.env.GITHUB_PASSWORD);
// 5. Click submit button
const submitElements = await cdp_find_elements("Sign in");
await cdp_click(submitElements[0].uid);
// 6. Wait for redirect
await cdp_wait_for(null, "header", 10000);
Pattern 4: Electron App Automation
// 1. Launch Electron app with debugging
await launch_app(
"Signal",
["--remote-debugging-port=9223"]
);
// 2. Connect CDP
await cdp_connect(9223);
// 3. Automate like a web app
await cdp_find_elements("compose");
await cdp_click("d1");
await cdp_type("d1", "Hello from automation!");
await cdp_press_key("Enter");
Pattern 5: Template Matching for Custom UI
// 1. Load icon template
await load_image("/path/to/gear-icon.png", "settings_icon");
// 2. Take screenshot
const screenshot = await take_screenshot();
// 3. Find icon
const matches = await find_image("settings_icon", screenshot.id, null, 0.85);
// 4. Click if found
if (matches.length > 0) {
await click(matches[0].x, matches[0].y, screenshot.id);
}
Pattern 6: Android UI Automation
// 1. List devices
const devices = await adb_devices();
const deviceId = devices[0].id;
// 2. Take screenshot
const screenshot = await adb_screenshot(deviceId);
// 3. Find text
const matches = await adb_find_text("Settings", deviceId);
// 4. Tap
if (matches.length > 0) {
await adb_tap(matches[0].center_x, matches[0].center_y, deviceId);
}
// 5. Type in a field
await adb_tap(500, 300, deviceId); // Focus input
await adb_type("Hello Android", deviceId);
await adb_press_key("KEYCODE_ENTER", deviceId);
Operational Safety
- Hands off: When the agent is clicking/typing, don't move your mouse or type. Real hardware inputs conflict with simulated ones.
- Focus matters: Ensure the target window is visible. If a popup steals focus, clicks may land in the wrong window.
- Prefer AX Dispatch on macOS: For native apps, use
take_ax_snapshot+ax_click/ax_set_valueto avoid moving the cursor and stealing focus.
Permissions (macOS)
The server requires:
- Accessibility: For input simulation and AX tree access
- Screen Recording: For screenshots
Grant both in System Settings → Privacy & Security → Accessibility and Screen Recording.
Without these, clicks silently fail and screenshots return black rectangles.
Troubleshooting
macOS: Clicks don't work
- Cause: Missing Accessibility permission
- Fix: System Settings → Privacy & Security → Accessibility → enable the app
macOS: Screenshots are black
- Cause: Missing Screen Recording permission
- Fix: System Settings → Privacy & Security → Screen Recording → enable the app
CDP: Can't connect
- Cause: App not launched with
--remote-debugging-port - Fix: Use
launch_appwith the correct args:await launch_app("Google Chrome", ["--remote-debugging-port=9222"]);
ADB: No devices found
- Cause: USB debugging not enabled or device not connected
- Fix:
- Enable USB debugging on Android device (Settings → Developer options)
- Connect via USB or Wi-Fi (
adb tcpip 5555, thenadb connect <ip>:5555) - Run
adb devicesto verify
OCR finds nothing
- Cause: Text too small, low contrast, or obscured
- Workarounds:
- Use template matching instead (
load_image+find_image) - Use AX Dispatch on macOS (
take_ax_snapshot) - Use CDP for web content (
cdp_find_elements)
- Use template matching instead (
Windows: UI Automation elements missing
- Cause: Some Qt/Electron apps don't expose UI Automation
- Workaround: Use visual approach (screenshots + OCR) or CDP for Electron
Real-World Examples
Example 1: Automate System Settings on macOS
// 1. Launch System Settings
await launch_app("System Settings");
// 2. Wait for window
const windows = await list_windows();
const settingsWindow = windows.find(w => w.app_name === "System Settings");
// 3. Take AX snapshot
const snapshot = await take_ax_snapshot(settingsWindow.id, true);
// 4. Find "Privacy & Security" in the sidebar (e.g., a12)
await ax_click("a12");
// 5. Take another snapshot to find the next element
const privacySnapshot = await take_ax_snapshot(settingsWindow.id, true);
// 6. Click "Screen Recording" (e.g., a25)
await ax_click("a25");
Example 2: Fill a Web Form with CDP
// 1. Launch Chrome
await launch_app("Google Chrome", [
"--remote-debugging-port=9222",
"--new-window",
"https://example.com/contact"
]);
// 2. Connect CDP
await cdp_connect(9222);
// 3. Find form fields
const fields = await cdp_find_elements("contact form");
// Returns: [
// { uid: "d1", tag: "input", placeholder: "Name", ... },
// { uid: "d2", tag: "input", placeholder: "Email", ... },
// { uid: "d3", tag: "textarea", placeholder: "Message", ... }
// ]
// 4. Fill the form
await cdp_fill("d1", "John Doe");
await cdp_fill("d2", "john@example.com");
await cdp_fill("d3", "This is a test message.");
// 5. Submit
const submitBtn = await cdp_find_elements("Submit");
await cdp_click(submitBtn[0].uid);
// 6. Wait for confirmation
await cdp_wait_for(["Thank you"], null, 5000);
Example 3: Android App Testing
// 1. List devices
const devices = await adb_devices();
const device = devices[0].id;
// 2. Launch app
await adb_launch_app("com.example.app", device);
// 3. Wait and screenshot
await new Promise(resolve => setTimeout(resolve, 2000));
const screenshot = await adb_screenshot(device);
// 4. Find and tap "Sign In" button
const signInMatches = await adb_find_text("Sign In", device);
if (signInMatches.length > 0) {
await adb_tap(signInMatches[0].center_x, signInMatches[0].center_y, device);
}
// 5. Fill username
const usernameMatches = await adb_find_text("Username", device);
await adb_tap(usernameMatches[0].center_x, usernameMatches[0].center_y, device);
await adb_type(process.env.TEST_USERNAME, device);
// 6. Fill password
await adb_press_key("KEYCODE_TAB", device);
await adb_type(process.env.TEST_PASSWORD, device);
// 7. Submit
await adb_press_key("KEYCODE_ENTER", device);
Example 4: Automate VS Code with CDP
// 1. Launch VS Code with debugging
await launch_app("Visual Studio Code", ["--remote-debugging-port=9224"]);
// 2. Connect
await cdp_connect(9224);
// 3. Open command palette
await cdp_press_key("Meta+Shift+P"); // Meta = Cmd on macOS, Win on Windows
// 4. Wait for palette
await cdp_wait_for(null, ".quick-input-widget", 2000);
// 5. Type command
await cdp_type(".quick-input-widget input", "File: Open File");
await cdp_press_key("Enter");
// 6. Navigate file picker with JS
await cdp_eval(`
document.querySelector('.monaco-inputbox input').value = '/path/to/file.js';
document.querySelector('.monaco-inputbox input').dispatchEvent(new Event('input'));
`);
await cdp_press_key("Enter");
Best Practices
- Always verify state: Take a screenshot or snapshot after an action to confirm it succeeded.
- Use the right tool for the job:
- Native macOS apps → AX Dispatch
- Web/Electron → CDP
- Custom/legacy UI → Visual (screenshots + OCR or template matching)
- Handle timing: Add
cdp_wait_foror manual delays after navigation/clicks before the next action. - Reference env vars for secrets: Never hardcode credentials in automation scripts.
- Use
record_windowfor debugging: Record a window's state over time to understand UI behavior. - Test permissions early: Run
npx native-devtools-mcp setupbefore writing automation scripts.
Additional Resources
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/reason-machines/devtools-skills/native-devtools-mcp-automation">View native-devtools-mcp-automation on skillZs</a>