autocli-web-scraping

AutoCLI Web Scraping Skill

Skill by ara.so — Devtools Skills collection.

AutoCLI is a blazing fast, memory-safe command-line tool written in Rust that fetches information from 55+ websites with a single command. It covers Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, and more, with support for browser session reuse, AI-powered adapter generation, and multi-format output.

Key Features:

• 55 sites, 333 commands built-in
• Browser session reuse (no token management needed)
• AI-powered adapter generation via autocli.ai
• Declarative YAML pipeline for custom adapters
• Single 4.7MB binary, zero runtime dependencies
• Up to 12x faster and 10x less memory than Node.js alternatives

Installation

One-line Install (macOS / Linux)

curl -fsSL https://raw.githubusercontent.com/nashsu/autocli/main/scripts/install.sh | sh

Manual Installation

Download the appropriate binary from GitHub Releases:

• macOS (Apple Silicon): autocli-aarch64-apple-darwin.tar.gz
• macOS (Intel): autocli-x86_64-apple-darwin.tar.gz
• Linux (x86_64): autocli-x86_64-unknown-linux-musl.tar.gz
• Windows (x64): autocli-x86_64-pc-windows-msvc.zip

Extract and place in your PATH:

tar -xzf autocli-*.tar.gz
sudo mv autocli /usr/local/bin/

Chrome Extension (Required for Browser Commands)

1. Download autocli-chrome-extension.zip from releases
2. Extract to any directory
3. Open chrome://extensions
4. Enable "Developer mode"
5. Click "Load unpacked" and select the extracted folder

Public API commands (hackernews, devto, etc.) work without the extension.

Basic Usage

Discovery Commands

# List all available commands
autocli --help

# List commands for specific site
autocli twitter --help
autocli bilibili --help

# Run diagnostics
autocli doctor

# List all sites and their command counts
autocli list

Public API Commands (No Browser Required)

# Hacker News top stories
autocli hackernews top --limit 10

# Hacker News search
autocli hackernews search "rust" --limit 5

# Dev.to top articles
autocli devto top --limit 10

# Lobsters hot stories
autocli lobsters hot --limit 10

# Stack Overflow hot questions
autocli stackoverflow hot --limit 10

# arXiv paper search
autocli arxiv search "machine learning" --limit 5

# Wikipedia search
autocli wikipedia search "rust programming"

# Linux.do forum hot topics
autocli linux-do hot --limit 10

Browser-Based Commands (Requires Extension + Login)

# Twitter trending topics
autocli twitter trending

# Twitter search
autocli twitter search "rust lang" --limit 10

# Twitter timeline
autocli twitter timeline --limit 20

# Reddit frontpage
autocli reddit frontpage --limit 10

# Reddit subreddit posts
autocli reddit subreddit rust --limit 15

# Bilibili hot videos
autocli bilibili hot --limit 20

# Bilibili search
autocli bilibili search "rust programming" --limit 10

# Xiaohongshu search
autocli xiaohongshu search "travel" --limit 10

# Zhihu hot topics
autocli zhihu hot --limit 10

# YouTube search
autocli youtube search "rust tutorial" --limit 5

# Weibo hot topics
autocli weibo hot --limit 10

Output Formats

AutoCLI supports multiple output formats for easy integration with other tools:

# Table format (default)
autocli hackernews top --limit 5

# JSON output
autocli hackernews top --limit 5 --format json

# YAML output
autocli hackernews top --limit 5 --format yaml

# CSV output
autocli hackernews top --limit 5 --format csv

# Markdown output
autocli hackernews top --limit 5 --format markdown

JSON Processing with jq

# Extract specific fields
autocli hackernews top --limit 5 --format json | jq '.[].title'

# Filter by score
autocli hackernews top --limit 20 --format json | jq '.[] | select(.score > 100)'

# Count results
autocli reddit frontpage --limit 50 --format json | jq 'length'

AI-Powered Features

AutoCLI integrates with autocli.ai for AI-powered adapter generation and sharing.

Authentication

# Authenticate with autocli.ai
autocli auth

This opens your browser to get an API token and saves it to ~/.autocli/config.json.

Generate Adapters with AI

Use the Chrome extension to visually select data elements:

1. Navigate to any website in Chrome
2. Click the AutoCLI extension icon
3. Use the selector tool to pick data elements
4. Click "Generate" to let AI create the adapter
5. The adapter is saved locally and synced to autocli.ai

Search Community Adapters

# Search by URL
autocli search https://news.ycombinator.com

# Search by domain
autocli search producthunt.com

Searches autocli.ai for community-shared adapters. Select one to download and use immediately.

Environment Variables

# Override API server (default: https://www.autocli.ai)
export AUTOCLI_API_BASE=https://custom-server.com

Advanced Usage

Download Media

# Download YouTube video (requires yt-dlp)
autocli youtube download "https://youtube.com/watch?v=..."

# Download Bilibili video
autocli bilibili download "https://www.bilibili.com/video/..."

# Download Xiaohongshu content
autocli xiaohongshu download "https://www.xiaohongshu.com/..."

Social Media Interactions

# Twitter operations
autocli twitter post "Hello from AutoCLI!"
autocli twitter reply TWEET_ID "Great post!"
autocli twitter like TWEET_ID
autocli twitter bookmark TWEET_ID
autocli twitter follow USERNAME
autocli twitter bookmarks --limit 10

# Reddit operations
autocli reddit upvote POST_ID
autocli reddit comment POST_ID "Interesting discussion"
autocli reddit subscribe SUBREDDIT_NAME
autocli reddit saved --limit 10

# Xiaohongshu publishing
autocli xiaohongshu publish --title "My Post" --content "Content here" --images "image1.jpg,image2.jpg"

Job Search (BOSS Zhipin)

# Search jobs
autocli boss search "Rust developer" --city "Beijing"

# Greet recruiter
autocli boss greet JOB_ID "Hello, I'm interested in this position"

# Batch greet
autocli boss batchgreet JOB_ID1,JOB_ID2,JOB_ID3 "Hello, I'm interested"

# View chat list
autocli boss chatlist

# View messages
autocli boss chatmsg BOSS_ID

Shell Completion

Generate shell completions for better autocomplete experience:

# Bash
autocli completion bash >> ~/.bashrc

# Zsh
autocli completion zsh >> ~/.zshrc

# Fish
autocli completion fish > ~/.config/fish/completions/autocli.fish

# PowerShell
autocli completion powershell >> $PROFILE

Configuration

AutoCLI stores configuration in ~/.autocli/config.json:

{
  "api_token": "your-autocli-ai-token",
  "browser_ws_endpoint": "ws://localhost:9222",
  "default_format": "table",
  "adapters_dir": "~/.autocli/adapters"
}

Creating Custom Adapters

AutoCLI uses declarative YAML pipelines for custom adapters. Adapters are stored in ~/.autocli/adapters/.

Example Adapter Structure

name: example-site
description: Example site adapter
version: 1.0.0
author: Your Name
commands:
  - name: hot
    description: Get hot posts
    mode: public  # or 'browser'
    pipeline:
      - step: fetch
        url: https://api.example.com/hot
        method: GET
      - step: extract
        selector: $.data[*]
        fields:
          - name: title
            path: $.title
          - name: url
            path: $.url
          - name: score
            path: $.score
      - step: output
        format: table

Adapter Modes

• public: Uses public APIs, no authentication needed
• browser: Requires Chrome extension and browser session

Pipeline Steps

1. fetch: HTTP request to target URL
2. extract: Extract data using JSON path or CSS selectors
3. transform: Modify/filter extracted data
4. output: Format and display results

Integration with AI Agents

Register in Agent Configuration

Add to .cursorrules or AGENT.md:

# Discover all available AutoCLI commands
autocli list

# Use specific commands as needed
autocli hackernews top --limit 10 --format json
autocli twitter search "topic" --format json

Register Local CLI Tools

# Register local CLI tools for AI agent access
autocli register gh      # GitHub CLI
autocli register docker  # Docker CLI
autocli register kubectl # Kubernetes CLI

Common Patterns

Pipeline Data from Multiple Sites

# Get tech news from multiple sources
autocli hackernews top --limit 5 --format json > hn.json
autocli lobsters hot --limit 5 --format json > lobsters.json
autocli devto top --limit 5 --format json > devto.json
jq -s 'add' hn.json lobsters.json devto.json > combined.json

Monitor Topics Across Platforms

#!/bin/bash
TOPIC="rust"

echo "=== Hacker News ==="
autocli hackernews search "$TOPIC" --limit 3

echo "=== Reddit ==="
autocli reddit search "$TOPIC" --limit 3

echo "=== Twitter ==="
autocli twitter search "$TOPIC" --limit 3

echo "=== Dev.to ==="
autocli devto tag "$TOPIC" --limit 3

Archive Bookmarks

# Export Twitter bookmarks
autocli twitter bookmarks --limit 100 --format json > bookmarks_$(date +%Y%m%d).json

# Export Reddit saved posts
autocli reddit saved --limit 100 --format json > reddit_saved_$(date +%Y%m%d).json

Troubleshooting

Browser Connection Issues

# Run diagnostics
autocli doctor

# Check Chrome extension is loaded and active
# Verify browser is running on ws://localhost:9222

# Restart Chrome with remote debugging:
# macOS:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222

# Linux:
google-chrome --remote-debugging-port=9222

# Windows:
"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222

Authentication Errors

# Re-authenticate
autocli auth

# Verify token in config
cat ~/.autocli/config.json | jq '.api_token'

# Clear and re-auth
rm ~/.autocli/config.json
autocli auth

Command Not Found

# Verify installation
which autocli

# Check available commands
autocli --help

# Update to latest version
curl -fsSL https://raw.githubusercontent.com/nashsu/autocli/main/scripts/install.sh | sh

Rate Limiting

Some sites may rate-limit requests. Use --limit to reduce request size:

# Reduce limit to avoid rate limiting
autocli twitter search "topic" --limit 5

# Add delays between requests (if supported)
autocli twitter timeline --limit 10 --delay 1000

Extension Not Connecting

1. Ensure Chrome extension is enabled
2. Check extension has permissions for target sites
3. Verify autocli daemon is running
4. Restart browser and extension

Performance Tips

• Use --format json for processing with other tools
• Limit results with --limit for faster responses
• Public API commands are faster than browser commands
• Browser commands reuse logged-in sessions (no token refresh)
• Use --quiet flag to suppress progress output in scripts

Resources

• GitHub: https://github.com/nashsu/autocli
• AutoCLI.ai: https://autocli.ai (adapter marketplace)
• Documentation: Check autocli --help and subcommand help
• Issues: https://github.com/nashsu/autocli/issues