crawler

Crawler Skill

Converts any URL into clean markdown using a robust 3-tier fallback chain.

Quick start

uv run scripts/crawl.py --url https://example.com --output reports/example.md

Markdown is saved to the file specified by --output. Progress/errors go to stderr. Exit code 0 on success, 1 if all scrapers fail.

How it works

The script tries each tier in order and returns the first success:

Tier	Module	Requires
1	Firecrawl (firecrawl_scraper.py)	FIRECRAWL_API_KEY env var (optional; falls back if missing)
2	Jina Reader (jina_reader.py)	Nothing — free, no key needed
3	Scrapling (scrapling_scraper.py)	Local headless browser (auto-installs via pip)

File layout

crawler-skill/
├── SKILL.md            ← this file
├── scripts/
│   ├── crawl.py               ← main CLI entry point (PEP 723 inline deps)
│   └── src/
│       ├── domain_router.py       ← URL-to-tier routing rules
│       ├── firecrawl_scraper.py   ← Tier 1: Firecrawl API
│       ├── jina_reader.py         ← Tier 2: Jina r.jina.ai proxy
│       └── scrapling_scraper.py   ← Tier 3: local headless scraper
└── tests/
    └── test_crawl.py          ← 70 pytest tests (all passing)

Usage examples

# Basic fetch — tries Firecrawl, falls back to Jina, then Scrapling
# Always prefer using --output to avoid terminal encoding issues
uv run scripts/crawl.py --url https://docs.python.org/3/ --output reports/python_docs.md

# If no --output is provided, markdown goes to stdout (not recommended on Windows)
uv run scripts/crawl.py --url https://example.com

# With a Firecrawl API key for best results
FIRECRAWL_API_KEY=fc-... uv run scripts/crawl.py --url https://example.com --output reports/example.md

URL requirements

Only http:// and https:// URLs are accepted. Passing any other scheme (ftp://, file://, javascript:, a bare path, etc.) exits with code 1 and prints a clear error — no scraping is attempted.

Saving Reports

When the user asks to save the crawled content or a summary to a file, ALWAYS use the --output argument and save the file into the reports/ directory at the project root (for example, {project_root}/reports). If the directory does not exist, the script will create it.

Example: If asked to "save to result.md", you should run: uv run scripts/crawl.py --url <URL> --output reports/result.md

Point at a self-hosted Firecrawl instance

FIRECRAWL_API_URL=http://localhost:3002 uv run scripts/crawl.py --url https://example.com

Content validation

Each scraper validates its output before returning success:

• Minimum 100 characters of content (rejects empty/error pages)
• Detection of CAPTCHA / bot-verification pages (Firecrawl)
• Detection of Cloudflare interstitial pages (Scrapling — escalates to StealthyFetcher)
• Detection of Jina error page indicators (Error:, Access Denied, etc.)

Domain routing

Certain hostnames bypass one or more scraper tiers to avoid known compatibility issues. The logic lives in scripts/src/domain_router.py.

Domain	Skipped tiers	Active chain
medium.com (and subdomains)	firecrawl	jina → scrapling
mp.weixin.qq.com	firecrawl + jina	scrapling only
everything else	—	firecrawl → jina → scrapling

Sub-domain matching follows a suffix rule: blog.medium.com matches the medium.com rule because its hostname ends with .medium.com. An exact sub-domain like other.weixin.qq.com does not match mp.weixin.qq.com.

Running tests

uv run pytest tests/ -v

All 70 tests use mocking — no network calls, no API keys required.

Dependencies (auto-installed by uv run)

• firecrawl-py>=2.0 — Firecrawl Python SDK
• httpx>=0.27 — HTTP client for Jina Reader
• scrapling>=0.2 — Headless scraping with stealth support
• html2text>=2024.2.26 — HTML-to-markdown conversion

When to invoke this skill

Invoke crawl.py whenever you need the text content of a web page:

result = subprocess.run(
    ["uv", "run", "scripts/crawl.py", "--url", url],
    capture_output=True, text=True
)
if result.returncode == 0:
    markdown = result.stdout

Or simply run it directly from the terminal as shown in Quick start above.