scrapling
Scrape pages locally with anti-bot bypass, TLS impersonation, and adaptive element tracking — no API keys, no cloud. Handles Cloudflare protection, CSS/XPath element extraction, and survives site redesigns. Complements firecrawl (cloud) with 100% local execution. Triggers on Cloudflare bypass, anti-bot scraping, stealth fetch, local scraping, Scrapling.
How do I install this agent skill?
npx skills add https://github.com/tdimino/claude-code-minoan --skill scraplingIs this agent skill safe to install?
- Gen Agent Trust Hubpass
Scrapling provides local web scraping with anti-bot bypass capabilities. The analysis identified potential security risks including the disabling of SSL certificate verification in the fetcher wrapper and the exposure to indirect prompt injection through the processing of untrusted web content.
- Socketwarn
1 alert: gptAnomaly
- Snykwarn
Risk: MEDIUM · 1 issue
- Runlayerwarn
6/6 files flagged
- ZeroLeakspass
Score: 93/100 · 2 sections analyzed
What does this agent skill do?
Scrapling -- Local Stealth Web Scraping
100% local Python library (BSD-3, D4Vinci/Scrapling). No API keys, no cloud dependencies. Built-in Cloudflare solver, TLS impersonation, and adaptive element tracking.
When to Use Scrapling vs Firecrawl
| Need | Use | Why |
|---|---|---|
| Clean markdown from a URL | firecrawl scrape --only-main-content | Optimized for LLM markdown conversion |
| Bypass Cloudflare/anti-bot | scrapling stealth fetch | Built-in Turnstile solver, Patchright stealth |
| Extract specific elements | scrapling with CSS/XPath selectors | Element-level precision, adaptive tracking |
| No API key available | scrapling | 100% local, zero credentials |
| Batch cloud scraping | firecrawl crawl / batch-scrape | Cloud infrastructure, parallel processing |
| Site redesign resilience | scrapling adaptive mode | SQLite-backed similarity matching |
| Full-site concurrent crawl | scrapling Spider framework | Scrapy-like with pause/resume |
| Web search + scrape | firecrawl search --scrape | Combined search + extraction |
Installation
Run once to set up Scrapling with all features:
~/.claude/skills/scrapling/scripts/scrapling_install.sh
Installs scrapling[all] via uv, downloads Chromium + system dependencies, and verifies all fetchers load.
Quick Start -- Stdout Wrapper
The wrapper uses Scrapling's Python API directly (faster than CLI, avoids curl_cffi cert issues) and outputs to stdout for piping into filter_web_results.py:
# Basic HTTP fetch (fastest, TLS impersonation)
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://example.com
# Stealth mode (Patchright, anti-bot bypass)
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://protected.site --stealth
# Stealth + Cloudflare solver
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://cf-protected.site --stealth --solve-cloudflare
# Dynamic (Playwright Chromium, JS rendering)
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://js-heavy.site --dynamic
# With CSS selector for targeted extraction
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://example.com --css ".product-list"
# Pipe through firecrawl's filter for token efficiency
python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py https://example.com --stealth | \
python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --sections "Pricing" --max-chars 5000
Full path: python3 ~/.claude/skills/scrapling/scripts/scrapling_fetch.py
Flags: --stealth, --dynamic, --css SELECTOR, --solve-cloudflare, --impersonate BROWSER, --format {text,html}, --no-headless, --timeout SECONDS
For --network-idle, --real-chrome, or POST requests, use the CLI direct path below.
Quick Start -- CLI Direct
For file-based output (Scrapling's native CLI):
# HTTP fetch -> markdown
scrapling extract get 'https://example.com' content.md
# HTTP with CSS selector and browser impersonation
scrapling extract get 'https://example.com' content.md --css-selector '.main-content' --impersonate chrome
# Dynamic fetch (Playwright, JS rendering)
scrapling extract fetch 'https://example.com' content.md
# Stealth with Cloudflare bypass
scrapling extract stealthy-fetch 'https://protected.site' content.md --solve-cloudflare
# Stealth with CSS selector, visible browser for debugging
scrapling extract stealthy-fetch 'https://site.com' content.md --css-selector '.data' --no-headless
Three Fetcher Tiers
| CLI verb | Engine | Stealth | JS | Speed |
|---|---|---|---|---|
get | curl_cffi (HTTP) | TLS impersonation | No | Fast |
fetch | Playwright/Chromium | Medium | Yes | Medium |
stealthy-fetch | Patchright/Chrome | Maximum | Yes | Slower |
Python API
For element-level extraction, automation, or when the CLI is insufficient:
from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher
# Simple HTTP fetch with CSS extraction
page = Fetcher.get('https://example.com', impersonate='chrome')
titles = page.css('.item h2::text').getall()
links = page.css('a::attr(href)').getall()
# Stealth with Cloudflare bypass
page = StealthyFetcher.fetch('https://protected.site',
headless=True, solve_cloudflare=True,
hide_canvas=True, block_webrtc=True)
data = page.css('.content').get_all_text()
# Page automation (login, click, fill)
def login(page):
page.fill('#username', 'user')
page.fill('#password', 'pass')
page.click('#submit')
page = StealthyFetcher.fetch('https://app.example.com', page_action=login)
Parsing (no fetching)
from scrapling.parser import Selector
page = Selector("<html>...</html>")
page.css('.item::text').getall() # CSS with pseudo-elements
page.xpath('//div[@class="item"]') # XPath
page.find_all('div', class_='item') # BeautifulSoup-style
page.find_by_text('Add to Cart') # Text search
page.find_by_regex(r'Price: \$\d+') # Regex search
Adaptive Scraping
Scrapling's signature feature. Elements are fingerprinted to SQLite and relocated by similarity scoring after site redesigns.
from scrapling.fetchers import Fetcher
Fetcher.adaptive = True
page = Fetcher.get('https://example.com')
# First run: save element fingerprint
products = page.css('.product-list', auto_save=True)
# Later, after site redesign breaks the selector:
products = page.css('.product-list', adaptive=True) # Still finds it
Read: references/adaptive-scraping.md
Spider Framework
For full-site crawling with concurrency, pause/resume, and session routing:
from scrapling.spiders import Spider, Response, Request
from scrapling.fetchers import FetcherSession, StealthySession
class ResearchSpider(Spider):
name = "research"
start_urls = ["https://example.com/"]
concurrent_requests = 10
def configure_sessions(self, manager):
manager.add("fast", FetcherSession(impersonate="chrome"))
manager.add("stealth", StealthySession(headless=True), lazy=True)
async def parse(self, response: Response):
for link in response.css('a::attr(href)').getall():
if "protected" in link:
yield Request(link, sid="stealth")
else:
yield Request(link, sid="fast")
for item in response.css('.article'):
yield {"title": item.css('h2::text').get(), "url": response.url}
result = ResearchSpider().start(crawldir="./data") # Pause/resume enabled
result.items.to_json("output.json")
Troubleshooting
- Browser not found: Run
scrapling installto download browser dependencies - Import error on fetchers: Install the full package:
uv pip install "scrapling[all]" - Cloudflare still blocking: Combine flags:
--solve-cloudflare --block-webrtc --hide-canvas - SSL cert error with HTTP fetcher: curl_cffi's bundled CA certs can be stale on macOS/pyenv. The wrapper handles this automatically (
verify=False). For Python API, passverify=FalsetoFetcher.get(). - Slow stealthy fetch: Expected--browser automation is inherently slower than HTTP requests. Use
get(HTTP) when stealth is not needed.
Reference Documentation
| File | Contents |
|---|---|
references/cli-reference.md | Full CLI extract command reference (get, post, fetch, stealthy-fetch) |
references/python-api-reference.md | Python API (Fetcher, DynamicFetcher, StealthyFetcher, Sessions, Spiders) |
references/adaptive-scraping.md | Adaptive element tracking deep-dive (save, match, similarity scoring) |
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/tdimino/claude-code-minoan/scrapling">View scrapling on skillZs</a>