skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
bgauryy/octocode76 installs

octocode-scraping

Use when extracting or mapping public web content into a local cited corpus: scrape or crawl a URL/docs site, pull tables/pricing/product fields, diagnose blocked or thin pages, or answer from saved pages. Phrases like scrape this URL, crawl the docs, build a corpus, extract pricing. Prefer keyless fetch; ask before hosted spend. Live clicks/HAR/perf → octocode-chrome-devtools.

How do I install this agent skill?

npx skills add https://github.com/bgauryy/octocode --skill octocode-scraping
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    The skill is a comprehensive web scraping and crawling utility. It emphasizes security and privacy through automated redaction of sensitive data (cookies, auth headers, tokens) and includes explicit policies for handling untrusted content. While it possesses powerful capabilities like local script execution and broad web ingestion, these are architected with developer-focused controls and safety warnings.

  • Socketwarn

    2 alerts: gptAnomaly

  • Snykwarn

    Risk: MEDIUM · 1 issue

What does this agent skill do?

Octocode Scraping

Flow: FRAME → POLICY → ROUTE → FETCH → CORPUS → SEARCH → CITE → RECOVER.

FRAME before the first fetch: fix target URL/domain, goal, depth, and output shape — vague ask → references/user-inputs.md.

Defaults: one public URL, --mode html, omit --provider (keyless cdp→direct), session .octocode/tmp/scrape/{sessionId}, compact stdout. Search corpus before refetch. Live interaction → chrome-devtools on one port, then har-ingest + corpus-run into the same session. Ask before auth, hosted spend, crawl widen, CAPTCHA/MFA, destructive actions. Cite paths + URL metadata — not raw dumps.

Stop when: two same-class failures (report evidence, route tried, sanitized status, next approval); hosted 403 (wrong key or credits gone — status only, no retry); CAPTCHA/MFA, auth wall, or cookie/profile transfer needed; still blocked after one cdp try (ask before --provider scrapingant); personal data, form submits, purchases, sends, deletes, or account changes in scope; the saved corpus already proves the claim (cite it, do not refetch); crawl widen before reports/summary.md is useful. Recovery table: references/failure-recovery.md.

Route (pick one)

NeedDoSkip
Vague scrape--mode html, omit --providermarkdown / auto hosted
See auto pickscripts/provider-check.mjsguessing
Fetch/crawlscripts/fetch.mjsdeprecated scrapingant-fetch
Corpus on diskcorpus-inspect → find helpers → corpus-runblind refetch
Live click/DOM/authchrome-devtools--provider cdp alone for interaction
Page healthchrome measure + measure-queryhosted scrape for scores
CDP → corpusscripts/har-ingest.mjs --from-cdp-dirnew sessionId
Prove fieldscripts/corpus-run.mjs --regex|--scriptreopen browser
Still blockedevidence + ask → --provider scrapingantsilent spend

Scripts (Node, no install; every one takes --help)

WhenRun
fetch / crawl / extract a URL — the owner of every network callscripts/fetch.mjs --url <u> [--mode html] [--crawl --same-domain --max-pages <n>] [--no-raw]
want the fetch plus an immediate corpus brief in one shotscripts/fetch-and-brief.mjs --url <u> (wraps fetch.mjs → corpus-inspect.mjs)
before routing or hosted spend: which provider auto-wins, credits leftscripts/provider-check.mjs [--provider <p>], scripts/provider-usage.mjs — sanitized, never prints the key
read a saved session first, then search itscripts/corpus-inspect.mjs --session-dir <d> [--page <n>], then scripts/corpus-find.mjs --session-dir <d> --query <t>
pull static DOM, assets, or graph paths out of the corpus (live DOM → chrome-devtools)scripts/dom-find.mjs --kind form/button/table, scripts/resource-list.mjs --kind asset/external, scripts/graph-navigate.mjs --from <nodeId> — each with --session-dir <d>
prove one field locally instead of reopening a browserscripts/corpus-run.mjs --session-dir <d> --roots cdp,extracts --regex <re> (or --script <file>, --concat-parts)
bridge chrome-devtools artifacts into this session, or hand a packet backscripts/har-ingest.mjs --session-dir <d> --from-cdp-dir <run>; reverse with --export-packet
need field names before an extractionscripts/schema-helper.mjs --intent "extract pricing and features"
an old transcript uses the legacy namesdeprecated shims scripts/scrapingant-fetch.mjs, scripts/scrapingant-check.mjs, scripts/scrapingant-usage.mjs forward to fetch.mjs / provider-check.mjs / provider-usage.mjs — call the new names
editing or extending a scriptshared modules in scripts/lib/ (providers registry, client fetch, corpus, analyzers, extractors, text, args, bridge); env/key resolution vendored in scripts/octocode-config.mjs; JSON contracts in scripts/schemas/

Full flags, roles, and the library map: read scripts/README.md before adding a flag or a vendor.

References

WhenLoad
frame a vague scopereferences/user-inputs.md
legal/safety/privacy policyreferences/scraping-policy.md
cost / keyless vs hosted routereferences/route-selection.md
provider registry / add a vendorreferences/providers.md
hosted API, after approvalreferences/scrapingant.md
search a saved session corpusreferences/session-corpus.md
site graph / navigation / workflowsreferences/website-analysis.md
stdout and corpus file shapesreferences/data-contract.md
extract quality and citationsreferences/extraction-quality.md
live browser bridge playbookreferences/browser-scraping.md
blocked, thin, or oversized resultsreferences/failure-recovery.md

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/bgauryy/octocode/octocode-scraping">View octocode-scraping on skillZs</a>