skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
octen-team/octen-skills69 installs

octen-extract

USE FOR extracting clean, LLM-ready content from one or more web page URLs, powered by Octen. Fetch 1-20 URLs in one call and get markdown/text content plus a page category, page-structure label, and (optionally) query-driven highlights. Use it for reading articles, scraping pages for RAG/grounding, summarization, or fact lookup from known URLs. It can also list the links on a page.

How do I install this agent skill?

npx skills add https://github.com/octen-team/octen-skills --skill octen-extract
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    This skill enables AI agents to read and process web page content through the Octen API. It uses standard cURL commands for network requests and provides clear, safe instructions for API key management across various agent environments. The identified indirect prompt injection surface is a characteristic inherent to the skill's primary function of reading external web content.

  • Socketpass

    No alerts

  • Snykwarn

    Risk: MEDIUM · 1 issue

What does this agent skill do?

Octen Extract

Turn one or more URLs into clean, LLM-ready content. Beyond the page body, each result also carries a category, a page-structure label, and — when you pass a query — query-relevant highlights instead of the full page.

Requires API Key: Get one at https://octen.ai · Set it: export OCTEN_API_KEY=your-api-key

API Key Setup

Before extracting, ensure OCTEN_API_KEY is set. On 401, stop and tell the user a key is required (https://octen.ai), help them configure it, then continue. Configure the key for the relevant agent (same as other Octen skills):

  • Claude Code — add { "env": { "OCTEN_API_KEY": "your-key" } } to ~/.claude/settings.json
  • Cursor / generic shell — export OCTEN_API_KEY="your-key" in ~/.zshrc or ~/.bashrc
  • Codex — ~/.codex/config.toml → [shell_environment_policy], set = { OCTEN_API_KEY = "your-key" }
  • OpenClaw — add OCTEN_API_KEY=your-key to ~/.openclaw/.env

Endpoint

POST https://api.octen.ai/extract

Authentication: X-Api-Key: <API_KEY> header · Content-Type: application/json

Quick Start (cURL)

Batch extract (full content)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://example.com", "https://octen.ai"],
    "format": "markdown"
  }'

Query-driven highlights (instead of full content)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://en.wikipedia.org/wiki/Python_(programming_language)"],
    "query": "async programming"
  }'

Hard site (advanced mode)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://www.reddit.com/r/LocalLLaMA/"],
    "mode": "advanced",
    "timeout": 60
  }'

Mixed batch, difficulty unknown (auto mode)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://example.com", "https://octen.ai", "https://news.ycombinator.com/item?id=1"],
    "mode": "auto",
    "timeout": 60
  }'

Page links (outbound references first)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://octen.ai"],
    "include_links": {"scope": "prefer_external", "max_links": 50}
  }'

Parameters

ParameterTypeRequiredDefaultDescription
urlsstring[]Yes-1–20 URLs to fetch in one batch
modestringNostandard (when omitted)standard, advanced, or auto — see Choosing a mode
querystringNo-When set, each result returns query-relevant highlights and OMITS full_content (max 500 chars)
max_age_secondsintegerNo86400Accept cached results within this age (min 300); older cached versions are re-fetched
formatstringNomarkdownContent format: markdown or text
timeoutintegerNo30Per-URL fetch budget in seconds (1–60)
include_imagesbooleanNofalseInclude image resources found on the page (also returns cover_image when the page has one)
include_videosbooleanNofalseInclude video URLs found on the page
include_audiobooleanNofalseInclude audio URLs found on the page
include_linksobjectNo-Use it to discover a site's pages or follow outbound references. Returns the page's links: {scope, max_links}. scope: prefer_internal (default; same-site links first) or prefer_external (other registered domains first); page order is kept within each group. max_links: 1–1000 (default 200; out of range → 400, not clamped). {} uses both defaults

Choosing a mode

You pick the mode; omitting it means standard.

  • standard (default): fastest and cheapest. Right for ordinary, well-structured pages without anti-bot protection — news, blogs, docs, product pages, static sites. On login-walled, JS-heavy, or anti-bot sites it can return empty content or just a page skeleton while still reporting status: "success".
  • advanced: highest success rate — renders in a real browser with stronger anti-bot handling; slower and 2.5x the price. Use it for known hard sites: anti-bot/WAF-protected sites, JS-heavy SPAs, social sites (Reddit, X, LinkedIn), academic sites (ResearchGate), dynamically loaded content.
  • auto: decides per URL — standard where it suffices, advanced only for hard URLs; each URL is billed at the mode it actually used. Use it for a mixed batch when you can't tell which URLs are hard. If most URLs are hard, its cost approaches all-advanced.
Scenariomode
News / blogs / docs / product pages (ordinary sites)standard (or omit)
Known hard sites (Reddit / X / LinkedIn / ResearchGate / anti-bot / SPA)advanced
Mixed batch, unsure which are hardauto
Cost-sensitive, mostly ordinary sitesstandard
Quality first, mostly hard sitesadvanced

Rule of thumb: all ordinary sites → standard (or omit); known hard sites → advanced; mixed or unsure → auto.

Recovery: if a standard result failed, or came back empty/skeletal (page_structure.primary is "No Main Content", or very short content on a page that should have a body), retry just those URLs with "mode": "advanced".

Timeout: advanced and auto are slower — raise timeout (e.g. 60).

Response Format

FieldTypeDescription
request_idstringUnique request identifier
data.results[]arrayOne result per URL
data.results[].urlstringThe URL
data.results[].statusstringsuccess or failed
data.results[].resolved_modestring?standard or advanced — the mode actually used for this URL. Present on success, absent on failure. May differ from the requested mode (auto picks per URL; advanced can resolve to standard)
data.results[].error_messagestring?Why it failed (when status = failed)
data.results[].titlestring?Page title
data.results[].full_contentstring?Cleaned page content (when no query)
data.results[].highlightsstring[]?Query-relevant snippets (when query is set)
data.results[].categoryobject?{primary, secondary} — what the page is about
data.results[].page_structureobject?{primary, secondary} — what kind of page it is
data.results[].time_publishedstring?Publish time, ISO 8601
data.results[].time_last_crawledstring?Last crawl time, ISO 8601
data.results[].faviconstring?Page favicon URL — returned by default when available
data.results[].cover_imageobject?{url} — page cover image (when include_images is true and the page has one)
data.results[].images / videos / audioobject[]?[{url}] — media found on the page (when the matching include_* is true)
data.results[].linksobject[]?[{url, anchor_text, is_external}] — page links (on success, when include_links is set and links were found)
meta.usage.total_urlsintegerURLs requested
meta.usage.successful_urlsintegerURLs successfully fetched
meta.usage.successful_by_modeobject{standard_urls, advanced_urls} — successful URLs per mode. Actual billing is authoritative from these counts, not from summing resolved_mode
meta.warningstringe.g. "1 URL(s) failed and were not billed"; empty string when there is nothing to report

meta is top-level (a sibling of data), not data.meta.

Error Codes

HTTP StatusDescription
400Missing or invalid parameter
401Invalid or missing API key
403Insufficient balance
429Rate limited
500Internal server error

Notes

  • Partial success is first-class: a failed URL is returned with status: "failed" + an error_message; sibling URLs in the same batch still succeed. Always check per-result status.
  • Pass query only when you want focused highlights for RAG/grounding; omit it to get the full cleaned page.
  • Pricing: standard $1 / 1K successful URLs, advanced $2.5 / 1K. Failed URLs are not billed.
  • category and page_structure let you route or filter pages (e.g., skip an index/operation page, keep a content page) without reading the whole body.

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/octen-team/octen-skills/octen-extract">View octen-extract on skillZs</a>