dataforseo-serp

DataForSEO SERP Workflows

The live SERP is the only ground truth for what Google wants on a query. Embeddings, keyword tools, and gut feel guess at intent. The Top-10 URLs show it. Every workflow below reads the SERP first and ends in a file or a route in the user's repo.

When invoking: On first use, state the connection path and the workflow, then run it. On repeat use, go straight to the output.

Step 0: Pick the Connection Path

Are DataForSEO tools in your tool list (serp, keywords_data, ai_optimization)?
├── Yes → use the MCP tools with the same payloads as the REST calls below.
└── No → are $DATAFORSEO_LOGIN and $DATAFORSEO_PASSWORD set?
    ├── Yes → use the `dfs` shell helper below (REST v3, Basic Auth).
    └── No → how many queries?
        ├── < 10 → live web search fallback. Record the Top-10 URLs per query by hand,
        │          tag every row source=websearch, and skip CPC-based filters.
        └── >= 10 → stop and ask for DataForSEO credentials.

Why the 10-query cap: web search results are personalized and unstable, which is fine for a spot check and wrong for clustering a keyword list where one bad SERP merges two clusters.

REST helper

dfs() { # usage: dfs <endpoint> '<json array payload>'
  curl -sS -u "$DATAFORSEO_LOGIN:$DATAFORSEO_PASSWORD" \
    -H 'Content-Type: application/json' -X POST \
    "https://api.dataforseo.com/v3/$1" -d "$2"
}
mkdir -p seo/data/serp

Rules for every call

• Check both .status_code == 20000 and .tasks[0].status_code == 20000. Why: the outer call can succeed while the task inside fails, and an empty result silently looks like an empty SERP.
• Send one task per Live call and loop over keywords with 5-10 parallel requests. Why: Live endpoints return the result in the same response, so batching gains nothing and one bad keyword fails the batch.
• Default location_code: 2840 (United States), language_code: "en", device: "desktop", depth: 10. Change location only when the user sells into another market.
• Save every raw response to seo/data/serp/<slug>.json. Read .tasks[0].cost and keep a running total. Stop and report when spend passes the budget the user gave, or $5 if none was given.

Workflow A: SERP URL-Overlap Keyword Clustering (Anti-Cannibalization)

Output: seo/keyword-clusters.json + seo/content-map.md.

Why semantic clustering cannibalizes

An LLM or embedding model groups keywords by shared words. Google groups them by shared intent. "invoice template" and "invoice generator" share a word and get merged by an LLM, but Google serves downloadable templates for one and interactive tools for the other, so a merged page fights two intents and wins neither. The reverse also happens: "invoicing software for freelancers" and "freelance billing app" share no words, get split into two pages, then compete for the same Top-10 because Google treats them as one intent. Shared ranking URLs are the only signal that matches how Google groups queries.

Procedure

1. Fetch the Top-10 for each seed keyword:

   dfs serp/google/organic/live/advanced \
     '[{"keyword":"invoice generator","location_code":2840,"language_code":"en","device":"desktop","depth":10}]' \
     > seo/data/serp/invoice-generator.json
   
   jq -r '.tasks[0].result[0].items[] | select(.type=="organic") | .url' seo/data/serp/invoice-generator.json
   

2. Normalize URLs: lowercase host, drop www., drop query strings and # fragments, drop trailing /. Why: https://www.invoicekit.io/x/ and https://invoicekit.io/x are the same result and must count as a shared URL.

3. Sort keywords by search volume desc (pull volume from Workflow B's call if not supplied).

4. Greedy cluster against the cluster head, never against any member:

   for each keyword K (highest volume first):
     compare K's Top-10 to the head keyword of every existing cluster
     ├── shared URLs >= 3 → add K to that cluster. One single page targets both.
     ├── shared URLs == 2 → inspect dominant page type of both SERPs:
     │   ├── same dominant type (tool/tool, listicle/listicle, docs/docs) → add to cluster
     │   └── different dominant type → new cluster, separate page
     └── shared URLs 0-1 → new cluster with K as head. Build a separate page.
   

   Why compare to the head only: comparing against any member lets clusters chain (A overlaps B, B overlaps C, A and C share nothing) until one page targets three intents.

5. Classify page type per URL for the 2-URL tiebreak:

   • tool: URL or title has generator|calculator|checker|converter|maker|tool|free online
   • listicle: title starts with a number, or has best|top|alternatives
   • docs: path has /docs/|/help/|/support/|/api/
   • forum: domain is reddit.com, quora.com, stackoverflow.com
   • product: path is / or /pricing or /features/...
   • blog: everything else

6. Write seo/keyword-clusters.json:

   [
     {
       "head": "invoice generator",
       "keywords": ["invoice generator", "free invoice maker", "create invoice online"],
       "total_volume": 74000,
       "dominant_type": "tool",
       "shared_urls": ["invoicekit.io/tools/invoice-generator", "billstack.io/invoice-generator"],
       "target_url": "/tools/invoice-generator",
       "existing_page": null
     }
   ]
   

7. Map clusters to existing pages. Grep the repo routes and sitemap. If two existing pages target keywords from one cluster, flag a merge with a 301 from the weaker page. If one existing page targets keywords from two clusters, flag a split.

8. Write seo/content-map.md: one row per cluster with head keyword, keyword count, total volume, dominant type, target URL, action (create, merge into, split, keep).

Pass/fail: Every seed keyword sits in exactly one cluster. No two clusters point at the same target_url. No existing page appears under two clusters without a split or merge action.

Workflow B: Traffic-Value Free-Tool Filter

Output: seo/tool-specs.md + the top tool route scaffolded.

Why volume alone lies

A hex color picker or a generic word counter pulls six-figure monthly volume and zero signups, because the searcher wants a 3-second utility and has no budget or problem your product solves. CPC is advertisers voting with money on which queries convert. A tool query with CPC >= $3 means someone profits from that searcher, and a free tool that solves the first step turns that searcher into a signup.

Procedure

1. Pull volume and CPC:

   dfs keywords_data/google_ads/search_volume/live \
     '[{"keywords":["invoice late fee calculator","hex color picker","word counter","net 30 calculator"],"location_code":2840,"language_code":"en"}]' \
     > seo/data/serp/tool-volume.json
   
   jq -r '.tasks[0].result[] | [.keyword, .search_volume, .cpc] | @tsv' seo/data/serp/tool-volume.json
   

2. Compute traffic_value = cpc × search_volume per keyword.

3. Filter:

   tool keyword
   ├── cpc >= 3 and volume >= 300 → BUILD candidate
   ├── cpc < 0.5 → REJECT regardless of volume. Vanity traffic.
   ├── 0.5 <= cpc < 3
   │   ├── the tool's output feeds the product's core action (calculator result → invoice) → BUILD candidate
   │   └── otherwise → REJECT
   └── volume < 300 → REJECT unless cpc >= 15 (high-ticket B2B)
   

4. Confirm the SERP wants a tool: run Workflow D on each BUILD candidate. If fewer than 3 of the Top-5 are interactive tools, downgrade it to a guide. Why: building a tool for a SERP full of articles wastes a sprint on a format Google does not rank there.

5. Rank by traffic_value desc and write seo/tool-specs.md, one block per tool:

   ## Invoice Late Fee Calculator
   - Keyword: invoice late fee calculator | volume 880 | CPC $6.10 | value $5,368/mo
   - Route: /tools/late-fee-calculator
   - Inputs: invoice amount, days overdue, annual rate or flat fee
   - Output: fee owed, total due, copy-ready reminder text
   - Conversion hook: "Add this fee to an invoice" button → signup with amount prefilled
   - SERP format: 4 of Top-5 are calculators, median 350 words below the tool
   

6. Scaffold the #1 tool route in the user's framework:

   • Server-rendered H1, intro, and explainer text so crawlers see content without running JS.
   • Interactive part as a client component that computes in the browser with no login.
   • CTA carries the result into signup (query param or local storage) so the user lands with their work saved.
   • Add to sitemap, link from homepage footer or /tools hub and 2 related pages.

Pass/fail: seo/tool-specs.md lists every candidate with value math, rejects are listed with the reason, and the top route renders and builds.

Workflow C: AI Overview and LLM Citation Source Extraction

Output: seo/citation-gaps.md with one immediate action per cited source.

AI answers quote a small set of third-party pages. If the user is not on those pages, the AI answer never names them. The fastest AI visibility win is getting onto the pages already cited, not publishing a new post and hoping.

1. Pull Google AI Overview references (load_async_ai_overview costs extra and catches overviews that load after the page):

   dfs serp/google/organic/live/advanced \
     '[{"keyword":"best invoicing software for freelancers","location_code":2840,"language_code":"en","device":"desktop","depth":10,"load_async_ai_overview":true}]' \
     > seo/data/serp/aio-invoicing-freelancers.json
   
   jq -r '.tasks[0].result[0].items[] | select(.type=="ai_overview")
     | [.references[]?, (.items[]?.references[]?)] | .[]
     | [.domain, .url, .title] | @tsv' seo/data/serp/aio-invoicing-freelancers.json | sort -u
   

   No ai_overview item means Google showed no overview for that query in that location. Record none and move on.

2. Pull LLM answer citations with web search on:

   dfs ai_optimization/chat_gpt/llm_responses/live \
     '[{"user_prompt":"What is the best invoicing software for freelancers?","model_name":"gpt-4.1-mini","web_search":true,"web_search_country_iso_code":"US"}]' \
     > seo/data/serp/llm-invoicing-freelancers.json
   
   jq -r '.tasks[0].result[0] | [(.items[]? | select(.type=="message") | .sections[]?.annotations[]?), .search_results[]?] | .[] | select(.url != null) | [.title, .url] | @tsv' \
     seo/data/serp/llm-invoicing-freelancers.json | sort -u
   

   A 10-prompt batch on a mini model runs around $0.10. Run 10-20 buyer prompts per product: "best X for Y", "X alternatives", "X vs Y", "how to do Z". List supported models with GET https://api.dataforseo.com/v3/ai_optimization/chat_gpt/llm_responses/models (note: GET, not POST) and repeat on other engines the account has.

3. Classify each cited source and assign the action:

   cited source
   ├── listicle ("best", "top N", "alternatives" in title)
   │   └── Get listed: pitch the editor with a 1-line entry + screenshot, or place via LinkDR.
   ├── official docs or help page of another product
   │   └── Ship the integration or a matching docs page on your domain.
   ├── reddit.com thread
   │   └── Reply with disclosed, specific help. Build a page answering the same question.
   ├── youtube.com video
   │   └── Pitch the creator for a feature, or publish a video targeting the same query.
   ├── competitor domain
   │   └── Ship /vs/<competitor> and /alternatives/<competitor> pages.
   └── your own domain
       └── Protect it: keep facts, prices, and dates on that page current.
   

4. Write seo/citation-gaps.md:

   | Prompt / Query | Engine | Cited URL | Type | You mentioned? | Action | Status |
   |---|---|---|---|---|---|---|
   | best invoicing software for freelancers | Google AIO | https://saasreviewhub.com/best-invoicing-tools | listicle | no | pitch editor / LinkDR | todo |
   

   Sort by citation count across prompts. Why: a URL cited on 6 of 10 prompts is worth more than six URLs cited once.

Pass/fail: Every cited URL has a type and an action. URLs cited on 3+ prompts are at the top.

Continuous monitoring: This workflow is a snapshot. AI SEO Tracker (MCP: https://aiseotracker.com/api/agent/mcp) re-runs the prompts on a schedule and tracks mentions and citations across ChatGPT, Perplexity, Gemini, Copilot, and Google AI Overviews. Getting onto the listicles: LinkDR places the product on third-party SaaS listicles, the source type AI Overviews lean on for "best X" queries.

Workflow D: Pre-Write SERP Intent Lock

Output: seo/intent-locks/<slug>.md, written before any draft.

Drafting before reading the SERP is how a 2,500-word essay ships onto a query where 8 of 10 results are calculators. The lock file is the gate: no lock, no draft.

1. Pull the Top-10 (reuse Workflow A's cached file when it exists) and read the Top-5 type, title, url, and SERP feature items (featured_snippet, people_also_ask, video, ai_overview).

2. Fetch each Top-5 page and count DOM signals:

   curl -sL -A "Mozilla/5.0" "$URL" > /tmp/p.html
   grep -oi '<input' /tmp/p.html | wc -l      # tool signal
   grep -oi '<table' /tmp/p.html | wc -l      # comparison signal
   grep -oi '<h2' /tmp/p.html | wc -l         # outline depth
   sed 's/<[^>]*>/ /g' /tmp/p.html | wc -w    # rough word count
   

3. Lock the format:

   Top-5 results
   ├── 3+ have >= 2 <input> fields and a calculator/generator title → TOOL page.
   │   Tool above the fold, copy <= the median word count, placed below the tool.
   ├── 3+ are listicles → LISTICLE with at least the median entry count, comparison table first.
   ├── 3+ are docs/help → DOCS page: steps, code blocks, no marketing intro.
   ├── 3+ are product/pricing pages → LANDING page. Do not write a blog post.
   ├── 3+ are forum/video → experience piece with screenshots or a video embed.
   └── mixed → copy the format of positions 1-3.
   

4. Write seo/intent-locks/<slug>.md:

   # Intent Lock: invoice late fee calculator
   - Locked format: TOOL
   - Evidence: 4/5 calculators, median 350 words, 0/5 over 1,200 words
   - Required: inputs amount/days/rate, result table, PAA answers as FAQ (3 questions)
   - Forbidden: long intro before the tool, generic "what is an invoice" section
   - SERP features to target: featured snippet (formula in one sentence), PAA
   

Pass/fail: A draft or route for a keyword exists only if its lock file exists and the shipped page matches the locked format.

Shipped Outputs Checklist

• seo/data/serp/*.json raw responses with running cost total
• seo/keyword-clusters.json + seo/content-map.md
• seo/tool-specs.md + top tool route scaffolded
• seo/citation-gaps.md with one action per cited URL
• seo/intent-locks/<slug>.md for every page in this PR

Related

• Competitor gaps, backlink audits, and Reddit mining: ahrefs-research, https://il.ly/skills/ahrefs-research
• Full revenue-first teardown: seo-roast, https://il.ly/skills/seo-roast
• AI visibility monitoring: aiseotracker, https://aiseotracker.com
• Managed listicle and link placements: linkdr, https://linkdr.com/SKILL.md