firecrawl-scraping

Firecrawl Scraping

Overview

Scrape individual web pages and convert them to clean, LLM-ready markdown. Handles JavaScript rendering, anti-bot protection, and dynamic content.

Quick Decision Tree

What are you scraping?
│
├── Single page (article, blog, docs)
│   └── references/single-page.md
│   └── Script: scripts/firecrawl_scrape.py
│
└── Entire website (multiple pages, crawling)
    └── references/website-crawler.md
    └── (Use Apify Website Content Crawler for multi-page)

Environment Setup

# Required in .env
FIRECRAWL_API_KEY=fc-your-api-key-here

Get your API key: https://firecrawl.dev/app/api-keys

Common Usage

Simple Scrape

python scripts/firecrawl_scrape.py "https://example.com/article"

With Options

python scripts/firecrawl_scrape.py "https://wsj.com/article" \
  --proxy stealth \
  --format markdown summary \
  --timeout 60000

Proxy Modes

Mode	Use Case
basic	Standard sites, fastest
stealth	Anti-bot protection, premium content (WSJ, NYT)
auto	Let Firecrawl decide (recommended)

Output Formats

• markdown - Clean markdown content (default)
• html - Raw HTML
• summary - AI-generated summary
• screenshot - Page screenshot
• links - All links on page

Cost

~1 credit per page. Stealth proxy may use additional credits.

Security Notes

Credential Handling

• Store FIRECRAWL_API_KEY in .env file (never commit to git)
• API keys can be regenerated at https://firecrawl.dev/app/api-keys
• Never log or print API keys in script output
• Use environment variables, not hardcoded values

Data Privacy

• Only scrapes publicly accessible web pages
• Scraped content is processed by Firecrawl servers temporarily
• Markdown output stored locally in .tmp/ directory
• Screenshots (if requested) are stored locally
• No persistent data retention by Firecrawl after request

Access Scopes

• API key provides full access to scraping features
• No granular permission scopes available
• Monitor usage via Firecrawl dashboard

Compliance Considerations

• Robots.txt: Firecrawl respects robots.txt by default
• Public Content Only: Only scrape publicly accessible pages
• Terms of Service: Respect target site ToS
• Rate Limiting: Built-in rate limiting prevents abuse
• Stealth Proxy: Use stealth mode only when necessary (paywalled news, not auth bypass)
• GDPR: Scraped content may contain PII - handle accordingly
• Copyright: Respect intellectual property rights of scraped content

Troubleshooting

Common Issues

Issue: Credits exhausted

Symptoms: API returns "insufficient credits" or quota exceeded error Cause: Account credits depleted Solution:

• Check credit balance at https://firecrawl.dev/app
• Upgrade plan or purchase additional credits
• Reduce scraping frequency
• Use basic proxy mode to conserve credits

Issue: Page not rendering correctly

Symptoms: Empty content or partial HTML returned Cause: JavaScript-heavy page not fully loading Solution:

• Enable JavaScript rendering with --js-render flag
• Increase timeout with --timeout 60000 (60 seconds)
• Try stealth proxy mode for protected sites
• Wait for specific elements with --wait-for selector

Issue: 403 Forbidden error

Symptoms: Script returns 403 status code Cause: Site blocking automated access Solution:

• Enable stealth proxy mode
• Add delay between requests
• Try at different times (some sites rate limit by time)
• Check if site requires login (not supported)

Issue: Empty markdown output

Symptoms: Scrape succeeds but markdown is empty or malformed Cause: Dynamic content loaded after page load, or unusual page structure Solution:

• Increase wait time for JavaScript to execute
• Use --wait-for to wait for specific content
• Try html format to see raw content
• Check if content is in an iframe (not always supported)

Issue: Timeout errors

Symptoms: Request times out before completion Cause: Slow page load or large page content Solution:

• Increase timeout value (up to 120000ms)
• Use basic proxy for faster response
• Target specific page sections if possible
• Check if site is experiencing issues

Resources

• references/single-page.md - Single page scraping details
• references/website-crawler.md - Multi-page website crawling

Integration Patterns

Scrape and Analyze

Skills: firecrawl-scraping → parallel-research Use case: Scrape competitor pages, then analyze content strategy Flow:

1. Scrape competitor website pages with Firecrawl
2. Convert to clean markdown
3. Use parallel-research to analyze positioning, messaging, features

Scrape and Document

Skills: firecrawl-scraping → content-generation Use case: Create summary documents from web research Flow:

1. Scrape multiple article pages on a topic
2. Combine markdown content
3. Generate summary document via content-generation

Scrape and Enrich CRM

Skills: firecrawl-scraping → attio-crm Use case: Enrich company records with website data Flow:

1. Scrape company website (about page, team page, product pages)
2. Extract key information (funding, team size, products)
3. Update company record in Attio CRM with enriched data