our-claude-skills/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/CLAUDE.md

Web Crawler Orchestrator

Orchestrates web crawling with intelligent backend selection. Primary backend is the OurSEO async crawler (rate-limited, resumable, metadata-rich). Falls back to Firecrawl MCP for JS-rendered sites or WebFetch for simple pages.

Trigger Keywords

"crawl URLs", "fetch documents", "scrape pages", "download references"

Intelligent Crawler Selection

# Get crawler recommendation for a URL
uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.com"

Crawler	Best For	Auto-Selected When
SEO aiohttp (default)	Documentation sites, sitemaps	docs/developer/api domains, has sitemap
Firecrawl MCP	Dynamic sites, SPAs	React/Vue/Angular, JS-rendered
WebFetch	Quick single pages	Fallback, simple pages
Node.js	Small docs sites	≤50 pages, static content
Scrapy	Enterprise crawls	>200 pages, multi-domain

Workflow

Step 1: Analyze Target Site

Run select-crawler to determine site characteristics and get a recommendation.

Step 2a: Execute Crawl — SEO aiohttp (Recommended)

The SEO crawler adapter provides rate limiting, progress tracking, resume support, and full metadata extraction.

# Crawl specific URLs
uv run python scripts/seo_crawler_adapter.py crawl \
  --urls "https://docs.example.com/page1,https://docs.example.com/page2" \
  --output ~/Documents/reference-library/topic-slug/raw/

# Crawl from XML sitemap (auto-discovers URLs)
uv run python scripts/seo_crawler_adapter.py crawl-sitemap \
  --sitemap https://docs.example.com/sitemap.xml \
  --output ~/Documents/reference-library/topic-slug/raw/ \
  --max-pages 50

# Discover URLs from root page (follows internal links)
uv run python scripts/seo_crawler_adapter.py discover \
  --url https://docs.example.com \
  --output ~/Documents/reference-library/topic-slug/raw/ \
  --max-depth 2 --max-pages 50 --same-domain

# Crawl from reference-curator manifest
uv run python scripts/seo_crawler_adapter.py crawl-manifest \
  --manifest ./manifest.json \
  --output ~/Documents/reference-library/topic-slug/raw/

# Check crawl progress
uv run python scripts/seo_crawler_adapter.py status

SEO crawler capabilities:

• Token-bucket rate limiting (via BaseAsyncClient)
• Semaphore-controlled concurrency (default: 5 concurrent)
• Tenacity retry with exponential backoff
• Full PageMetadata extraction: title, headings, links, schema, OG
• Progress tracking with ETA and resume files
• Sitemap XML parsing (including sitemap indexes)
• Internal link discovery (breadth-first, depth-limited)
• Direct .raw.md output with enriched YAML frontmatter

Step 2b: Execute Crawl — Firecrawl MCP (for JS-rendered sites)

Use Firecrawl MCP tools directly when JS rendering is required:

firecrawl_map(url, limit=100)           # Discover URLs
firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
firecrawl_crawl(url, max_depth=2, limit=50)

Step 2c: Execute Crawl — WebFetch (fallback)

For quick single-page fetches when no crawl infrastructure is needed:

WebFetch(url, prompt="Return the COMPLETE page content as clean markdown.")

Step 3: Store Crawl Results

# Store crawled files and create result manifest
uv run python scripts/crawl_mgr.py store-result \
  --raw-dir ~/Documents/reference-library/raw/ \
  --crawler seo-aiohttp \
  --source-id 1 \
  --output crawl_result.json

# List recent crawls
uv run python scripts/crawl_mgr.py list-crawls --status completed

Step 4: Write Raw Files (when --save-raw or --no-distill)

When the pipeline is invoked with --save-raw (or --no-distill, which implies it), write each crawled page's markdown verbatim to the raw output directory.

Note: The SEO crawler adapter (seo_crawler_adapter.py) writes .raw.md files directly with enriched YAML frontmatter — this step is automatic when using seo-aiohttp.

File naming: {NN}-{page-slug}.raw.md — zero-padded index matching crawl order.

Output path:

• --save-raw: {output}/{topic-slug}/raw/
• --no-distill: {output}/{topic-slug}/ (raw files are primary)

YAML frontmatter (SEO crawler enriched):

---
source_url: {original_url}
crawled_at: {ISO 8601 timestamp}
crawler: seo-aiohttp
content_length: {character count}
raw: true
status_code: {HTTP status}
title: {page title}
word_count: {word count}
internal_links: {count}
external_links: {count}
h1: {h1 text}
schema_types: [{types found}]
response_time_ms: {ms}
---

After all pages are written, a combined raw bundle ({topic-slug}-raw-complete.md) is auto-generated by concatenating all .raw.md files with --- separators.

Rate Limiting

All crawlers respect these limits:

• 20 requests/minute (SEO crawler: configurable via BaseAsyncClient)
• 3-5 concurrent requests (SEO crawler: semaphore-controlled)
• Exponential backoff on 429/5xx (SEO crawler: tenacity)
• Use --delay flag for polite crawling (recommended: 2-3s for Google properties)

Error Handling

Error	Action
Timeout	Retry with exponential backoff (max 3 attempts)
Rate limit (429)	Token-bucket + exponential backoff
Not found (404)	Log and skip

⚠️ GOTCHA: IP Block / CAPTCHA Detection & Wayback Fallback

Problem: Large-volume crawls on sites like support.google.com trigger IP-level bot detection. Google redirects to google.com/sorry/ with a CAPTCHA. Once triggered:

• Direct requests/curl returns 429 indefinitely (not a temporary rate limit)
• Playwright headless/headful also gets blocked (Google detects automation)
• The block persists even after the user solves the CAPTCHA in their browser (cookie-based, not IP-based lift)
• WebFetch and read_webpage MCP tools also fail (same IP or server-side block)

Detection: Check for these signals in crawl responses:

1. HTTP 302 redirect to google.com/sorry/index?continue=...
2. Response body contains solveSimpleChallenge or unusual traffic
3. Persistent 429 after retries with exponential backoff (>3 consecutive 429s)

Solution: Wayback Machine Detour

When IP block / CAPTCHA is detected, automatically fall back to Wayback Machine:

# Instead of: https://support.google.com/analytics/answer/9304153
# Fetch from: https://web.archive.org/web/2024/https://support.google.com/analytics/answer/9304153

The SEO crawler adapter supports this with the --wayback-fallback flag:

uv run python scripts/seo_crawler_adapter.py crawl \
  --urls "@urls.txt" \
  --output ~/Documents/reference-library/topic/raw/ \
  --delay 1.5 \
  --wayback-fallback

How it works:

1. First attempts direct fetch for each URL
2. On 429 or CAPTCHA redirect, flags captcha_detected
3. After 3 consecutive CAPTCHA blocks, switches ALL remaining URLs to Wayback prefix
4. Wayback URLs use https://web.archive.org/web/2024/{original_url}
5. Frontmatter records source: wayback for traceability

Limitations:

• Wayback snapshots may be weeks/months old (acceptable for documentation/help pages)
• Not all pages are archived (rare for major sites like Google, MDN, etc.)
• Wayback has its own rate limits (gentler — 1.5s delay is sufficient)
• Links in content point to Wayback URLs — the extractor strips these automatically

When to use --delay vs --wayback-fallback:

• Start with --delay 3.0 for large Google crawls (preventive)
• If 429s still occur, add --wayback-fallback (reactive)
• For non-Google sites, --delay 2.0 is usually sufficient | Access denied (403) | Log, mark as failed | | JS rendering needed | Switch to Firecrawl MCP | | Connection error | Retry with backoff, then skip |

Scripts

Command	Purpose
seo_crawler_adapter.py crawl	Crawl URLs with SEO metadata extraction
seo_crawler_adapter.py crawl-sitemap	Discover and crawl from XML sitemap
seo_crawler_adapter.py discover	BFS link discovery from root URL
seo_crawler_adapter.py crawl-manifest	Crawl from reference-curator manifest
seo_crawler_adapter.py status	Check crawl progress
crawl_mgr.py select-crawler	Recommend optimal crawler for a URL
crawl_mgr.py store-result	Store crawl results and create manifest
crawl_mgr.py list-crawls	List recent crawl records

Integration

From	To
reference-discovery	URL manifest input
SEO aiohttp / Firecrawl MCP	Raw crawled files
→	content-repository (crawl manifest + raw files)
→	raw/ directory (when --save-raw or --no-distill)
quality-reviewer (deep_research)	Additional crawl requests