our-claude-skills/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/CLAUDE.md

Web Crawler Orchestrator

Orchestrates web crawling with intelligent backend selection. Primary backend is the OurSEO async crawler (rate-limited, resumable, metadata-rich). Falls back to Firecrawl MCP for JS-rendered sites or WebFetch for simple pages.

Trigger Keywords

"crawl URLs", "fetch documents", "scrape pages", "download references"

Intelligent Crawler Selection

# Get crawler recommendation for a URL
uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.com"

Crawler	Best For	Auto-Selected When
SEO aiohttp (default)	Documentation sites, sitemaps	docs/developer/api domains, has sitemap
Firecrawl MCP	Dynamic sites, SPAs	React/Vue/Angular, JS-rendered
WebFetch	Quick single pages	Fallback, simple pages
Node.js	Small docs sites	≤50 pages, static content
Scrapy	Enterprise crawls	>200 pages, multi-domain

Workflow

Step 1: Analyze Target Site

Run select-crawler to determine site characteristics and get a recommendation.

Step 2a: Execute Crawl — SEO aiohttp (Recommended)

The SEO crawler adapter provides rate limiting, progress tracking, resume support, and full metadata extraction.

# Crawl specific URLs
uv run python scripts/seo_crawler_adapter.py crawl \
  --urls "https://docs.example.com/page1,https://docs.example.com/page2" \
  --output ~/Documents/reference-library/topic-slug/raw/

# Crawl from XML sitemap (auto-discovers URLs)
uv run python scripts/seo_crawler_adapter.py crawl-sitemap \
  --sitemap https://docs.example.com/sitemap.xml \
  --output ~/Documents/reference-library/topic-slug/raw/ \
  --max-pages 50

# Discover URLs from root page (follows internal links)
uv run python scripts/seo_crawler_adapter.py discover \
  --url https://docs.example.com \
  --output ~/Documents/reference-library/topic-slug/raw/ \
  --max-depth 2 --max-pages 50 --same-domain

# Crawl from reference-curator manifest
uv run python scripts/seo_crawler_adapter.py crawl-manifest \
  --manifest ./manifest.json \
  --output ~/Documents/reference-library/topic-slug/raw/

# Check crawl progress
uv run python scripts/seo_crawler_adapter.py status

SEO crawler capabilities:

• Token-bucket rate limiting (via BaseAsyncClient)
• Semaphore-controlled concurrency (default: 5 concurrent)
• Tenacity retry with exponential backoff
• Full PageMetadata extraction: title, headings, links, schema, OG
• Progress tracking with ETA and resume files
• Sitemap XML parsing (including sitemap indexes)
• Internal link discovery (breadth-first, depth-limited)
• Direct .raw.md output with enriched YAML frontmatter

Step 2b: Execute Crawl — Firecrawl MCP (for JS-rendered sites)

Use Firecrawl MCP tools directly when JS rendering is required:

firecrawl_map(url, limit=100)           # Discover URLs
firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
firecrawl_crawl(url, max_depth=2, limit=50)

Step 2c: Execute Crawl — WebFetch (fallback)

For quick single-page fetches when no crawl infrastructure is needed:

WebFetch(url, prompt="Return the COMPLETE page content as clean markdown.")

Step 3: Store Crawl Results

# Store crawled files and create result manifest
uv run python scripts/crawl_mgr.py store-result \
  --raw-dir ~/Documents/reference-library/raw/ \
  --crawler seo-aiohttp \
  --source-id 1 \
  --output crawl_result.json

# List recent crawls
uv run python scripts/crawl_mgr.py list-crawls --status completed

Step 4: Write Raw Files (when --save-raw or --no-distill)

When the pipeline is invoked with --save-raw (or --no-distill, which implies it), write each crawled page's markdown verbatim to the raw output directory.

Note: The SEO crawler adapter (seo_crawler_adapter.py) writes .raw.md files directly with enriched YAML frontmatter — this step is automatic when using seo-aiohttp.

File naming: {NN}-{page-slug}.raw.md — zero-padded index matching crawl order.

Output path:

• --save-raw: {output}/{topic-slug}/raw/
• --no-distill: {output}/{topic-slug}/ (raw files are primary)

YAML frontmatter (SEO crawler enriched):

---
source_url: {original_url}
crawled_at: {ISO 8601 timestamp}
crawler: seo-aiohttp
content_length: {character count}
raw: true
status_code: {HTTP status}
title: {page title}
word_count: {word count}
internal_links: {count}
external_links: {count}
h1: {h1 text}
schema_types: [{types found}]
response_time_ms: {ms}
---

After all pages are written, a combined raw bundle ({topic-slug}-raw-complete.md) is auto-generated by concatenating all .raw.md files with --- separators.

Rate Limiting

All crawlers respect these limits:

• 20 requests/minute (SEO crawler: configurable via BaseAsyncClient)
• 3-5 concurrent requests (SEO crawler: semaphore-controlled)
• Exponential backoff on 429/5xx (SEO crawler: tenacity)

Error Handling

Error	Action
Timeout	Retry with exponential backoff (max 3 attempts)
Rate limit (429)	Token-bucket + exponential backoff
Not found (404)	Log and skip
Access denied (403)	Log, mark as failed
JS rendering needed	Switch to Firecrawl MCP
Connection error	Retry with backoff, then skip

Scripts

Command	Purpose
seo_crawler_adapter.py crawl	Crawl URLs with SEO metadata extraction
seo_crawler_adapter.py crawl-sitemap	Discover and crawl from XML sitemap
seo_crawler_adapter.py discover	BFS link discovery from root URL
seo_crawler_adapter.py crawl-manifest	Crawl from reference-curator manifest
seo_crawler_adapter.py status	Check crawl progress
crawl_mgr.py select-crawler	Recommend optimal crawler for a URL
crawl_mgr.py store-result	Store crawl results and create manifest
crawl_mgr.py list-crawls	List recent crawl records

Integration

From	To
reference-discovery	URL manifest input
SEO aiohttp / Firecrawl MCP	Raw crawled files
→	content-repository (crawl manifest + raw files)
→	raw/ directory (when --save-raw or --no-distill)
quality-reviewer (deep_research)	Additional crawl requests