# Web Crawler Orchestrator

Orchestrates web crawling with intelligent backend selection. Primary backend is the OurSEO async crawler (rate-limited, resumable, metadata-rich). Falls back to Firecrawl MCP for JS-rendered sites or WebFetch for simple pages.

## Trigger Keywords
"crawl URLs", "fetch documents", "scrape pages", "download references"

## Intelligent Crawler Selection

```bash
# Get crawler recommendation for a URL
uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.com"
```

| Crawler | Best For | Auto-Selected When |
|---------|----------|-------------------|
| **SEO aiohttp** (default) | Documentation sites, sitemaps | docs/developer/api domains, has sitemap |
| **Firecrawl MCP** | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered |
| **WebFetch** | Quick single pages | Fallback, simple pages |
| **Node.js** | Small docs sites | ≤50 pages, static content |
| **Scrapy** | Enterprise crawls | >200 pages, multi-domain |

## Workflow

### Step 1: Analyze Target Site
Run `select-crawler` to determine site characteristics and get a recommendation.

### Step 2a: Execute Crawl — SEO aiohttp (Recommended)

The SEO crawler adapter provides rate limiting, progress tracking, resume support, and full metadata extraction.

```bash
# Crawl specific URLs
uv run python scripts/seo_crawler_adapter.py crawl \
  --urls "https://docs.example.com/page1,https://docs.example.com/page2" \
  --output ~/Documents/reference-library/topic-slug/raw/

# Crawl from XML sitemap (auto-discovers URLs)
uv run python scripts/seo_crawler_adapter.py crawl-sitemap \
  --sitemap https://docs.example.com/sitemap.xml \
  --output ~/Documents/reference-library/topic-slug/raw/ \
  --max-pages 50

# Discover URLs from root page (follows internal links)
uv run python scripts/seo_crawler_adapter.py discover \
  --url https://docs.example.com \
  --output ~/Documents/reference-library/topic-slug/raw/ \
  --max-depth 2 --max-pages 50 --same-domain

# Crawl from reference-curator manifest
uv run python scripts/seo_crawler_adapter.py crawl-manifest \
  --manifest ./manifest.json \
  --output ~/Documents/reference-library/topic-slug/raw/

# Check crawl progress
uv run python scripts/seo_crawler_adapter.py status
```

**SEO crawler capabilities:**
- Token-bucket rate limiting (via `BaseAsyncClient`)
- Semaphore-controlled concurrency (default: 5 concurrent)
- Tenacity retry with exponential backoff
- Full `PageMetadata` extraction: title, headings, links, schema, OG
- Progress tracking with ETA and resume files
- Sitemap XML parsing (including sitemap indexes)
- Internal link discovery (breadth-first, depth-limited)
- Direct `.raw.md` output with enriched YAML frontmatter

### Step 2b: Execute Crawl — Firecrawl MCP (for JS-rendered sites)

Use Firecrawl MCP tools directly when JS rendering is required:
```
firecrawl_map(url, limit=100)           # Discover URLs
firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
firecrawl_crawl(url, max_depth=2, limit=50)
```

### Step 2c: Execute Crawl — WebFetch (fallback)

For quick single-page fetches when no crawl infrastructure is needed:
```
WebFetch(url, prompt="Return the COMPLETE page content as clean markdown.")
```

### Step 3: Store Crawl Results
```bash
# Store crawled files and create result manifest
uv run python scripts/crawl_mgr.py store-result \
  --raw-dir ~/Documents/reference-library/raw/ \
  --crawler seo-aiohttp \
  --source-id 1 \
  --output crawl_result.json

# List recent crawls
uv run python scripts/crawl_mgr.py list-crawls --status completed
```

### Step 4: Write Raw Files (when `--save-raw` or `--no-distill`)

When the pipeline is invoked with `--save-raw` (or `--no-distill`, which implies it), write each crawled page's markdown verbatim to the raw output directory.

**Note:** The SEO crawler adapter (`seo_crawler_adapter.py`) writes `.raw.md` files directly with enriched YAML frontmatter — this step is automatic when using seo-aiohttp.

**File naming:** `{NN}-{page-slug}.raw.md` — zero-padded index matching crawl order.

**Output path:**
- `--save-raw`: `{output}/{topic-slug}/raw/`
- `--no-distill`: `{output}/{topic-slug}/` (raw files are primary)

**YAML frontmatter (SEO crawler enriched):**
```yaml
---
source_url: {original_url}
crawled_at: {ISO 8601 timestamp}
crawler: seo-aiohttp
content_length: {character count}
raw: true
status_code: {HTTP status}
title: {page title}
word_count: {word count}
internal_links: {count}
external_links: {count}
h1: {h1 text}
schema_types: [{types found}]
response_time_ms: {ms}
---
```

**After all pages are written**, a combined raw bundle (`{topic-slug}-raw-complete.md`) is auto-generated by concatenating all `.raw.md` files with `---` separators.

## Rate Limiting

All crawlers respect these limits:
- 20 requests/minute (SEO crawler: configurable via `BaseAsyncClient`)
- 3-5 concurrent requests (SEO crawler: semaphore-controlled)
- Exponential backoff on 429/5xx (SEO crawler: tenacity)
- Use `--delay` flag for polite crawling (recommended: 2-3s for Google properties)

## Error Handling

| Error | Action |
|-------|--------|
| Timeout | Retry with exponential backoff (max 3 attempts) |
| Rate limit (429) | Token-bucket + exponential backoff |
| Not found (404) | Log and skip |

## ⚠️ GOTCHA: IP Block / CAPTCHA Detection & Wayback Fallback

**Problem:** Large-volume crawls on sites like `support.google.com` trigger IP-level bot detection. Google redirects to `google.com/sorry/` with a CAPTCHA. Once triggered:
- Direct `requests`/`curl` returns 429 indefinitely (not a temporary rate limit)
- Playwright headless/headful also gets blocked (Google detects automation)
- The block persists even after the user solves the CAPTCHA in their browser (cookie-based, not IP-based lift)
- `WebFetch` and `read_webpage` MCP tools also fail (same IP or server-side block)

**Detection:** Check for these signals in crawl responses:
1. HTTP 302 redirect to `google.com/sorry/index?continue=...`
2. Response body contains `solveSimpleChallenge` or `unusual traffic`
3. Persistent 429 after retries with exponential backoff (>3 consecutive 429s)

**Solution: Wayback Machine Detour**

When IP block / CAPTCHA is detected, automatically fall back to Wayback Machine:

```bash
# Instead of: https://support.google.com/analytics/answer/9304153
# Fetch from: https://web.archive.org/web/2024/https://support.google.com/analytics/answer/9304153
```

The SEO crawler adapter supports this with the `--wayback-fallback` flag:

```bash
uv run python scripts/seo_crawler_adapter.py crawl \
  --urls "@urls.txt" \
  --output ~/Documents/reference-library/topic/raw/ \
  --delay 1.5 \
  --wayback-fallback
```

**How it works:**
1. First attempts direct fetch for each URL
2. On 429 or CAPTCHA redirect, flags `captcha_detected`
3. After 3 consecutive CAPTCHA blocks, switches ALL remaining URLs to Wayback prefix
4. Wayback URLs use `https://web.archive.org/web/2024/{original_url}`
5. Frontmatter records `source: wayback` for traceability

**Limitations:**
- Wayback snapshots may be weeks/months old (acceptable for documentation/help pages)
- Not all pages are archived (rare for major sites like Google, MDN, etc.)
- Wayback has its own rate limits (gentler — 1.5s delay is sufficient)
- Links in content point to Wayback URLs — the extractor strips these automatically

**When to use `--delay` vs `--wayback-fallback`:**
- Start with `--delay 3.0` for large Google crawls (preventive)
- If 429s still occur, add `--wayback-fallback` (reactive)
- For non-Google sites, `--delay 2.0` is usually sufficient
| Access denied (403) | Log, mark as `failed` |
| JS rendering needed | Switch to Firecrawl MCP |
| Connection error | Retry with backoff, then skip |

## Scripts

| Command | Purpose |
|---------|---------|
| `seo_crawler_adapter.py crawl` | Crawl URLs with SEO metadata extraction |
| `seo_crawler_adapter.py crawl-sitemap` | Discover and crawl from XML sitemap |
| `seo_crawler_adapter.py discover` | BFS link discovery from root URL |
| `seo_crawler_adapter.py crawl-manifest` | Crawl from reference-curator manifest |
| `seo_crawler_adapter.py status` | Check crawl progress |
| `crawl_mgr.py select-crawler` | Recommend optimal crawler for a URL |
| `crawl_mgr.py store-result` | Store crawl results and create manifest |
| `crawl_mgr.py list-crawls` | List recent crawl records |

## Integration

| From | To |
|------|-----|
| reference-discovery | URL manifest input |
| SEO aiohttp / Firecrawl MCP | Raw crawled files |
| → | content-repository (crawl manifest + raw files) |
| → | raw/ directory (when --save-raw or --no-distill) |
| quality-reviewer (deep_research) | Additional crawl requests |