# Web Crawler Orchestrator

Orchestrates web crawling with intelligent backend selection. Claude performs actual crawling via Firecrawl MCP tools. This skill manages crawl results, selects crawlers, and tracks crawl metadata.

## Trigger Keywords
"crawl URLs", "fetch documents", "scrape pages", "download references"

## Intelligent Crawler Selection

```bash
# Get crawler recommendation for a URL
uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.com"
```

| Crawler | Best For | Auto-Selected When |
|---------|----------|-------------------|
| **Firecrawl MCP** (default) | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered |
| **Node.js** | Small docs sites | ≤50 pages, static content |
| **Python aiohttp** | Technical docs | ≤200 pages, needs SEO data |
| **Scrapy** | Enterprise crawls | >200 pages, multi-domain |

## Workflow

### Step 1: Analyze Target Site
Run `select-crawler` to determine site characteristics and get a recommendation.

### Step 2: Execute Crawl
Use Firecrawl MCP tools directly:
```
firecrawl_map(url, limit=100)           # Discover URLs
firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
firecrawl_crawl(url, max_depth=2, limit=50)
```

### Step 3: Store Crawl Results
```bash
# Store crawled files and create result manifest
uv run python scripts/crawl_mgr.py store-result \
  --raw-dir ~/Documents/reference-library/raw/ \
  --crawler firecrawl \
  --source-id 1 \
  --output crawl_result.json

# List recent crawls
uv run python scripts/crawl_mgr.py list-crawls --status completed
```

## Rate Limiting

All crawlers respect these limits:
- 20 requests/minute
- 3 concurrent requests
- Exponential backoff on 429/5xx

## Error Handling

| Error | Action |
|-------|--------|
| Timeout | Retry once with 2x timeout |
| Rate limit (429) | Exponential backoff, max 3 retries |
| Not found (404) | Log and skip |
| Access denied (403) | Log, mark as `failed` |
| JS rendering needed | Switch to Firecrawl |

## Scripts

| Command | Purpose |
|---------|---------|
| `crawl_mgr.py select-crawler` | Recommend optimal crawler for a URL |
| `crawl_mgr.py store-result` | Store crawl results and create manifest |
| `crawl_mgr.py list-crawls` | List recent crawl records |

## Integration

| From | To |
|------|-----|
| reference-discovery | URL manifest input |
| Firecrawl MCP | Raw crawled files |
| → | content-repository (crawl manifest + raw files) |
| quality-reviewer (deep_research) | Additional crawl requests |