feat(reference-curator): add --wayback-fallback flag + CAPTCHA/IP block detection

- Add _is_captcha_response() to detect Google sorry page, 429, and
  "unusual traffic" markers in crawler responses
- Add --wayback-fallback CLI flag: auto-switches remaining URLs to
  web.archive.org/web/2024/ after 3 consecutive CAPTCHA blocks
- Add --delay CLI flag for polite crawling with configurable interval
- Add retry logic with exponential backoff on 429/5xx responses
- Document GOTCHA section in crawler CLAUDE.md with detection signals,
  Wayback detour solution, and prevention tips
- Add GOTCHA callout in pipeline orchestrator Stage 2 docs
- Record source: wayback in frontmatter for traceability

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/CLAUDE.md

@@ -134,6 +134,7 @@ All crawlers respect these limits:
 - 20 requests/minute (SEO crawler: configurable via `BaseAsyncClient`)
 - 3-5 concurrent requests (SEO crawler: semaphore-controlled)
 - Exponential backoff on 429/5xx (SEO crawler: tenacity)
+- Use `--delay` flag for polite crawling (recommended: 2-3s for Google properties)
 
 ## Error Handling
 
@@ -142,6 +143,56 @@ All crawlers respect these limits:
 | Timeout | Retry with exponential backoff (max 3 attempts) |
 | Rate limit (429) | Token-bucket + exponential backoff |
 | Not found (404) | Log and skip |
+
+## ⚠️ GOTCHA: IP Block / CAPTCHA Detection & Wayback Fallback
+
+**Problem:** Large-volume crawls on sites like `support.google.com` trigger IP-level bot detection. Google redirects to `google.com/sorry/` with a CAPTCHA. Once triggered:
+- Direct `requests`/`curl` returns 429 indefinitely (not a temporary rate limit)
+- Playwright headless/headful also gets blocked (Google detects automation)
+- The block persists even after the user solves the CAPTCHA in their browser (cookie-based, not IP-based lift)
+- `WebFetch` and `read_webpage` MCP tools also fail (same IP or server-side block)
+
+**Detection:** Check for these signals in crawl responses:
+1. HTTP 302 redirect to `google.com/sorry/index?continue=...`
+2. Response body contains `solveSimpleChallenge` or `unusual traffic`
+3. Persistent 429 after retries with exponential backoff (>3 consecutive 429s)
+
+**Solution: Wayback Machine Detour**
+
+When IP block / CAPTCHA is detected, automatically fall back to Wayback Machine:
+
+```bash
+# Instead of: https://support.google.com/analytics/answer/9304153
+# Fetch from: https://web.archive.org/web/2024/https://support.google.com/analytics/answer/9304153
+```
+
+The SEO crawler adapter supports this with the `--wayback-fallback` flag:
+
+```bash
+uv run python scripts/seo_crawler_adapter.py crawl \
+  --urls "@urls.txt" \
+  --output ~/Documents/reference-library/topic/raw/ \
+  --delay 1.5 \
+  --wayback-fallback
+```
+
+**How it works:**
+1. First attempts direct fetch for each URL
+2. On 429 or CAPTCHA redirect, flags `captcha_detected`
+3. After 3 consecutive CAPTCHA blocks, switches ALL remaining URLs to Wayback prefix
+4. Wayback URLs use `https://web.archive.org/web/2024/{original_url}`
+5. Frontmatter records `source: wayback` for traceability
+
+**Limitations:**
+- Wayback snapshots may be weeks/months old (acceptable for documentation/help pages)
+- Not all pages are archived (rare for major sites like Google, MDN, etc.)
+- Wayback has its own rate limits (gentler — 1.5s delay is sufficient)
+- Links in content point to Wayback URLs — the extractor strips these automatically
+
+**When to use `--delay` vs `--wayback-fallback`:**
+- Start with `--delay 3.0` for large Google crawls (preventive)
+- If 429s still occur, add `--wayback-fallback` (reactive)
+- For non-Google sites, `--delay 2.0` is usually sufficient
 | Access denied (403) | Log, mark as `failed` |
 | JS rendering needed | Switch to Firecrawl MCP |
 | Connection error | Retry with backoff, then skip |