- Add _is_captcha_response() to detect Google sorry page, 429, and
"unusual traffic" markers in crawler responses
- Add --wayback-fallback CLI flag: auto-switches remaining URLs to
web.archive.org/web/2024/ after 3 consecutive CAPTCHA blocks
- Add --delay CLI flag for polite crawling with configurable interval
- Add retry logic with exponential backoff on 429/5xx responses
- Document GOTCHA section in crawler CLAUDE.md with detection signals,
Wayback detour solution, and prevention tips
- Add GOTCHA callout in pipeline orchestrator Stage 2 docs
- Record source: wayback in frontmatter for traceability
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Andrew Yim
382b55e9c8