Files

Andrew Yim 382b55e9c8 feat(reference-curator): add --wayback-fallback flag + CAPTCHA/IP block detection

- Add _is_captcha_response() to detect Google sorry page, 429, and
  "unusual traffic" markers in crawler responses
- Add --wayback-fallback CLI flag: auto-switches remaining URLs to
  web.archive.org/web/2024/ after 3 consecutive CAPTCHA blocks
- Add --delay CLI flag for polite crawling with configurable interval
- Add retry logic with exponential backoff on 429/5xx responses
- Document GOTCHA section in crawler CLAUDE.md with detection signals,
  Wayback detour solution, and prevention tips
- Add GOTCHA callout in pipeline orchestrator Stage 2 docs
- Record source: wayback in frontmatter for traceability

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-13 01:55:11 +09:00

7.1 KiB

Raw Blame History

Pipeline Orchestrator

Coordinates the full 6-skill reference curation workflow with QA loop handling. Manages pipeline state (init, advance, pause, resume, complete) while Claude orchestrates the actual stage execution.

Trigger Keywords

"curate references", "full pipeline", "run curation", "reference-curator-pipeline"

Architecture

[Input] → discovery → crawler → repository → distiller ◄──┐
                          │                       │     │
                          │ (--save-raw)    reviewer     │
                          │ writes raw/           │     │
                          │               ┌───────┼─────┤
                          │               ▼       ▼     │
                          │            APPROVE  REJECT  REFACTOR ────┤
                          │               │       │              │
                          │               ▼       ▼       DEEP_RESEARCH
                          │            export   archive        │
                          │               │                    ▼
                          │               │               crawler ─┘
                          │               │
                          └── (--no-distill) ──► export (raw only)
                               skip stages 4+5

Pipeline State Management

Initialize a Run

uv run python scripts/pipeline.py init \
  --input "prompt engineering" --type topic \
  --options '{"max_sources": 10, "auto_approve": true, "save_raw": false, "no_distill": false}'

Advance to Next Stage

uv run python scripts/pipeline.py advance \
  --run-id 1 --stage crawling \
  --stats '{"sources_discovered": 8}'

Pause on Error

uv run python scripts/pipeline.py pause \
  --run-id 1 --error "Crawl timeout on page 45" --stage crawling

Resume from Pause

uv run python scripts/pipeline.py resume --run-id 1

Complete Pipeline

uv run python scripts/pipeline.py complete \
  --run-id 1 --export-path ~/reference-library/exports/ --export-count 40

Check Status

uv run python scripts/pipeline.py status --run-id 1
uv run python scripts/pipeline.py status --all

QA Loop Tracking

# Track a refactor iteration for a document
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action refactor

# Track a deep research iteration
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action deep_research

Returns one of:

re_distill — proceed with refactor
re_crawl_and_distill — proceed with deep research
needs_manual_review — max iterations exceeded

Decision	Max Iterations
REFACTOR	3
DEEP_RESEARCH	2
Combined total	5

Pipeline Execution Flow

Stage 1: Reference Discovery (Topic Mode Only)

If mode == 'topic':
  → Claude runs WebSearch
  → discover.py create-manifest
  → discover.py dedup
  → pipeline.py advance --stage crawling

Stage 2: Web Crawler

→ Claude uses Firecrawl MCP tools or seo_crawler_adapter.py
→ crawl_mgr.py store-result
→ If --save-raw or --no-distill: write raw .raw.md files to output
→ pipeline.py advance --stage storing

⚠️ GOTCHA — IP Block / CAPTCHA Detour: If the crawl stage encounters persistent 429 or CAPTCHA responses (e.g., Google's google.com/sorry/ redirect), the pipeline should NOT retry indefinitely. Instead:

Detect: 3+ consecutive 429/CAPTCHA responses = IP blocked
Detour: Re-invoke seo_crawler_adapter.py with --wayback-fallback flag
The crawler auto-switches remaining URLs to web.archive.org/web/2024/{url}
Frontmatter records source: wayback for traceability
Resume normal pipeline from Stage 3

Common triggers: Large-volume crawls (>50 pages) on support.google.com, developer.mozilla.org, or any site with aggressive bot detection. Prevention: Always use --delay 2.0 or higher for these domains.

Stage 3: Content Repository

→ repo.py store (for each crawled doc)
→ pipeline.py advance --stage evaluating

Flag Resolution (after Stage 3)

If --no-distill:
  → set save_raw = true (implied)
  → skip Stage 4 (content-distiller)
  → skip Stage 5 (quality-reviewer)
  → pipeline.py advance --stage exporting --skip-reason "no-distill"
  → proceed directly to Stage 6 with raw files as input

If --save-raw (without --no-distill):
  → raw files already written during Stage 2
  → continue normal pipeline through stages 4-5
  → Stage 6 exports both distilled AND raw content

Stage 4: Gemini Quality Gate (Pre-Distillation)

Skipped when --no-distill is active.

→ reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve
→ APPROVE: proceed to distillation
→ DEEP_RESEARCH: pipeline.py track-iteration → crawler (re-crawl)
→ REJECT: skip document entirely
→ pipeline.py advance --stage distilling

Stage 5: Content Distiller (Approved Only)