our-claude-skills/custom-skills/90-reference-curator/07-pipeline-orchestrator/code/CLAUDE.md

Pipeline Orchestrator

Coordinates the full 6-skill reference curation workflow with QA loop handling. Manages pipeline state (init, advance, pause, resume, complete) while Claude orchestrates the actual stage execution.

Trigger Keywords

"curate references", "full pipeline", "run curation", "reference-curator-pipeline"

Architecture

[Input] → discovery → crawler → repository → distiller ◄──┐
                          │                       │     │
                          │ (--save-raw)    reviewer     │
                          │ writes raw/           │     │
                          │               ┌───────┼─────┤
                          │               ▼       ▼     │
                          │            APPROVE  REJECT  REFACTOR ────┤
                          │               │       │              │
                          │               ▼       ▼       DEEP_RESEARCH
                          │            export   archive        │
                          │               │                    ▼
                          │               │               crawler ─┘
                          │               │
                          └── (--no-distill) ──► export (raw only)
                               skip stages 4+5

Pipeline State Management

Initialize a Run

uv run python scripts/pipeline.py init \
  --input "prompt engineering" --type topic \
  --options '{"max_sources": 10, "auto_approve": true, "save_raw": false, "no_distill": false}'

Advance to Next Stage

uv run python scripts/pipeline.py advance \
  --run-id 1 --stage crawling \
  --stats '{"sources_discovered": 8}'

Pause on Error

uv run python scripts/pipeline.py pause \
  --run-id 1 --error "Crawl timeout on page 45" --stage crawling

Resume from Pause

uv run python scripts/pipeline.py resume --run-id 1

Complete Pipeline

uv run python scripts/pipeline.py complete \
  --run-id 1 --export-path ~/reference-library/exports/ --export-count 40

Check Status

uv run python scripts/pipeline.py status --run-id 1
uv run python scripts/pipeline.py status --all

QA Loop Tracking

# Track a refactor iteration for a document
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action refactor

# Track a deep research iteration
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action deep_research

Returns one of:

• re_distill — proceed with refactor
• re_crawl_and_distill — proceed with deep research
• needs_manual_review — max iterations exceeded

Decision	Max Iterations
REFACTOR	3
DEEP_RESEARCH	2
Combined total	5

Pipeline Execution Flow

Stage 1: Reference Discovery (Topic Mode Only)

If mode == 'topic':
  → Claude runs WebSearch
  → discover.py create-manifest
  → discover.py dedup
  → pipeline.py advance --stage crawling

Stage 2: Web Crawler

→ Claude uses Firecrawl MCP tools
→ crawl_mgr.py store-result
→ If --save-raw or --no-distill: write raw .raw.md files to output
→ pipeline.py advance --stage storing

Stage 3: Content Repository

→ repo.py store (for each crawled doc)
→ pipeline.py advance --stage evaluating

Flag Resolution (after Stage 3)

If --no-distill:
  → set save_raw = true (implied)
  → skip Stage 4 (content-distiller)
  → skip Stage 5 (quality-reviewer)
  → pipeline.py advance --stage exporting --skip-reason "no-distill"
  → proceed directly to Stage 6 with raw files as input

If --save-raw (without --no-distill):
  → raw files already written during Stage 2
  → continue normal pipeline through stages 4-5
  → Stage 6 exports both distilled AND raw content

Stage 4: Gemini Quality Gate (Pre-Distillation)

Skipped when --no-distill is active.

→ reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve
→ APPROVE: proceed to distillation
→ DEEP_RESEARCH: pipeline.py track-iteration → crawler (re-crawl)
→ REJECT: skip document entirely
→ pipeline.py advance --stage distilling

Stage 5: Content Distiller (Approved Only)

Skipped when --no-distill is active.

→ distiller.py load-pending
→ Claude distills each approved document
→ distiller.py store
→ pipeline.py advance --stage exporting

Stage 6: Markdown Exporter

If --no-distill:
  → exporter.py raw --primary (raw files are the main output)
  → exporter.py index (generate README from raw file metadata)
  → exporter.py verify
  → pipeline.py complete

If --save-raw (with distillation):
  → exporter.py project (normal distilled export)
  → exporter.py raw (write raw/ subdirectory + bundle)
  → exporter.py index (includes raw section)
  → exporter.py crossrefs
  → exporter.py verify
  → pipeline.py complete

Default (no raw flags):
  → exporter.py project
  → exporter.py index
  → exporter.py crossrefs
  → exporter.py verify
  → pipeline.py complete

Checkpoint Strategy

Stage	Checkpoint	Resume Point
discovery	manifest.json created	→ crawler
crawl	crawl_result.json + raw files	→ repository
store	DB records	→ distiller (or skip)
no-distill skip	stages 4-5 marked skipped	→ exporter (raw)
distill	distilled_content records	→ reviewer
review	review_logs records	→ exporter or loop
export	final export complete	Done

Scripts

Command	Purpose
pipeline.py init	Initialize a new pipeline run
pipeline.py advance	Advance to next stage
pipeline.py pause	Pause on error
pipeline.py resume	Resume from pause
pipeline.py complete	Mark pipeline complete
pipeline.py status	Show run status
pipeline.py track-iteration	Track QA loop iterations

Configuration

Reads from ~/.config/reference-curator/pipeline_config.yaml:

pipeline:
  max_sources: 10
  max_pages: 50
  auto_approve: false
  approval_threshold: 0.85
  save_raw: false
  no_distill: false

qa_loop:
  max_refactor_iterations: 3
  max_deep_research_iterations: 2
  max_total_iterations: 5

export:
  default_format: project_files