# Pipeline Orchestrator

Coordinates the full 6-skill reference curation workflow with QA loop handling. Manages pipeline state (init, advance, pause, resume, complete) while Claude orchestrates the actual stage execution.

## Trigger Keywords
"curate references", "full pipeline", "run curation", "reference-curator-pipeline"

## Architecture

```
[Input] → discovery → crawler → repository → distiller ◄──┐
                          │                       │     │
                          │ (--save-raw)    reviewer     │
                          │ writes raw/           │     │
                          │               ┌───────┼─────┤
                          │               ▼       ▼     │
                          │            APPROVE  REJECT  REFACTOR ────┤
                          │               │       │              │
                          │               ▼       ▼       DEEP_RESEARCH
                          │            export   archive        │
                          │               │                    ▼
                          │               │               crawler ─┘
                          │               │
                          └── (--no-distill) ──► export (raw only)
                               skip stages 4+5
```

## Pipeline State Management

### Initialize a Run
```bash
uv run python scripts/pipeline.py init \
  --input "prompt engineering" --type topic \
  --options '{"max_sources": 10, "auto_approve": true, "save_raw": false, "no_distill": false}'
```

### Advance to Next Stage
```bash
uv run python scripts/pipeline.py advance \
  --run-id 1 --stage crawling \
  --stats '{"sources_discovered": 8}'
```

### Pause on Error
```bash
uv run python scripts/pipeline.py pause \
  --run-id 1 --error "Crawl timeout on page 45" --stage crawling
```

### Resume from Pause
```bash
uv run python scripts/pipeline.py resume --run-id 1
```

### Complete Pipeline
```bash
uv run python scripts/pipeline.py complete \
  --run-id 1 --export-path ~/reference-library/exports/ --export-count 40
```

### Check Status
```bash
uv run python scripts/pipeline.py status --run-id 1
uv run python scripts/pipeline.py status --all
```

## QA Loop Tracking

```bash
# Track a refactor iteration for a document
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action refactor

# Track a deep research iteration
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action deep_research
```

Returns one of:
- `re_distill` — proceed with refactor
- `re_crawl_and_distill` — proceed with deep research
- `needs_manual_review` — max iterations exceeded

| Decision | Max Iterations |
|----------|----------------|
| REFACTOR | 3 |
| DEEP_RESEARCH | 2 |
| Combined total | 5 |

## Pipeline Execution Flow

### Stage 1: Reference Discovery (Topic Mode Only)
```
If mode == 'topic':
  → Claude runs WebSearch
  → discover.py create-manifest
  → discover.py dedup
  → pipeline.py advance --stage crawling
```

### Stage 2: Web Crawler
```
→ Claude uses Firecrawl MCP tools or seo_crawler_adapter.py
→ crawl_mgr.py store-result
→ If --save-raw or --no-distill: write raw .raw.md files to output
→ pipeline.py advance --stage storing
```

**⚠️ GOTCHA — IP Block / CAPTCHA Detour:**
If the crawl stage encounters persistent 429 or CAPTCHA responses (e.g., Google's `google.com/sorry/` redirect), the pipeline should **NOT** retry indefinitely. Instead:
1. Detect: 3+ consecutive 429/CAPTCHA responses = IP blocked
2. Detour: Re-invoke `seo_crawler_adapter.py` with `--wayback-fallback` flag
3. The crawler auto-switches remaining URLs to `web.archive.org/web/2024/{url}`
4. Frontmatter records `source: wayback` for traceability
5. Resume normal pipeline from Stage 3

**Common triggers:** Large-volume crawls (>50 pages) on `support.google.com`, `developer.mozilla.org`, or any site with aggressive bot detection. **Prevention:** Always use `--delay 2.0` or higher for these domains.

### Stage 3: Content Repository
```
→ repo.py store (for each crawled doc)
→ pipeline.py advance --stage evaluating
```

### Flag Resolution (after Stage 3)
```
If --no-distill:
  → set save_raw = true (implied)
  → skip Stage 4 (content-distiller)
  → skip Stage 5 (quality-reviewer)
  → pipeline.py advance --stage exporting --skip-reason "no-distill"
  → proceed directly to Stage 6 with raw files as input

If --save-raw (without --no-distill):
  → raw files already written during Stage 2
  → continue normal pipeline through stages 4-5
  → Stage 6 exports both distilled AND raw content
```

### Stage 4: Gemini Quality Gate (Pre-Distillation)
**Skipped when `--no-distill` is active.**
```
→ reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve
→ APPROVE: proceed to distillation
→ DEEP_RESEARCH: pipeline.py track-iteration → crawler (re-crawl)
→ REJECT: skip document entirely
→ pipeline.py advance --stage distilling
```

### Stage 5: Content Distiller (Approved Only)
**Skipped when `--no-distill` is active.**
```
→ distiller.py load-pending
→ Claude distills each approved document
→ distiller.py store
→ pipeline.py advance --stage exporting
```

### Stage 6: Markdown Exporter
```
If --no-distill:
  → exporter.py raw --primary (raw files are the main output)
  → exporter.py index (generate README from raw file metadata)
  → exporter.py verify
  → pipeline.py complete

If --save-raw (with distillation):
  → exporter.py project (normal distilled export)
  → exporter.py raw (write raw/ subdirectory + bundle)
  → exporter.py index (includes raw section)
  → exporter.py crossrefs
  → exporter.py verify
  → pipeline.py complete

Default (no raw flags):
  → exporter.py project
  → exporter.py index
  → exporter.py crossrefs
  → exporter.py verify
  → pipeline.py complete
```

## Checkpoint Strategy

| Stage | Checkpoint | Resume Point |
|-------|------------|--------------|
| discovery | manifest.json created | → crawler |
| crawl | crawl_result.json + raw files | → repository |
| store | DB records | → distiller (or skip) |
| no-distill skip | stages 4-5 marked skipped | → exporter (raw) |
| distill | distilled_content records | → reviewer |
| review | review_logs records | → exporter or loop |
| export | final export complete | Done |

## Scripts

| Command | Purpose |
|---------|---------|
| `pipeline.py init` | Initialize a new pipeline run |
| `pipeline.py advance` | Advance to next stage |
| `pipeline.py pause` | Pause on error |
| `pipeline.py resume` | Resume from pause |
| `pipeline.py complete` | Mark pipeline complete |
| `pipeline.py status` | Show run status |
| `pipeline.py track-iteration` | Track QA loop iterations |

## Configuration

Reads from `~/.config/reference-curator/pipeline_config.yaml`:

```yaml
pipeline:
  max_sources: 10
  max_pages: 50
  auto_approve: false
  approval_threshold: 0.85
  save_raw: false
  no_distill: false

qa_loop:
  max_refactor_iterations: 3
  max_deep_research_iterations: 2
  max_total_iterations: 5

export:
  default_format: project_files
```
