# Pipeline Orchestrator

Coordinates the full 6-skill reference curation workflow with QA loop handling. Manages pipeline state (init, advance, pause, resume, complete) while Claude orchestrates the actual stage execution.

## Trigger Keywords
"curate references", "full pipeline", "run curation", "reference-curator-pipeline"

## Architecture

```
[Input] → discovery → crawler → repository → distiller ◄──┐
                                                    │     │
                                              reviewer     │
                                                    │     │
                    ┌───────────────────────────────┼─────┤
                    ▼           ▼                   ▼     │
                 APPROVE     REJECT          REFACTOR ────┤
                    │           │                         │
                    ▼           ▼                DEEP_RESEARCH
                 export      archive                 │
                                                     ▼
                                                  crawler ─┘
```

## Pipeline State Management

### Initialize a Run
```bash
uv run python scripts/pipeline.py init \
  --input "prompt engineering" --type topic \
  --options '{"max_sources": 10, "auto_approve": true}'
```

### Advance to Next Stage
```bash
uv run python scripts/pipeline.py advance \
  --run-id 1 --stage crawling \
  --stats '{"sources_discovered": 8}'
```

### Pause on Error
```bash
uv run python scripts/pipeline.py pause \
  --run-id 1 --error "Crawl timeout on page 45" --stage crawling
```

### Resume from Pause
```bash
uv run python scripts/pipeline.py resume --run-id 1
```

### Complete Pipeline
```bash
uv run python scripts/pipeline.py complete \
  --run-id 1 --export-path ~/reference-library/exports/ --export-count 40
```

### Check Status
```bash
uv run python scripts/pipeline.py status --run-id 1
uv run python scripts/pipeline.py status --all
```

## QA Loop Tracking

```bash
# Track a refactor iteration for a document
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action refactor

# Track a deep research iteration
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action deep_research
```

Returns one of:
- `re_distill` — proceed with refactor
- `re_crawl_and_distill` — proceed with deep research
- `needs_manual_review` — max iterations exceeded

| Decision | Max Iterations |
|----------|----------------|
| REFACTOR | 3 |
| DEEP_RESEARCH | 2 |
| Combined total | 5 |

## Pipeline Execution Flow

### Stage 1: Reference Discovery (Topic Mode Only)
```
If mode == 'topic':
  → Claude runs WebSearch
  → discover.py create-manifest
  → discover.py dedup
  → pipeline.py advance --stage crawling
```

### Stage 2: Web Crawler
```
→ Claude uses Firecrawl MCP tools
→ crawl_mgr.py store-result
→ pipeline.py advance --stage storing
```

### Stage 3: Content Repository
```
→ repo.py store (for each crawled doc)
→ pipeline.py advance --stage evaluating
```

### Stage 4: Gemini Quality Gate (Pre-Distillation)
```
→ reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve
→ APPROVE: proceed to distillation
→ DEEP_RESEARCH: pipeline.py track-iteration → crawler (re-crawl)
→ REJECT: skip document entirely
→ pipeline.py advance --stage distilling
```

### Stage 5: Content Distiller (Approved Only)
```
→ distiller.py load-pending
→ Claude distills each approved document
→ distiller.py store
→ pipeline.py advance --stage exporting
```

### Stage 6: Markdown Exporter
```
→ exporter.py project
→ exporter.py index
→ exporter.py crossrefs
→ exporter.py verify
→ pipeline.py complete
```

## Checkpoint Strategy

| Stage | Checkpoint | Resume Point |
|-------|------------|--------------|
| discovery | manifest.json created | → crawler |
| crawl | crawl_result.json | → repository |
| store | DB records | → distiller |
| distill | distilled_content records | → reviewer |
| review | review_logs records | → exporter or loop |
| export | final export complete | Done |

## Scripts

| Command | Purpose |
|---------|---------|
| `pipeline.py init` | Initialize a new pipeline run |
| `pipeline.py advance` | Advance to next stage |
| `pipeline.py pause` | Pause on error |
| `pipeline.py resume` | Resume from pause |
| `pipeline.py complete` | Mark pipeline complete |
| `pipeline.py status` | Show run status |
| `pipeline.py track-iteration` | Track QA loop iterations |

## Configuration

Reads from `~/.config/reference-curator/pipeline_config.yaml`:

```yaml
pipeline:
  max_sources: 10
  max_pages: 50
  auto_approve: false
  approval_threshold: 0.85

qa_loop:
  max_refactor_iterations: 3
  max_deep_research_iterations: 2
  max_total_iterations: 5

export:
  default_format: project_files
```