our-claude-skills/custom-skills/90-reference-curator/commands/reference-curator-pipeline.md

---
description: Orchestrates full reference curation pipeline with configurable depth. Runs discovery > crawl > store > distill > review > export with QA loop handling.
argument-hint: <topic|urls|manifest> [--depth light|standard|deep|full] [--output ~/Documents/reference-library/] [--max-sources 100] [--max-pages 50] [--auto-approve] [--threshold 0.85] [--max-iterations 3] [--export-format project_files] [--save-raw] [--no-distill]
allowed-tools: WebSearch, WebFetch, Read, Write, Bash, Grep, Glob, Task
---

# Reference Curator Pipeline

Full-stack orchestration of the 6-skill reference curation workflow.

## Input Modes

| Mode | Input Example | Pipeline Start |
|------|---------------|----------------|
| **Topic** | `"Claude system prompts"` | reference-discovery |
| **URLs** | `https://docs.anthropic.com/...` | web-crawler (skip discovery) |
| **Manifest** | `./manifest.json` | web-crawler (resume from discovery) |

## Arguments

- `<input>`: Required. Topic string, URL(s), or manifest file path
- `--depth`: Crawl depth level (default: `standard`). See Depth Levels below
- `--output`: Output directory path (default: `~/Documents/reference-library/`)
- `--max-sources`: Maximum sources to discover (topic mode, default: 100)
- `--max-pages`: Maximum pages per source to crawl (default varies by depth)
- `--auto-approve`: Auto-approve scores above threshold
- `--threshold`: Approval threshold (default: 0.85)
- `--max-iterations`: Max QA loop iterations per document (default: 3)
- `--export-format`: Output format: `project_files`, `fine_tuning`, `jsonl` (default: project_files)
- `--include-subdomains`: Include subdomains in site mapping (default: false)
- `--follow-external`: Follow external links found in content (default: false)
- `--save-raw`: Save intact crawled markdown alongside distilled content in a `raw/` subdirectory. Each file uses `.raw.md` extension. A combined raw bundle `{topic-slug}-raw-complete.md` is also generated.
- `--no-distill`: Skip stages 4-5 (content-distiller and quality-reviewer) entirely. Pure archival mode. Implies `--save-raw`. Raw crawled files become the primary output — no distilled subdirectory is created.

## Output Directory

**IMPORTANT: Never store output in any claude-skills, `.claude/`, or skill-related directory.**

The `--output` argument sets the base path for all pipeline output. If omitted, the default is `~/Documents/reference-library/`.

### Directory Structure

**Default (distilled output):**
```
{output}/
├── {topic-slug}/
│   ├── README.md                   # Index with table of contents
│   ├── 00-page-name.md             # Distilled individual page files
│   ├── 01-page-name.md
│   ├── {topic-slug}-complete.md    # Combined bundle (all pages)
│   └── manifest.json               # Crawl metadata
├── pipeline_state/                 # Resume state (auto-managed)
│   └── run_XXX/state.json
└── exports/                        # Fine-tuning / JSONL exports
```

**With `--save-raw` (distilled + raw):**
```
{output}/
├── {topic-slug}/
│   ├── README.md
│   ├── 00-page-name.md             # Distilled
│   ├── {topic-slug}-complete.md    # Combined distilled
│   ├── raw/                        # Intact crawled markdown
│   │   ├── 00-page-name.raw.md
│   │   ├── 01-page-name.raw.md
│   │   └── {topic-slug}-raw-complete.md  # Combined raw bundle
│   └── manifest.json
```

**With `--no-distill` (raw only, archival):**
```
{output}/
├── {topic-slug}/
│   ├── README.md
│   ├── 00-page-name.raw.md         # Raw crawled (primary output)
│   ├── 01-page-name.raw.md
│   ├── {topic-slug}-raw-complete.md # Combined raw bundle
│   └── manifest.json
```

### Resolution Rules

1. If `--output` is provided, use that path exactly
2. If not provided, use `~/Documents/reference-library/`
3. **Before writing any files**, check if the output directory exists
4. If the directory does NOT exist, **ask the user for permission** before creating it:
   - Show the full resolved path that will be created
   - Wait for explicit user approval
   - Only then run `mkdir -p <path>`
5. The topic slug is derived from the input (URL domain+path or topic string)

### Examples

```bash
# Uses default: ~/Documents/reference-library/glossary-for-wordpress/
/reference-curator https://docs.codeat.co/glossary/

# Custom path: /tmp/research/mcp-docs/
/reference-curator "MCP servers" --output /tmp/research

# Explicit home subfolder: ~/Projects/client-docs/api-reference/
/reference-curator https://api.example.com/docs --output ~/Projects/client-docs
```

## Depth Levels

### `--depth light`
Fast scan for quick reference. Main content only, minimal crawling.

| Parameter | Value |
|-----------|-------|
| `onlyMainContent` | `true` |
| `formats` | `["markdown"]` |
| `maxDiscoveryDepth` | 1 |
| `max-pages` default | 20 |
| Map limit | 50 |
| `deduplicateSimilarURLs` | `true` |
| Follow sub-links | No |
| JS rendering wait | None |

**Best for:** Quick lookups, single-page references, API docs you already know.

### `--depth standard` (default)
Balanced crawl. Main content with links for cross-referencing.

| Parameter | Value |
|-----------|-------|
| `onlyMainContent` | `true` |
| `formats` | `["markdown", "links"]` |
| `maxDiscoveryDepth` | 2 |
| `max-pages` default | 50 |
| Map limit | 100 |
| `deduplicateSimilarURLs` | `true` |
| Follow sub-links | Same-domain only |
| JS rendering wait | None |

**Best for:** Documentation sites, plugin guides, knowledge bases.

### `--depth deep`
Thorough crawl. Full page content including sidebars, nav, and related pages.

| Parameter | Value |
|-----------|-------|
| `onlyMainContent` | `false` |
| `formats` | `["markdown", "links", "html"]` |
| `maxDiscoveryDepth` | 3 |
| `max-pages` default | 150 |
| Map limit | 300 |
| `deduplicateSimilarURLs` | `true` |
| Follow sub-links | Same-domain + linked resources |
| JS rendering wait | `waitFor: 3000` |
| `includeSubdomains` | `true` |

**Best for:** Complete product documentation, research material, sites with sidebars/code samples hidden behind tabs.

### `--depth full`
Exhaustive crawl. Everything captured including raw HTML, screenshots, and external references.

| Parameter | Value |
|-----------|-------|
| `onlyMainContent` | `false` |
| `formats` | `["markdown", "html", "rawHtml", "links"]` |
| `maxDiscoveryDepth` | 5 |
| `max-pages` default | 500 |
| Map limit | 1000 |
| `deduplicateSimilarURLs` | `false` |
| Follow sub-links | All (same-domain + external references) |
| JS rendering wait | `waitFor: 5000` |
| `includeSubdomains` | `true` |
| Screenshots | Capture for JS-heavy pages |

**Best for:** Archival, migration references, preserving sites, training data collection.

## Depth Comparison

```
light       ████░░░░░░░░░░░░  Speed: fastest   Pages: ~20    Content: main only
standard    ████████░░░░░░░░  Speed: fast       Pages: ~50    Content: main + links
deep        ████████████░░░░  Speed: moderate   Pages: ~150   Content: full page + HTML
full        ████████████████  Speed: slow       Pages: ~500   Content: everything + raw
```

## Pipeline Stages

```
1. reference-discovery  (topic mode only)
2. web-crawler              ← depth controls this stage
                            ← --save-raw: writes raw/ alongside crawl
3. content-repository
4. content-distiller    <--------+    ← SKIPPED when --no-distill
5. quality-reviewer              |    ← SKIPPED when --no-distill
   +-- APPROVE -> export         |
   +-- REFACTOR -----------------+
   +-- DEEP_RESEARCH -> crawler -+
   +-- REJECT -> archive
6. markdown-exporter            ← --no-distill: exports raw files only
                                ← --save-raw: also exports raw/ bundle
```

## Crawl Execution by Depth

When executing the crawl (Stage 2), apply the depth settings to the Firecrawl tools:

### Site Mapping (`firecrawl_map`)
```
firecrawl_map:
  url: <target>
  limit: {depth.map_limit}
  includeSubdomains: {depth.includeSubdomains}
```

### Page Scraping (`firecrawl_scrape`)
```
firecrawl_scrape:
  url: <page>
  formats: {depth.formats}
  onlyMainContent: {depth.onlyMainContent}
  waitFor: {depth.waitFor}           # deep/full only
  excludeTags: ["nav", "footer"]     # light/standard only
```

### Batch Crawling (`firecrawl_crawl`) - for deep/full only
```
firecrawl_crawl:
  url: <target>
  maxDiscoveryDepth: {depth.maxDiscoveryDepth}
  limit: {depth.max_pages}
  deduplicateSimilarURLs: {depth.dedup}
  scrapeOptions:
    formats: {depth.formats}
    onlyMainContent: {depth.onlyMainContent}
    waitFor: {depth.waitFor}
```

## QA Loop Handling

| Decision | Action | Max Iterations |
|----------|--------|----------------|
| REFACTOR | Re-distill with feedback | 3 |
| DEEP_RESEARCH | Crawl more sources, re-distill | 2 |
| Combined | Total loops per document | 5 |

After max iterations, document marked as `needs_manual_review`.

## Example Usage

```
# Quick scan of a single doc page
/reference-curator https://docs.example.com/api --depth light

# Standard documentation crawl (default)
/reference-curator "Glossary for WordPress" --max-sources 5

# Deep crawl capturing full page content and HTML
/reference-curator https://docs.codeat.co/glossary/ --depth deep

# Full archival crawl with all formats
/reference-curator https://docs.anthropic.com --depth full --max-pages 300

# Deep crawl with auto-approval and fine-tuning export
/reference-curator "MCP servers" --depth deep --auto-approve --export-format fine_tuning

# Resume from existing manifest
/reference-curator ./manifest.json --auto-approve

# Save raw crawled content alongside distilled output
/reference-curator https://docs.anthropic.com --depth standard --save-raw

# Pure archival — no distillation, just raw markdown files
/reference-curator https://docs.anthropic.com --depth full --no-distill
```

## State Management

Pipeline state is saved after each stage to allow resume:

**With MySQL:**
```sql
SELECT * FROM pipeline_runs WHERE run_id = 123;
```

**File-based fallback:**
```
~/Documents/reference-library/pipeline_state/run_XXX/state.json
```

## Output

Pipeline returns summary on completion:

```json
{
  "run_id": 123,
  "status": "completed",
  "depth": "standard",
  "stats": {
    "sources_discovered": 5,
    "pages_crawled": 45,
    "documents_stored": 45,
    "approved": 40,
    "refactored": 8,
    "deep_researched": 2,
    "rejected": 3,
    "needs_manual_review": 2
  },
  "exports": {
    "format": "project_files",
    "path": "~/Documents/reference-library/exports/"
  }
}
```

## See Also

- `/reference-discovery` - Run discovery stage only
- `/web-crawler` - Run crawler stage only
- `/content-repository` - Manage stored content
- `/quality-reviewer` - Run QA review only
- `/markdown-exporter` - Run export only