# Markdown Exporter

Exports approved reference content as structured markdown files for project knowledge or fine-tuning datasets.

## Trigger Keywords
"export references", "generate project files", "create markdown output", "export for fine-tuning", "build knowledge base"

## Export Types

| Type | Format | Use Case |
|------|--------|----------|
| `project` | Nested markdown | Claude Projects knowledge |
| `raw` | Raw markdown | Archival, full-text reference, NotebookLM sources |
| `finetuning` | JSONL | Model fine-tuning dataset |

## Workflow

### Step 1: Export Project Files
```bash
uv run python scripts/exporter.py project \
  --output ~/reference-library/exports/ \
  --min-score 0.80 \
  --structure nested_by_topic
```

Output structure:
```
exports/
├── INDEX.md
├── prompt-engineering/
│   ├── _index.md
│   ├── 00-chain-of-thought.md
│   └── 01-few-shot-prompting.md
└── claude-models/
    ├── _index.md
    └── 00-model-comparison.md
```

### Step 2: Generate INDEX
```bash
uv run python scripts/exporter.py index --output ~/reference-library/exports/INDEX.md
```

### Step 3: Add Cross-References
```bash
uv run python scripts/exporter.py crossrefs --input ~/reference-library/exports/
```

### Step 4: Verify Export
```bash
uv run python scripts/exporter.py verify --path ~/reference-library/exports/
```

### Step 5: Fine-tuning Export (Optional)
```bash
uv run python scripts/exporter.py finetuning \
  --output ~/reference-library/exports/training.jsonl \
  --max-tokens 4096
```

JSONL format:
```json
{
  "messages": [
    {"role": "system", "content": "You are an expert on AI and prompt engineering."},
    {"role": "user", "content": "Explain {title}"},
    {"role": "assistant", "content": "{structured_content}"}
  ],
  "metadata": {"source": "{url}", "quality_score": 0.92}
}
```

### Step 6: Log Export Job
```bash
uv run python scripts/exporter.py log --name "April 2026 Export" --type project_files --docs 45
```

### Raw Export (when `--save-raw` or `--no-distill`)

#### Raw alongside distilled (`--save-raw`)
```bash
uv run python scripts/exporter.py raw \
  --output ~/reference-library/exports/{topic-slug}/raw/ \
  --source-dir {crawl-raw-dir} \
  --bundle {topic-slug}-raw-complete.md
```

Output structure:
```
exports/{topic-slug}/
├── INDEX.md              # Includes links to raw/ section
├── 00-page-name.md       # Distilled (normal)
├── raw/
│   ├── 00-page-name.raw.md
│   ├── 01-page-name.raw.md
│   └── {topic-slug}-raw-complete.md
└── manifest.json
```

#### Raw-only mode (`--no-distill`)
```bash
uv run python scripts/exporter.py raw \
  --output ~/reference-library/exports/{topic-slug}/ \
  --source-dir {crawl-raw-dir} \
  --bundle {topic-slug}-raw-complete.md \
  --primary
```

Output structure (raw files are primary, no subdirectory wrapper):
```
exports/{topic-slug}/
├── README.md
├── 00-page-name.raw.md
├── 01-page-name.raw.md
├── {topic-slug}-raw-complete.md
└── manifest.json
```

The `--primary` flag tells the exporter that raw files are the main output. README is generated from raw file frontmatter (source URLs, crawl timestamps).

**Note:** When combined with `--export-format fine_tuning`, the JSONL exporter uses raw content directly as the assistant response with a simpler system prompt (no structured summaries available).

## Scripts

| Command | Purpose |
|---------|---------|
| `exporter.py project` | Export as nested markdown files |
| `exporter.py raw` | Export raw crawled markdown files with optional bundle |
| `exporter.py finetuning` | Export as JSONL training dataset |
| `exporter.py index` | Generate INDEX.md table of contents |
| `exporter.py crossrefs` | Add cross-reference links |
| `exporter.py verify` | Verify export integrity |
| `exporter.py log` | Log export job to DB |

## Integration

| From | To |
|------|-----|
| quality-reviewer (approved) | → |
| web-crawler (raw files) | Raw markdown input (--save-raw / --no-distill) |
| → | Project knowledge / Fine-tuning dataset |