# Content Distiller

Analyzes and distills raw crawled content into concise reference materials. Claude performs the actual distillation (summarization, key concept extraction). Scripts handle data loading and storage.

## Trigger Keywords
"distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary"

## Goals

1. **Compress** — Reduce token count while preserving essential information
2. **Structure** — Organize content for easy retrieval
3. **Extract** — Pull out code snippets, key concepts, patterns
4. **Annotate** — Add metadata for searchability

## Workflow

### Step 1: Load Pending Documents
```bash
uv run python scripts/distiller.py load-pending --output pending.json
```

### Step 2: Analyze and Distill (Claude)
For each pending document, Claude reads the raw content and creates:
- Executive summary (2-3 sentences)
- Key concepts with definitions
- Techniques and patterns
- Code examples
- Best practices

### Step 3: Store Distilled Content
```bash
uv run python scripts/distiller.py store \
  --doc-id 123 \
  --content distilled.md \
  --summary summary.txt \
  --concepts concepts.json \
  --snippets snippets.json \
  --model claude-opus-4-6
```

### Step 4: Handle Refactor Requests
When quality-reviewer returns `refactor`, load context for re-distillation:
```bash
uv run python scripts/distiller.py refactor --distill-id 456 --output context.json
```

This outputs a context bundle with the current distilled content, raw source, and all review feedback.

### Step 5: View Distilled Content
```bash
uv run python scripts/distiller.py show --distill-id 456
```

## Distilled Output Template

```markdown
# {title}

**Source:** {url}
**Type:** {source_type} | **Tier:** {credibility_tier}
**Distilled:** {date}

## Executive Summary
{2-3 sentence overview}

## Key Concepts
{bulleted list with definitions}

## Techniques & Patterns
{extracted techniques with use cases}

## Code Examples
{relevant code snippets}

## Best Practices
{actionable recommendations}
```

## Quality Metrics

| Metric | Target |
|--------|--------|
| Compression Ratio | 25-35% of original |
| Key Concept Coverage | ≥90% of important terms |
| Code Snippet Retention | 100% of relevant examples |
| Readability | Clear, scannable structure |

## Scripts

| Command | Purpose |
|---------|---------|
| `distiller.py load-pending` | Load documents pending distillation |
| `distiller.py store` | Save distilled content to DB |
| `distiller.py refactor` | Load context for re-distillation |
| `distiller.py show` | Show distilled content details |

## Integration

| From | To |
|------|-----|
| content-repository | Raw document records |
| → | quality-reviewer (distilled content) |
| quality-reviewer | Refactor instructions (loop back) |