Andrew Yim
b1c2dca080
- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory alongside distilled content, with YAML frontmatter for traceability - Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer), pure archival mode for raw documentation capture - Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend: token-bucket rate limiting, semaphore concurrency, tenacity retries, full page metadata extraction (headings, links, schema, OG) - New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests with progress tracking and resume support - Update crawler selection: docs sites now default to seo-aiohttp - Update crawl_config.yaml with seo-aiohttp routing rules - Update pipeline orchestrator with flag resolution and skip paths - Update README.md and USER-GUIDE.md with raw mode documentation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
112 lines
3.2 KiB
Markdown
112 lines
3.2 KiB
Markdown
# Content Distiller
|
|
|
|
Analyzes and distills raw crawled content into concise reference materials. Claude performs the actual distillation (summarization, key concept extraction). Scripts handle data loading and storage.
|
|
|
|
## Trigger Keywords
|
|
"distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary"
|
|
|
|
## Skip Condition
|
|
|
|
**This entire stage is skipped when `--no-distill` is active.**
|
|
|
|
When the pipeline orchestrator passes `--no-distill`, this stage returns immediately with status `skipped`. The pipeline advances directly from Stage 3 (content-repository) to Stage 6 (markdown-exporter), and raw crawled content becomes the export source instead of distilled content.
|
|
|
|
No documents are loaded, no distillation occurs, and no records are written to the `distilled_content` table.
|
|
|
|
## Goals
|
|
|
|
1. **Compress** — Reduce token count while preserving essential information
|
|
2. **Structure** — Organize content for easy retrieval
|
|
3. **Extract** — Pull out code snippets, key concepts, patterns
|
|
4. **Annotate** — Add metadata for searchability
|
|
|
|
## Workflow
|
|
|
|
### Step 1: Load Pending Documents
|
|
```bash
|
|
uv run python scripts/distiller.py load-pending --output pending.json
|
|
```
|
|
|
|
### Step 2: Analyze and Distill (Claude)
|
|
For each pending document, Claude reads the raw content and creates:
|
|
- Executive summary (2-3 sentences)
|
|
- Key concepts with definitions
|
|
- Techniques and patterns
|
|
- Code examples
|
|
- Best practices
|
|
|
|
### Step 3: Store Distilled Content
|
|
```bash
|
|
uv run python scripts/distiller.py store \
|
|
--doc-id 123 \
|
|
--content distilled.md \
|
|
--summary summary.txt \
|
|
--concepts concepts.json \
|
|
--snippets snippets.json \
|
|
--model claude-opus-4-6
|
|
```
|
|
|
|
### Step 4: Handle Refactor Requests
|
|
When quality-reviewer returns `refactor`, load context for re-distillation:
|
|
```bash
|
|
uv run python scripts/distiller.py refactor --distill-id 456 --output context.json
|
|
```
|
|
|
|
This outputs a context bundle with the current distilled content, raw source, and all review feedback.
|
|
|
|
### Step 5: View Distilled Content
|
|
```bash
|
|
uv run python scripts/distiller.py show --distill-id 456
|
|
```
|
|
|
|
## Distilled Output Template
|
|
|
|
```markdown
|
|
# {title}
|
|
|
|
**Source:** {url}
|
|
**Type:** {source_type} | **Tier:** {credibility_tier}
|
|
**Distilled:** {date}
|
|
|
|
## Executive Summary
|
|
{2-3 sentence overview}
|
|
|
|
## Key Concepts
|
|
{bulleted list with definitions}
|
|
|
|
## Techniques & Patterns
|
|
{extracted techniques with use cases}
|
|
|
|
## Code Examples
|
|
{relevant code snippets}
|
|
|
|
## Best Practices
|
|
{actionable recommendations}
|
|
```
|
|
|
|
## Quality Metrics
|
|
|
|
| Metric | Target |
|
|
|--------|--------|
|
|
| Compression Ratio | 25-35% of original |
|
|
| Key Concept Coverage | ≥90% of important terms |
|
|
| Code Snippet Retention | 100% of relevant examples |
|
|
| Readability | Clear, scannable structure |
|
|
|
|
## Scripts
|
|
|
|
| Command | Purpose |
|
|
|---------|---------|
|
|
| `distiller.py load-pending` | Load documents pending distillation |
|
|
| `distiller.py store` | Save distilled content to DB |
|
|
| `distiller.py refactor` | Load context for re-distillation |
|
|
| `distiller.py show` | Show distilled content details |
|
|
|
|
## Integration
|
|
|
|
| From | To |
|
|
|------|-----|
|
|
| content-repository | Raw document records |
|
|
| → | quality-reviewer (distilled content) |
|
|
| quality-reviewer | Refactor instructions (loop back) |
|