Files

Andrew Yim b1c2dca080 feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration

- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory
  alongside distilled content, with YAML frontmatter for traceability
- Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer),
  pure archival mode for raw documentation capture
- Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend:
  token-bucket rate limiting, semaphore concurrency, tenacity retries,
  full page metadata extraction (headings, links, schema, OG)
- New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests
  with progress tracking and resume support
- Update crawler selection: docs sites now default to seo-aiohttp
- Update crawl_config.yaml with seo-aiohttp routing rules
- Update pipeline orchestrator with flag resolution and skip paths
- Update README.md and USER-GUIDE.md with raw mode documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 23:26:50 +09:00

3.2 KiB

Raw Blame History

Content Distiller

Analyzes and distills raw crawled content into concise reference materials. Claude performs the actual distillation (summarization, key concept extraction). Scripts handle data loading and storage.

Trigger Keywords

"distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary"

Skip Condition

This entire stage is skipped when --no-distill is active.

When the pipeline orchestrator passes --no-distill, this stage returns immediately with status skipped. The pipeline advances directly from Stage 3 (content-repository) to Stage 6 (markdown-exporter), and raw crawled content becomes the export source instead of distilled content.

No documents are loaded, no distillation occurs, and no records are written to the distilled_content table.

Goals

Compress — Reduce token count while preserving essential information
Structure — Organize content for easy retrieval
Extract — Pull out code snippets, key concepts, patterns
Annotate — Add metadata for searchability

Workflow

Step 1: Load Pending Documents

uv run python scripts/distiller.py load-pending --output pending.json

Step 2: Analyze and Distill (Claude)

For each pending document, Claude reads the raw content and creates:

Executive summary (2-3 sentences)
Key concepts with definitions
Techniques and patterns
Code examples
Best practices

Step 3: Store Distilled Content

uv run python scripts/distiller.py store \
  --doc-id 123 \
  --content distilled.md \
  --summary summary.txt \
  --concepts concepts.json \
  --snippets snippets.json \
  --model claude-opus-4-6

Step 4: Handle Refactor Requests

When quality-reviewer returns refactor, load context for re-distillation:

uv run python scripts/distiller.py refactor --distill-id 456 --output context.json

This outputs a context bundle with the current distilled content, raw source, and all review feedback.

Step 5: View Distilled Content

uv run python scripts/distiller.py show --distill-id 456

Distilled Output Template

# {title}

**Source:** {url}
**Type:** {source_type} | **Tier:** {credibility_tier}
**Distilled:** {date}

## Executive Summary
{2-3 sentence overview}

## Key Concepts
{bulleted list with definitions}

## Techniques & Patterns
{extracted techniques with use cases}

## Code Examples
{relevant code snippets}

## Best Practices
{actionable recommendations}

Quality Metrics

Metric	Target
Compression Ratio	25-35% of original
Key Concept Coverage	≥90% of important terms
Code Snippet Retention	100% of relevant examples
Readability	Clear, scannable structure

Scripts

Command	Purpose
`distiller.py load-pending`	Load documents pending distillation
`distiller.py store`	Save distilled content to DB
`distiller.py refactor`	Load context for re-distillation
`distiller.py show`	Show distilled content details

Integration

From	To
content-repository	Raw document records
→	quality-reviewer (distilled content)
quality-reviewer	Refactor instructions (loop back)

3.2 KiB Raw Blame History