Files

Andrew Yim f215c11c32 feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 18:22:28 +09:00

2.7 KiB

Raw Blame History

Content Distiller

Analyzes and distills raw crawled content into concise reference materials. Claude performs the actual distillation (summarization, key concept extraction). Scripts handle data loading and storage.

Trigger Keywords

"distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary"

Goals

Compress — Reduce token count while preserving essential information
Structure — Organize content for easy retrieval
Extract — Pull out code snippets, key concepts, patterns
Annotate — Add metadata for searchability

Workflow

Step 1: Load Pending Documents

uv run python scripts/distiller.py load-pending --output pending.json

Step 2: Analyze and Distill (Claude)

For each pending document, Claude reads the raw content and creates:

Executive summary (2-3 sentences)
Key concepts with definitions
Techniques and patterns
Code examples
Best practices

Step 3: Store Distilled Content

uv run python scripts/distiller.py store \
  --doc-id 123 \
  --content distilled.md \
  --summary summary.txt \
  --concepts concepts.json \
  --snippets snippets.json \
  --model claude-opus-4-6

Step 4: Handle Refactor Requests

When quality-reviewer returns refactor, load context for re-distillation:

uv run python scripts/distiller.py refactor --distill-id 456 --output context.json

This outputs a context bundle with the current distilled content, raw source, and all review feedback.

Step 5: View Distilled Content

uv run python scripts/distiller.py show --distill-id 456

Distilled Output Template

# {title}

**Source:** {url}
**Type:** {source_type} | **Tier:** {credibility_tier}
**Distilled:** {date}

## Executive Summary
{2-3 sentence overview}

## Key Concepts
{bulleted list with definitions}

## Techniques & Patterns
{extracted techniques with use cases}

## Code Examples
{relevant code snippets}

## Best Practices
{actionable recommendations}

Quality Metrics

Metric	Target
Compression Ratio	25-35% of original
Key Concept Coverage	≥90% of important terms
Code Snippet Retention	100% of relevant examples
Readability	Clear, scannable structure

Scripts

Command	Purpose
`distiller.py load-pending`	Load documents pending distillation
`distiller.py store`	Save distilled content to DB
`distiller.py refactor`	Load context for re-distillation
`distiller.py show`	Show distilled content details

Integration

From	To
content-repository	Raw document records
→	quality-reviewer (distilled content)
quality-reviewer	Refactor instructions (loop back)

2.7 KiB Raw Blame History