feat(reference-curator): implement Python scripts + Gemini quality gate
Build the refcurator shared Python package and 7 CLI scripts that were previously specification-only. Add Gemini CLI as an independent pre-distillation quality evaluator, replacing the circular Claude-self-review pattern. Key changes: - shared/lib/src/refcurator/: 7-module package (config, db, models, utils, manifest, gemini) with PyMySQL + JSON file dual backend - 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer, exporter, pipeline — each with subcommands for data management - Gemini quality gate: evaluates raw content BEFORE distillation using 5 criteria (relevance, authority, completeness, freshness, distill_value) - Pipeline reordered: discovery → crawl → store → evaluate → distill → export - Bug fixes from Codex adversarial review: - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries - Exporter uses MAX(review_id) to prevent shipping stale approvals - Distiller updates existing rows on refactor instead of forking - Updated all 7 CLAUDE.md directives with real script references - install.sh updated with refcurator package install step 51/51 E2E tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,44 +1,58 @@
|
||||
# Content Distiller
|
||||
|
||||
Analyzes and distills raw crawled content into concise reference materials. Extracts key concepts, code snippets, and creates structured summaries.
|
||||
Analyzes and distills raw crawled content into concise reference materials. Claude performs the actual distillation (summarization, key concept extraction). Scripts handle data loading and storage.
|
||||
|
||||
## Trigger Keywords
|
||||
"distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary"
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Compress** - Reduce token count while preserving essential information
|
||||
2. **Structure** - Organize content for easy retrieval
|
||||
3. **Extract** - Pull out code snippets, key concepts, patterns
|
||||
4. **Annotate** - Add metadata for searchability
|
||||
1. **Compress** — Reduce token count while preserving essential information
|
||||
2. **Structure** — Organize content for easy retrieval
|
||||
3. **Extract** — Pull out code snippets, key concepts, patterns
|
||||
4. **Annotate** — Add metadata for searchability
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Load Raw Content
|
||||
### Step 1: Load Pending Documents
|
||||
```bash
|
||||
python scripts/load_pending.py --output pending_docs.json
|
||||
uv run python scripts/distiller.py load-pending --output pending.json
|
||||
```
|
||||
|
||||
### Step 2: Analyze Content Structure
|
||||
Identify document characteristics:
|
||||
- Has code blocks?
|
||||
- Has headers?
|
||||
- Has tables?
|
||||
- Estimated tokens?
|
||||
|
||||
### Step 3: Extract Key Components
|
||||
```bash
|
||||
python scripts/extract_components.py --doc-id 123 --output components.json
|
||||
```
|
||||
|
||||
Extracts:
|
||||
- Code snippets with language tags
|
||||
- Key concepts and definitions
|
||||
- Best practices
|
||||
### Step 2: Analyze and Distill (Claude)
|
||||
For each pending document, Claude reads the raw content and creates:
|
||||
- Executive summary (2-3 sentences)
|
||||
- Key concepts with definitions
|
||||
- Techniques and patterns
|
||||
- Code examples
|
||||
- Best practices
|
||||
|
||||
### Step 3: Store Distilled Content
|
||||
```bash
|
||||
uv run python scripts/distiller.py store \
|
||||
--doc-id 123 \
|
||||
--content distilled.md \
|
||||
--summary summary.txt \
|
||||
--concepts concepts.json \
|
||||
--snippets snippets.json \
|
||||
--model claude-opus-4-6
|
||||
```
|
||||
|
||||
### Step 4: Handle Refactor Requests
|
||||
When quality-reviewer returns `refactor`, load context for re-distillation:
|
||||
```bash
|
||||
uv run python scripts/distiller.py refactor --distill-id 456 --output context.json
|
||||
```
|
||||
|
||||
This outputs a context bundle with the current distilled content, raw source, and all review feedback.
|
||||
|
||||
### Step 5: View Distilled Content
|
||||
```bash
|
||||
uv run python scripts/distiller.py show --distill-id 456
|
||||
```
|
||||
|
||||
## Distilled Output Template
|
||||
|
||||
### Step 4: Create Structured Summary
|
||||
Output template:
|
||||
```markdown
|
||||
# {title}
|
||||
|
||||
@@ -62,17 +76,6 @@ Output template:
|
||||
{actionable recommendations}
|
||||
```
|
||||
|
||||
### Step 5: Optimize for Tokens
|
||||
Target: 25-35% of original token count
|
||||
```bash
|
||||
python scripts/optimize_content.py --doc-id 123 --target-ratio 0.30
|
||||
```
|
||||
|
||||
### Step 6: Store Distilled Content
|
||||
```bash
|
||||
python scripts/store_distilled.py --doc-id 123 --content distilled.md
|
||||
```
|
||||
|
||||
## Quality Metrics
|
||||
|
||||
| Metric | Target |
|
||||
@@ -82,20 +85,14 @@ python scripts/store_distilled.py --doc-id 123 --content distilled.md
|
||||
| Code Snippet Retention | 100% of relevant examples |
|
||||
| Readability | Clear, scannable structure |
|
||||
|
||||
## Handling Refactor Requests
|
||||
|
||||
When `quality-reviewer` returns `refactor`:
|
||||
```bash
|
||||
python scripts/refactor_content.py --distill-id 456 --instructions "Add more examples"
|
||||
```
|
||||
|
||||
## Scripts
|
||||
|
||||
- `scripts/load_pending.py` - Load documents pending distillation
|
||||
- `scripts/extract_components.py` - Extract code, concepts, patterns
|
||||
- `scripts/optimize_content.py` - Token optimization
|
||||
- `scripts/store_distilled.py` - Save to database
|
||||
- `scripts/refactor_content.py` - Handle refactor requests
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `distiller.py load-pending` | Load documents pending distillation |
|
||||
| `distiller.py store` | Save distilled content to DB |
|
||||
| `distiller.py refactor` | Load context for re-distillation |
|
||||
| `distiller.py show` | Show distilled content details |
|
||||
|
||||
## Integration
|
||||
|
||||
|
||||
Reference in New Issue
Block a user