feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-12 18:19:52 +09:00
parent 133df68b81
commit f215c11c32
23 changed files with 3917 additions and 583 deletions

View File

@@ -1,44 +1,58 @@
# Content Distiller
Analyzes and distills raw crawled content into concise reference materials. Extracts key concepts, code snippets, and creates structured summaries.
Analyzes and distills raw crawled content into concise reference materials. Claude performs the actual distillation (summarization, key concept extraction). Scripts handle data loading and storage.
## Trigger Keywords
"distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary"
## Goals
1. **Compress** - Reduce token count while preserving essential information
2. **Structure** - Organize content for easy retrieval
3. **Extract** - Pull out code snippets, key concepts, patterns
4. **Annotate** - Add metadata for searchability
1. **Compress** Reduce token count while preserving essential information
2. **Structure** Organize content for easy retrieval
3. **Extract** Pull out code snippets, key concepts, patterns
4. **Annotate** Add metadata for searchability
## Workflow
### Step 1: Load Raw Content
### Step 1: Load Pending Documents
```bash
python scripts/load_pending.py --output pending_docs.json
uv run python scripts/distiller.py load-pending --output pending.json
```
### Step 2: Analyze Content Structure
Identify document characteristics:
- Has code blocks?
- Has headers?
- Has tables?
- Estimated tokens?
### Step 3: Extract Key Components
```bash
python scripts/extract_components.py --doc-id 123 --output components.json
```
Extracts:
- Code snippets with language tags
- Key concepts and definitions
- Best practices
### Step 2: Analyze and Distill (Claude)
For each pending document, Claude reads the raw content and creates:
- Executive summary (2-3 sentences)
- Key concepts with definitions
- Techniques and patterns
- Code examples
- Best practices
### Step 3: Store Distilled Content
```bash
uv run python scripts/distiller.py store \
--doc-id 123 \
--content distilled.md \
--summary summary.txt \
--concepts concepts.json \
--snippets snippets.json \
--model claude-opus-4-6
```
### Step 4: Handle Refactor Requests
When quality-reviewer returns `refactor`, load context for re-distillation:
```bash
uv run python scripts/distiller.py refactor --distill-id 456 --output context.json
```
This outputs a context bundle with the current distilled content, raw source, and all review feedback.
### Step 5: View Distilled Content
```bash
uv run python scripts/distiller.py show --distill-id 456
```
## Distilled Output Template
### Step 4: Create Structured Summary
Output template:
```markdown
# {title}
@@ -62,17 +76,6 @@ Output template:
{actionable recommendations}
```
### Step 5: Optimize for Tokens
Target: 25-35% of original token count
```bash
python scripts/optimize_content.py --doc-id 123 --target-ratio 0.30
```
### Step 6: Store Distilled Content
```bash
python scripts/store_distilled.py --doc-id 123 --content distilled.md
```
## Quality Metrics
| Metric | Target |
@@ -82,20 +85,14 @@ python scripts/store_distilled.py --doc-id 123 --content distilled.md
| Code Snippet Retention | 100% of relevant examples |
| Readability | Clear, scannable structure |
## Handling Refactor Requests
When `quality-reviewer` returns `refactor`:
```bash
python scripts/refactor_content.py --distill-id 456 --instructions "Add more examples"
```
## Scripts
- `scripts/load_pending.py` - Load documents pending distillation
- `scripts/extract_components.py` - Extract code, concepts, patterns
- `scripts/optimize_content.py` - Token optimization
- `scripts/store_distilled.py` - Save to database
- `scripts/refactor_content.py` - Handle refactor requests
| Command | Purpose |
|---------|---------|
| `distiller.py load-pending` | Load documents pending distillation |
| `distiller.py store` | Save distilled content to DB |
| `distiller.py refactor` | Load context for re-distillation |
| `distiller.py show` | Show distilled content details |
## Integration