feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-12 18:19:52 +09:00
parent 133df68b81
commit f215c11c32
23 changed files with 3917 additions and 583 deletions

View File

@@ -9,57 +9,51 @@ Exports approved reference content as structured markdown files for project know
| Type | Format | Use Case |
|------|--------|----------|
| `project_files` | Nested markdown | Claude Projects knowledge |
| `fine_tuning` | JSONL | Model fine-tuning dataset |
| `knowledge_base` | Flat markdown | Documentation |
| `project` | Nested markdown | Claude Projects knowledge |
| `finetuning` | JSONL | Model fine-tuning dataset |
## Workflow
### Step 1: Query Approved Content
### Step 1: Export Project Files
```bash
python scripts/query_approved.py --min-score 0.80 --output approved.json
uv run python scripts/exporter.py project \
--output ~/reference-library/exports/ \
--min-score 0.80 \
--structure nested_by_topic
```
### Step 2: Organize by Structure
**Nested by Topic (default):**
Output structure:
```
exports/
├── INDEX.md
├── prompt-engineering/
│ ├── _index.md
│ ├── 01-chain-of-thought.md
│ └── 02-few-shot-prompting.md
│ ├── 00-chain-of-thought.md
│ └── 01-few-shot-prompting.md
└── claude-models/
├── _index.md
└── 01-model-comparison.md
└── 00-model-comparison.md
```
**Flat Structure:**
```
exports/
├── INDEX.md
├── prompt-engineering-chain-of-thought.md
└── claude-models-comparison.md
```
### Step 3: Generate Files
### Step 2: Generate INDEX
```bash
python scripts/export_project.py \
--structure nested_by_topic \
--output ~/reference-library/exports/ \
--include-metadata
uv run python scripts/exporter.py index --output ~/reference-library/exports/INDEX.md
```
### Step 4: Generate INDEX
### Step 3: Add Cross-References
```bash
python scripts/generate_index.py --output ~/reference-library/exports/INDEX.md
uv run python scripts/exporter.py crossrefs --input ~/reference-library/exports/
```
### Step 4: Verify Export
```bash
uv run python scripts/exporter.py verify --path ~/reference-library/exports/
```
### Step 5: Fine-tuning Export (Optional)
```bash
python scripts/export_finetuning.py \
--output ~/reference-library/exports/fine_tuning.jsonl \
uv run python scripts/exporter.py finetuning \
--output ~/reference-library/exports/training.jsonl \
--max-tokens 4096
```
@@ -71,62 +65,25 @@ JSONL format:
{"role": "user", "content": "Explain {title}"},
{"role": "assistant", "content": "{structured_content}"}
],
"metadata": {"source": "{url}", "topic": "{topic_slug}", "quality_score": 0.92}
"metadata": {"source": "{url}", "quality_score": 0.92}
}
```
### Step 6: Log Export Job
```bash
python scripts/log_export.py --name "January 2025 Export" --type project_files --docs 45
```
## Cross-Reference Generation
```bash
python scripts/add_crossrefs.py --input ~/reference-library/exports/
```
Links related documents based on overlapping key concepts.
## Output Verification
After export, verify:
- [ ] All files readable and valid markdown
- [ ] INDEX.md links resolve correctly
- [ ] No broken cross-references
- [ ] Total token count matches expectation
- [ ] No duplicate content
```bash
python scripts/verify_export.py --path ~/reference-library/exports/
uv run python scripts/exporter.py log --name "April 2026 Export" --type project_files --docs 45
```
## Scripts
- `scripts/query_approved.py` - Get approved content from DB
- `scripts/export_project.py` - Main export for project files
- `scripts/export_finetuning.py` - JSONL export for fine-tuning
- `scripts/generate_index.py` - Generate INDEX.md
- `scripts/add_crossrefs.py` - Add cross-references
- `scripts/log_export.py` - Log export job to DB
- `scripts/verify_export.py` - Verify export integrity
## Configuration
```yaml
# ~/.config/reference-curator/export_config.yaml
output:
base_path: ~/reference-library/exports/
project_files:
structure: nested_by_topic
index_file: INDEX.md
include_metadata: true
fine_tuning:
format: jsonl
max_tokens_per_sample: 4096
quality:
min_score_for_export: 0.80
```
| Command | Purpose |
|---------|---------|
| `exporter.py project` | Export as nested markdown files |
| `exporter.py finetuning` | Export as JSONL training dataset |
| `exporter.py index` | Generate INDEX.md table of contents |
| `exporter.py crossrefs` | Add cross-reference links |
| `exporter.py verify` | Verify export integrity |
| `exporter.py log` | Log export job to DB |
## Integration