feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-12 18:19:52 +09:00
parent 133df68b81
commit f215c11c32
23 changed files with 3917 additions and 583 deletions

View File

@@ -1,46 +1,42 @@
# Content Repository
MySQL storage management for the reference library. Handles document storage, version control, deduplication, and retrieval.
MySQL storage management for the reference library. Handles document storage, version control, deduplication, and retrieval. Supports MySQL primary backend with JSON file fallback.
## Trigger Keywords
"store content", "save to database", "check duplicates", "version tracking", "document retrieval", "reference library DB"
## Prerequisites
- MySQL 8.0+ with utf8mb4 charset
- Config file at `~/.config/reference-curator/db_config.yaml`
- Database `reference_library` initialized
- `refcurator` package installed (`uv pip install -e shared/lib/`)
- MySQL 8.0+ (optional — falls back to JSON file storage)
- Config at `~/.config/reference-curator/db_config.yaml` (optional)
## Database Setup
## Quick Start
```bash
# Initialize database
mysql -u root -p < references/schema.sql
# Verify tables
mysql -u root -p reference_library -e "SHOW TABLES;"
```
## Core Scripts
### Store Document
```bash
python scripts/store_document.py \
--source-id 1 \
--title "Prompt Engineering Guide" \
# Store a document
uv run python scripts/repo.py store \
--source-id 1 --title "Prompt Engineering Guide" \
--url "https://docs.anthropic.com/..." \
--doc-type webpage \
--raw-path ~/reference-library/raw/2025/01/abc123.md
```
--doc-type webpage --raw-path ~/reference-library/raw/abc123.md
### Check Duplicate
```bash
python scripts/check_duplicate.py --url "https://docs.anthropic.com/..."
```
# Check for duplicates
uv run python scripts/repo.py check-dup --url "https://docs.anthropic.com/..."
### Query by Topic
```bash
python scripts/query_topic.py --topic-slug prompt-engineering --min-quality 0.80
# Query by topic
uv run python scripts/repo.py query-topic --topic-slug prompt-engineering --min-quality 0.80
# Get repository stats
uv run python scripts/repo.py stats
# Find stale documents (older than 30 days)
uv run python scripts/repo.py find-stale --days 30
# Get pending reviews
uv run python scripts/repo.py pending-reviews --output pending.json
# Get export-ready content
uv run python scripts/repo.py export-ready --min-score 0.85
```
## Table Quick Reference
@@ -61,31 +57,17 @@ python scripts/query_topic.py --topic-slug prompt-engineering --min-quality 0.80
**review_status:** `pending``in_review``approved` | `needs_refactor` | `rejected`
## Common Queries
### Find Stale Documents
```bash
python scripts/find_stale.py --output stale_docs.json
```
### Get Pending Reviews
```bash
python scripts/pending_reviews.py --output pending.json
```
### Export-Ready Content
```bash
python scripts/export_ready.py --min-score 0.85 --output ready.json
```
## Scripts
- `scripts/store_document.py` - Store new document
- `scripts/check_duplicate.py` - URL deduplication
- `scripts/query_topic.py` - Query by topic
- `scripts/find_stale.py` - Find stale documents
- `scripts/pending_reviews.py` - Get pending reviews
- `scripts/db_utils.py` - Database connection utilities
| Command | Purpose |
|---------|---------|
| `repo.py store` | Store a new document |
| `repo.py check-dup` | URL deduplication check |
| `repo.py query-topic` | Query documents by topic |
| `repo.py find-stale` | Find stale documents |
| `repo.py pending-reviews` | Get pending reviews |
| `repo.py export-ready` | Get approved content ready for export |
| `repo.py stats` | Show repository statistics |
## Integration