feat(reference-curator): implement Python scripts + Gemini quality gate
Build the refcurator shared Python package and 7 CLI scripts that were previously specification-only. Add Gemini CLI as an independent pre-distillation quality evaluator, replacing the circular Claude-self-review pattern. Key changes: - shared/lib/src/refcurator/: 7-module package (config, db, models, utils, manifest, gemini) with PyMySQL + JSON file dual backend - 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer, exporter, pipeline — each with subcommands for data management - Gemini quality gate: evaluates raw content BEFORE distillation using 5 criteria (relevance, authority, completeness, freshness, distill_value) - Pipeline reordered: discovery → crawl → store → evaluate → distill → export - Bug fixes from Codex adversarial review: - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries - Exporter uses MAX(review_id) to prevent shipping stale approvals - Distiller updates existing rows on refactor instead of forking - Updated all 7 CLAUDE.md directives with real script references - install.sh updated with refcurator package install step 51/51 E2E tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,46 +1,42 @@
|
||||
# Content Repository
|
||||
|
||||
MySQL storage management for the reference library. Handles document storage, version control, deduplication, and retrieval.
|
||||
MySQL storage management for the reference library. Handles document storage, version control, deduplication, and retrieval. Supports MySQL primary backend with JSON file fallback.
|
||||
|
||||
## Trigger Keywords
|
||||
"store content", "save to database", "check duplicates", "version tracking", "document retrieval", "reference library DB"
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- MySQL 8.0+ with utf8mb4 charset
|
||||
- Config file at `~/.config/reference-curator/db_config.yaml`
|
||||
- Database `reference_library` initialized
|
||||
- `refcurator` package installed (`uv pip install -e shared/lib/`)
|
||||
- MySQL 8.0+ (optional — falls back to JSON file storage)
|
||||
- Config at `~/.config/reference-curator/db_config.yaml` (optional)
|
||||
|
||||
## Database Setup
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Initialize database
|
||||
mysql -u root -p < references/schema.sql
|
||||
|
||||
# Verify tables
|
||||
mysql -u root -p reference_library -e "SHOW TABLES;"
|
||||
```
|
||||
|
||||
## Core Scripts
|
||||
|
||||
### Store Document
|
||||
```bash
|
||||
python scripts/store_document.py \
|
||||
--source-id 1 \
|
||||
--title "Prompt Engineering Guide" \
|
||||
# Store a document
|
||||
uv run python scripts/repo.py store \
|
||||
--source-id 1 --title "Prompt Engineering Guide" \
|
||||
--url "https://docs.anthropic.com/..." \
|
||||
--doc-type webpage \
|
||||
--raw-path ~/reference-library/raw/2025/01/abc123.md
|
||||
```
|
||||
--doc-type webpage --raw-path ~/reference-library/raw/abc123.md
|
||||
|
||||
### Check Duplicate
|
||||
```bash
|
||||
python scripts/check_duplicate.py --url "https://docs.anthropic.com/..."
|
||||
```
|
||||
# Check for duplicates
|
||||
uv run python scripts/repo.py check-dup --url "https://docs.anthropic.com/..."
|
||||
|
||||
### Query by Topic
|
||||
```bash
|
||||
python scripts/query_topic.py --topic-slug prompt-engineering --min-quality 0.80
|
||||
# Query by topic
|
||||
uv run python scripts/repo.py query-topic --topic-slug prompt-engineering --min-quality 0.80
|
||||
|
||||
# Get repository stats
|
||||
uv run python scripts/repo.py stats
|
||||
|
||||
# Find stale documents (older than 30 days)
|
||||
uv run python scripts/repo.py find-stale --days 30
|
||||
|
||||
# Get pending reviews
|
||||
uv run python scripts/repo.py pending-reviews --output pending.json
|
||||
|
||||
# Get export-ready content
|
||||
uv run python scripts/repo.py export-ready --min-score 0.85
|
||||
```
|
||||
|
||||
## Table Quick Reference
|
||||
@@ -61,31 +57,17 @@ python scripts/query_topic.py --topic-slug prompt-engineering --min-quality 0.80
|
||||
|
||||
**review_status:** `pending` → `in_review` → `approved` | `needs_refactor` | `rejected`
|
||||
|
||||
## Common Queries
|
||||
|
||||
### Find Stale Documents
|
||||
```bash
|
||||
python scripts/find_stale.py --output stale_docs.json
|
||||
```
|
||||
|
||||
### Get Pending Reviews
|
||||
```bash
|
||||
python scripts/pending_reviews.py --output pending.json
|
||||
```
|
||||
|
||||
### Export-Ready Content
|
||||
```bash
|
||||
python scripts/export_ready.py --min-score 0.85 --output ready.json
|
||||
```
|
||||
|
||||
## Scripts
|
||||
|
||||
- `scripts/store_document.py` - Store new document
|
||||
- `scripts/check_duplicate.py` - URL deduplication
|
||||
- `scripts/query_topic.py` - Query by topic
|
||||
- `scripts/find_stale.py` - Find stale documents
|
||||
- `scripts/pending_reviews.py` - Get pending reviews
|
||||
- `scripts/db_utils.py` - Database connection utilities
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `repo.py store` | Store a new document |
|
||||
| `repo.py check-dup` | URL deduplication check |
|
||||
| `repo.py query-topic` | Query documents by topic |
|
||||
| `repo.py find-stale` | Find stale documents |
|
||||
| `repo.py pending-reviews` | Get pending reviews |
|
||||
| `repo.py export-ready` | Get approved content ready for export |
|
||||
| `repo.py stats` | Show repository statistics |
|
||||
|
||||
## Integration
|
||||
|
||||
|
||||
Reference in New Issue
Block a user