Andrew Yim
f215c11c32
Build the refcurator shared Python package and 7 CLI scripts that were previously specification-only. Add Gemini CLI as an independent pre-distillation quality evaluator, replacing the circular Claude-self-review pattern. Key changes: - shared/lib/src/refcurator/: 7-module package (config, db, models, utils, manifest, gemini) with PyMySQL + JSON file dual backend - 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer, exporter, pipeline — each with subcommands for data management - Gemini quality gate: evaluates raw content BEFORE distillation using 5 criteria (relevance, authority, completeness, freshness, distill_value) - Pipeline reordered: discovery → crawl → store → evaluate → distill → export - Bug fixes from Codex adversarial review: - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries - Exporter uses MAX(review_id) to prevent shipping stale approvals - Distiller updates existing rows on refactor instead of forking - Updated all 7 CLAUDE.md directives with real script references - install.sh updated with refcurator package install step 51/51 E2E tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
80 lines
2.7 KiB
Markdown
80 lines
2.7 KiB
Markdown
# Content Repository
|
|
|
|
MySQL storage management for the reference library. Handles document storage, version control, deduplication, and retrieval. Supports MySQL primary backend with JSON file fallback.
|
|
|
|
## Trigger Keywords
|
|
"store content", "save to database", "check duplicates", "version tracking", "document retrieval", "reference library DB"
|
|
|
|
## Prerequisites
|
|
|
|
- `refcurator` package installed (`uv pip install -e shared/lib/`)
|
|
- MySQL 8.0+ (optional — falls back to JSON file storage)
|
|
- Config at `~/.config/reference-curator/db_config.yaml` (optional)
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Store a document
|
|
uv run python scripts/repo.py store \
|
|
--source-id 1 --title "Prompt Engineering Guide" \
|
|
--url "https://docs.anthropic.com/..." \
|
|
--doc-type webpage --raw-path ~/reference-library/raw/abc123.md
|
|
|
|
# Check for duplicates
|
|
uv run python scripts/repo.py check-dup --url "https://docs.anthropic.com/..."
|
|
|
|
# Query by topic
|
|
uv run python scripts/repo.py query-topic --topic-slug prompt-engineering --min-quality 0.80
|
|
|
|
# Get repository stats
|
|
uv run python scripts/repo.py stats
|
|
|
|
# Find stale documents (older than 30 days)
|
|
uv run python scripts/repo.py find-stale --days 30
|
|
|
|
# Get pending reviews
|
|
uv run python scripts/repo.py pending-reviews --output pending.json
|
|
|
|
# Get export-ready content
|
|
uv run python scripts/repo.py export-ready --min-score 0.85
|
|
```
|
|
|
|
## Table Quick Reference
|
|
|
|
| Table | Purpose | Key Fields |
|
|
|-------|---------|------------|
|
|
| `sources` | Authorized sources | source_type, credibility_tier, vendor |
|
|
| `documents` | Document metadata | url_hash (dedup), version, crawl_status |
|
|
| `distilled_content` | Processed summaries | review_status, compression_ratio |
|
|
| `review_logs` | QA decisions | quality_score, decision |
|
|
| `topics` | Taxonomy | topic_slug, parent_topic_id |
|
|
| `document_topics` | Many-to-many links | relevance_score |
|
|
| `export_jobs` | Export tracking | export_type, status |
|
|
|
|
## Status Values
|
|
|
|
**crawl_status:** `pending` → `completed` | `failed` | `stale`
|
|
|
|
**review_status:** `pending` → `in_review` → `approved` | `needs_refactor` | `rejected`
|
|
|
|
## Scripts
|
|
|
|
| Command | Purpose |
|
|
|---------|---------|
|
|
| `repo.py store` | Store a new document |
|
|
| `repo.py check-dup` | URL deduplication check |
|
|
| `repo.py query-topic` | Query documents by topic |
|
|
| `repo.py find-stale` | Find stale documents |
|
|
| `repo.py pending-reviews` | Get pending reviews |
|
|
| `repo.py export-ready` | Get approved content ready for export |
|
|
| `repo.py stats` | Show repository statistics |
|
|
|
|
## Integration
|
|
|
|
| From | Action | To |
|
|
|------|--------|-----|
|
|
| crawler-orchestrator | Store crawled content | → |
|
|
| → | Query pending docs | content-distiller |
|
|
| quality-reviewer | Update review_status | → |
|
|
| → | Query approved content | markdown-exporter |
|