Files

Andrew Yim f215c11c32 feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 18:22:28 +09:00

2.7 KiB

Raw Blame History

Content Repository

MySQL storage management for the reference library. Handles document storage, version control, deduplication, and retrieval. Supports MySQL primary backend with JSON file fallback.

Trigger Keywords

"store content", "save to database", "check duplicates", "version tracking", "document retrieval", "reference library DB"

Prerequisites

refcurator package installed (uv pip install -e shared/lib/)
MySQL 8.0+ (optional — falls back to JSON file storage)
Config at ~/.config/reference-curator/db_config.yaml (optional)

Quick Start

# Store a document
uv run python scripts/repo.py store \
  --source-id 1 --title "Prompt Engineering Guide" \
  --url "https://docs.anthropic.com/..." \
  --doc-type webpage --raw-path ~/reference-library/raw/abc123.md

# Check for duplicates
uv run python scripts/repo.py check-dup --url "https://docs.anthropic.com/..."

# Query by topic
uv run python scripts/repo.py query-topic --topic-slug prompt-engineering --min-quality 0.80

# Get repository stats
uv run python scripts/repo.py stats

# Find stale documents (older than 30 days)
uv run python scripts/repo.py find-stale --days 30

# Get pending reviews
uv run python scripts/repo.py pending-reviews --output pending.json

# Get export-ready content
uv run python scripts/repo.py export-ready --min-score 0.85

Table Quick Reference

Table	Purpose	Key Fields
`sources`	Authorized sources	source_type, credibility_tier, vendor
`documents`	Document metadata	url_hash (dedup), version, crawl_status
`distilled_content`	Processed summaries	review_status, compression_ratio
`review_logs`	QA decisions	quality_score, decision
`topics`	Taxonomy	topic_slug, parent_topic_id
`document_topics`	Many-to-many links	relevance_score
`export_jobs`	Export tracking	export_type, status

Status Values

crawl_status: pending → completed | failed | stale

review_status: pending → in_review → approved | needs_refactor | rejected

Scripts

Command	Purpose
`repo.py store`	Store a new document
`repo.py check-dup`	URL deduplication check
`repo.py query-topic`	Query documents by topic
`repo.py find-stale`	Find stale documents
`repo.py pending-reviews`	Get pending reviews
`repo.py export-ready`	Get approved content ready for export
`repo.py stats`	Show repository statistics

Integration

From	Action	To
crawler-orchestrator	Store crawled content	→
→	Query pending docs	content-distiller
quality-reviewer	Update review_status	→
→	Query approved content	markdown-exporter

2.7 KiB Raw Blame History