Files

Andrew Yim f215c11c32 feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 18:22:28 +09:00

4.9 KiB

Raw Blame History

Pipeline Orchestrator

Coordinates the full 6-skill reference curation workflow with QA loop handling. Manages pipeline state (init, advance, pause, resume, complete) while Claude orchestrates the actual stage execution.

Trigger Keywords

"curate references", "full pipeline", "run curation", "reference-curator-pipeline"

Architecture

[Input] → discovery → crawler → repository → distiller ◄──┐
                                                    │     │
                                              reviewer     │
                                                    │     │
                    ┌───────────────────────────────┼─────┤
                    ▼           ▼                   ▼     │
                 APPROVE     REJECT          REFACTOR ────┤
                    │           │                         │
                    ▼           ▼                DEEP_RESEARCH
                 export      archive                 │
                                                     ▼
                                                  crawler ─┘

Pipeline State Management

Initialize a Run

uv run python scripts/pipeline.py init \
  --input "prompt engineering" --type topic \
  --options '{"max_sources": 10, "auto_approve": true}'

Advance to Next Stage

uv run python scripts/pipeline.py advance \
  --run-id 1 --stage crawling \
  --stats '{"sources_discovered": 8}'

Pause on Error

uv run python scripts/pipeline.py pause \
  --run-id 1 --error "Crawl timeout on page 45" --stage crawling

Resume from Pause

uv run python scripts/pipeline.py resume --run-id 1

Complete Pipeline

uv run python scripts/pipeline.py complete \
  --run-id 1 --export-path ~/reference-library/exports/ --export-count 40

Check Status

uv run python scripts/pipeline.py status --run-id 1
uv run python scripts/pipeline.py status --all

QA Loop Tracking

# Track a refactor iteration for a document
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action refactor

# Track a deep research iteration
uv run python scripts/pipeline.py track-iteration \
  --run-id 1 --doc-id 42 --action deep_research

Returns one of:

re_distill — proceed with refactor
re_crawl_and_distill — proceed with deep research
needs_manual_review — max iterations exceeded

Decision	Max Iterations
REFACTOR	3
DEEP_RESEARCH	2
Combined total	5

Pipeline Execution Flow

Stage 1: Reference Discovery (Topic Mode Only)

If mode == 'topic':
  → Claude runs WebSearch
  → discover.py create-manifest
  → discover.py dedup
  → pipeline.py advance --stage crawling

Stage 2: Web Crawler

→ Claude uses Firecrawl MCP tools
→ crawl_mgr.py store-result
→ pipeline.py advance --stage storing

Stage 3: Content Repository

→ repo.py store (for each crawled doc)
→ pipeline.py advance --stage evaluating

Stage 4: Gemini Quality Gate (Pre-Distillation)

→ reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve
→ APPROVE: proceed to distillation
→ DEEP_RESEARCH: pipeline.py track-iteration → crawler (re-crawl)
→ REJECT: skip document entirely
→ pipeline.py advance --stage distilling

Stage 5: Content Distiller (Approved Only)

→ distiller.py load-pending
→ Claude distills each approved document
→ distiller.py store
→ pipeline.py advance --stage exporting

Stage 6: Markdown Exporter

→ exporter.py project
→ exporter.py index
→ exporter.py crossrefs
→ exporter.py verify
→ pipeline.py complete

Checkpoint Strategy

Stage	Checkpoint	Resume Point
discovery	manifest.json created	→ crawler
crawl	crawl_result.json	→ repository
store	DB records	→ distiller
distill	distilled_content records	→ reviewer
review	review_logs records	→ exporter or loop
export	final export complete	Done

Scripts

Command	Purpose
`pipeline.py init`	Initialize a new pipeline run
`pipeline.py advance`	Advance to next stage
`pipeline.py pause`	Pause on error
`pipeline.py resume`	Resume from pause
`pipeline.py complete`	Mark pipeline complete
`pipeline.py status`	Show run status
`pipeline.py track-iteration`	Track QA loop iterations

Configuration

Reads from ~/.config/reference-curator/pipeline_config.yaml:

pipeline:
  max_sources: 10
  max_pages: 50
  auto_approve: false
  approval_threshold: 0.85

qa_loop:
  max_refactor_iterations: 3
  max_deep_research_iterations: 2
  max_total_iterations: 5

export:
  default_format: project_files

4.9 KiB Raw Blame History