feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-12 18:19:52 +09:00
parent 133df68b81
commit f215c11c32
23 changed files with 3917 additions and 583 deletions

View File

@@ -0,0 +1,34 @@
[project]
name = "refcurator"
version = "1.0.0"
description = "Reference Curator — data management for the reference curation pipeline"
requires-python = ">=3.12"
license = "MIT"
authors = [
{ name = "Andrew Yim", email = "andrew@ourdigital.org" },
]
dependencies = [
"pymysql>=1.1.0",
"click>=8.0",
"pydantic>=2.0",
"pyyaml>=6.0",
"rich>=13.0",
"python-dotenv>=1.0",
]
[project.optional-dependencies]
dev = [
"pytest>=8.0",
"ruff>=0.4",
]
[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
where = ["src"]
[tool.ruff]
line-length = 100
target-version = "py312"