feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-12 18:19:52 +09:00
parent 133df68b81
commit f215c11c32
23 changed files with 3917 additions and 583 deletions

View File

@@ -36,8 +36,25 @@ Apply credibility scoring:
- Relevance signals (0.15)
### Step 4: Output URL Manifest
Generate JSON manifest for the crawler skill:
Save discovered URLs as a manifest JSON, deduplicating against the existing repository:
```bash
# Create manifest from discovered URLs
uv run python scripts/discover.py create-manifest --topic "prompt engineering" --output manifest.json < urls.json
# Deduplicate against existing DB
uv run python scripts/discover.py dedup --manifest manifest.json
# Register a new source
uv run python scripts/discover.py register-source \
--name "Anthropic Docs" --type official_docs \
--url "https://docs.anthropic.com" --tier tier1_official --vendor anthropic
# List registered sources
uv run python scripts/discover.py list-sources --vendor anthropic
```
Manifest format:
```json
{
"discovery_date": "2025-01-28T10:30:00",
@@ -56,20 +73,31 @@ Generate JSON manifest for the crawler skill:
}
```
## Scripts
### `discover_sources.py`
Main discovery script. Usage:
```bash
python scripts/discover_sources.py --topic "prompt engineering" --vendors anthropic,openai --output manifest.json
```
## Output
- `manifest.json` → Handoff to `02-web-crawler-orchestrator`
- Register new sources in `sources` table via `03-content-repository`
- New sources registered in `sources` table via `register-source`
## Deduplication
Before outputting:
- Normalize URLs (remove trailing slashes, query params)
- Check against existing `documents` table
- Merge duplicates, keeping highest credibility score
## Scripts
All scripts require the `refcurator` package. Run with `uv run python` from the skill directory.
| Command | Purpose |
|---------|---------|
| `discover.py create-manifest` | Create manifest from URL entries JSON |
| `discover.py dedup` | Deduplicate manifest against DB |
| `discover.py register-source` | Register a new source |
| `discover.py list-sources` | List registered sources |
## Integration
| From | To |
|------|-----|
| WebSearch results | → manifest.json |
| → manifest.json | web-crawler-orchestrator |
| → register-source | content-repository (sources table) |