feat(reference-curator): implement Python scripts + Gemini quality gate
Build the refcurator shared Python package and 7 CLI scripts that were previously specification-only. Add Gemini CLI as an independent pre-distillation quality evaluator, replacing the circular Claude-self-review pattern. Key changes: - shared/lib/src/refcurator/: 7-module package (config, db, models, utils, manifest, gemini) with PyMySQL + JSON file dual backend - 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer, exporter, pipeline — each with subcommands for data management - Gemini quality gate: evaluates raw content BEFORE distillation using 5 criteria (relevance, authority, completeness, freshness, distill_value) - Pipeline reordered: discovery → crawl → store → evaluate → distill → export - Bug fixes from Codex adversarial review: - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries - Exporter uses MAX(review_id) to prevent shipping stale approvals - Distiller updates existing rows on refactor instead of forking - Updated all 7 CLAUDE.md directives with real script references - install.sh updated with refcurator package install step 51/51 E2E tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -36,8 +36,25 @@ Apply credibility scoring:
|
||||
- Relevance signals (0.15)
|
||||
|
||||
### Step 4: Output URL Manifest
|
||||
Generate JSON manifest for the crawler skill:
|
||||
Save discovered URLs as a manifest JSON, deduplicating against the existing repository:
|
||||
|
||||
```bash
|
||||
# Create manifest from discovered URLs
|
||||
uv run python scripts/discover.py create-manifest --topic "prompt engineering" --output manifest.json < urls.json
|
||||
|
||||
# Deduplicate against existing DB
|
||||
uv run python scripts/discover.py dedup --manifest manifest.json
|
||||
|
||||
# Register a new source
|
||||
uv run python scripts/discover.py register-source \
|
||||
--name "Anthropic Docs" --type official_docs \
|
||||
--url "https://docs.anthropic.com" --tier tier1_official --vendor anthropic
|
||||
|
||||
# List registered sources
|
||||
uv run python scripts/discover.py list-sources --vendor anthropic
|
||||
```
|
||||
|
||||
Manifest format:
|
||||
```json
|
||||
{
|
||||
"discovery_date": "2025-01-28T10:30:00",
|
||||
@@ -56,20 +73,31 @@ Generate JSON manifest for the crawler skill:
|
||||
}
|
||||
```
|
||||
|
||||
## Scripts
|
||||
|
||||
### `discover_sources.py`
|
||||
Main discovery script. Usage:
|
||||
```bash
|
||||
python scripts/discover_sources.py --topic "prompt engineering" --vendors anthropic,openai --output manifest.json
|
||||
```
|
||||
|
||||
## Output
|
||||
- `manifest.json` → Handoff to `02-web-crawler-orchestrator`
|
||||
- Register new sources in `sources` table via `03-content-repository`
|
||||
- New sources registered in `sources` table via `register-source`
|
||||
|
||||
## Deduplication
|
||||
Before outputting:
|
||||
- Normalize URLs (remove trailing slashes, query params)
|
||||
- Check against existing `documents` table
|
||||
- Merge duplicates, keeping highest credibility score
|
||||
|
||||
## Scripts
|
||||
|
||||
All scripts require the `refcurator` package. Run with `uv run python` from the skill directory.
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `discover.py create-manifest` | Create manifest from URL entries JSON |
|
||||
| `discover.py dedup` | Deduplicate manifest against DB |
|
||||
| `discover.py register-source` | Register a new source |
|
||||
| `discover.py list-sources` | List registered sources |
|
||||
|
||||
## Integration
|
||||
|
||||
| From | To |
|
||||
|------|-----|
|
||||
| WebSearch results | → manifest.json |
|
||||
| → manifest.json | web-crawler-orchestrator |
|
||||
| → register-source | content-repository (sources table) |
|
||||
|
||||
Reference in New Issue
Block a user