Andrew Yim
f215c11c32
Build the refcurator shared Python package and 7 CLI scripts that were previously specification-only. Add Gemini CLI as an independent pre-distillation quality evaluator, replacing the circular Claude-self-review pattern. Key changes: - shared/lib/src/refcurator/: 7-module package (config, db, models, utils, manifest, gemini) with PyMySQL + JSON file dual backend - 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer, exporter, pipeline — each with subcommands for data management - Gemini quality gate: evaluates raw content BEFORE distillation using 5 criteria (relevance, authority, completeness, freshness, distill_value) - Pipeline reordered: discovery → crawl → store → evaluate → distill → export - Bug fixes from Codex adversarial review: - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries - Exporter uses MAX(review_id) to prevent shipping stale approvals - Distiller updates existing rows on refactor instead of forking - Updated all 7 CLAUDE.md directives with real script references - install.sh updated with refcurator package install step 51/51 E2E tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
94 lines
3.2 KiB
Markdown
94 lines
3.2 KiB
Markdown
# Quality Reviewer
|
|
|
|
Pre-distillation quality gate using Gemini CLI as an independent evaluator. Assesses raw crawled content before distillation to filter out low-quality sources early. Also supports manual scoring and routing for edge cases.
|
|
|
|
## Trigger Keywords
|
|
"review content", "quality check", "QA review", "evaluate sources", "check reference quality"
|
|
|
|
## Primary Flow: Gemini Pre-Distillation Gate
|
|
|
|
```
|
|
[Raw Crawled Content]
|
|
│
|
|
▼
|
|
┌────────────────────┐
|
|
│ Gemini CLI Eval │ → relevance, authority, completeness, freshness, distill_value
|
|
└────────────────────┘
|
|
│
|
|
├── ≥ 0.75 → APPROVE → proceed to distillation
|
|
├── 0.50-0.74 → DEEP_RESEARCH → re-crawl for better sources
|
|
└── < 0.50 → REJECT → skip distillation entirely
|
|
```
|
|
|
|
## Evaluation Criteria (Gemini)
|
|
|
|
| Criterion | Weight | What It Checks |
|
|
|-----------|--------|----------------|
|
|
| **Relevance** | 0.25 | Does content match the curation topic? |
|
|
| **Authority** | 0.25 | Official docs / research paper, or blog spam? |
|
|
| **Completeness** | 0.20 | Full article, or nav fragment / error page / stub? |
|
|
| **Freshness** | 0.15 | Up-to-date or outdated information? |
|
|
| **Distill Value** | 0.15 | Unique info worth summarizing, or redundant? |
|
|
|
|
## Workflow
|
|
|
|
### Step 1: Evaluate Single Document
|
|
```bash
|
|
uv run python scripts/reviewer.py gemini-evaluate --doc-id 123 --topic "prompt engineering"
|
|
|
|
# With auto-logging of decision:
|
|
uv run python scripts/reviewer.py gemini-evaluate --doc-id 123 --topic "prompt engineering" --auto-approve
|
|
```
|
|
|
|
### Step 2: Batch Evaluate All Pending
|
|
```bash
|
|
uv run python scripts/reviewer.py gemini-evaluate-pending --topic "prompt engineering" --auto-approve --limit 20
|
|
```
|
|
|
|
### Step 3: Manual Review (Edge Cases)
|
|
For documents where Gemini evaluation fails or needs human judgment:
|
|
```bash
|
|
# Calculate score from manual assessment
|
|
uv run python scripts/reviewer.py calculate-score --assessment assessment.json
|
|
|
|
# Route based on score
|
|
uv run python scripts/reviewer.py route --score 0.78
|
|
|
|
# Log review decision
|
|
uv run python scripts/reviewer.py log-review \
|
|
--distill-id 123 --decision approve --score 0.85 \
|
|
--feedback "Manually verified"
|
|
```
|
|
|
|
### Step 4: Review History
|
|
```bash
|
|
uv run python scripts/reviewer.py history --distill-id 123
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
- Gemini CLI: `npm install -g @google/gemini-cli`
|
|
- Google auth: `gemini` (run once interactively to authenticate)
|
|
- `refcurator` package installed
|
|
|
|
## Scripts
|
|
|
|
| Command | Purpose |
|
|
|---------|---------|
|
|
| `reviewer.py gemini-evaluate` | Evaluate single doc via Gemini CLI |
|
|
| `reviewer.py gemini-evaluate-pending` | Batch evaluate all pending docs |
|
|
| `reviewer.py calculate-score` | Manual weighted score calculation |
|
|
| `reviewer.py route` | Decision routing from score |
|
|
| `reviewer.py log-review` | Log review decision to DB |
|
|
| `reviewer.py load-pending` | Get pending reviews |
|
|
| `reviewer.py history` | Show review history |
|
|
|
|
## Integration
|
|
|
|
| From | Action | To |
|
|
|------|--------|-----|
|
|
| content-repository (raw docs) | Gemini evaluation | → |
|
|
| → | APPROVE | content-distiller |
|
|
| → | DEEP_RESEARCH | web-crawler-orchestrator |
|
|
| → | REJECT | archive (skip distillation) |
|