Andrew Yim
b1c2dca080
- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory alongside distilled content, with YAML frontmatter for traceability - Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer), pure archival mode for raw documentation capture - Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend: token-bucket rate limiting, semaphore concurrency, tenacity retries, full page metadata extraction (headings, links, schema, OG) - New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests with progress tracking and resume support - Update crawler selection: docs sites now default to seo-aiohttp - Update crawl_config.yaml with seo-aiohttp routing rules - Update pipeline orchestrator with flag resolution and skip paths - Update README.md and USER-GUIDE.md with raw mode documentation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
100 lines
3.5 KiB
Markdown
100 lines
3.5 KiB
Markdown
# Quality Reviewer
|
|
|
|
Pre-distillation quality gate using Gemini CLI as an independent evaluator. Assesses raw crawled content before distillation to filter out low-quality sources early. Also supports manual scoring and routing for edge cases.
|
|
|
|
## Trigger Keywords
|
|
"review content", "quality check", "QA review", "evaluate sources", "check reference quality"
|
|
|
|
## Skip Condition
|
|
|
|
**This entire stage is skipped when `--no-distill` is active.**
|
|
|
|
When the pipeline uses `--no-distill`, quality review is bypassed because there is no distilled content to evaluate. Raw crawled content goes directly to the exporter.
|
|
|
|
## Primary Flow: Gemini Pre-Distillation Gate
|
|
|
|
```
|
|
[Raw Crawled Content]
|
|
│
|
|
▼
|
|
┌────────────────────┐
|
|
│ Gemini CLI Eval │ → relevance, authority, completeness, freshness, distill_value
|
|
└────────────────────┘
|
|
│
|
|
├── ≥ 0.75 → APPROVE → proceed to distillation
|
|
├── 0.50-0.74 → DEEP_RESEARCH → re-crawl for better sources
|
|
└── < 0.50 → REJECT → skip distillation entirely
|
|
```
|
|
|
|
## Evaluation Criteria (Gemini)
|
|
|
|
| Criterion | Weight | What It Checks |
|
|
|-----------|--------|----------------|
|
|
| **Relevance** | 0.25 | Does content match the curation topic? |
|
|
| **Authority** | 0.25 | Official docs / research paper, or blog spam? |
|
|
| **Completeness** | 0.20 | Full article, or nav fragment / error page / stub? |
|
|
| **Freshness** | 0.15 | Up-to-date or outdated information? |
|
|
| **Distill Value** | 0.15 | Unique info worth summarizing, or redundant? |
|
|
|
|
## Workflow
|
|
|
|
### Step 1: Evaluate Single Document
|
|
```bash
|
|
uv run python scripts/reviewer.py gemini-evaluate --doc-id 123 --topic "prompt engineering"
|
|
|
|
# With auto-logging of decision:
|
|
uv run python scripts/reviewer.py gemini-evaluate --doc-id 123 --topic "prompt engineering" --auto-approve
|
|
```
|
|
|
|
### Step 2: Batch Evaluate All Pending
|
|
```bash
|
|
uv run python scripts/reviewer.py gemini-evaluate-pending --topic "prompt engineering" --auto-approve --limit 20
|
|
```
|
|
|
|
### Step 3: Manual Review (Edge Cases)
|
|
For documents where Gemini evaluation fails or needs human judgment:
|
|
```bash
|
|
# Calculate score from manual assessment
|
|
uv run python scripts/reviewer.py calculate-score --assessment assessment.json
|
|
|
|
# Route based on score
|
|
uv run python scripts/reviewer.py route --score 0.78
|
|
|
|
# Log review decision
|
|
uv run python scripts/reviewer.py log-review \
|
|
--distill-id 123 --decision approve --score 0.85 \
|
|
--feedback "Manually verified"
|
|
```
|
|
|
|
### Step 4: Review History
|
|
```bash
|
|
uv run python scripts/reviewer.py history --distill-id 123
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
- Gemini CLI: `npm install -g @google/gemini-cli`
|
|
- Google auth: `gemini` (run once interactively to authenticate)
|
|
- `refcurator` package installed
|
|
|
|
## Scripts
|
|
|
|
| Command | Purpose |
|
|
|---------|---------|
|
|
| `reviewer.py gemini-evaluate` | Evaluate single doc via Gemini CLI |
|
|
| `reviewer.py gemini-evaluate-pending` | Batch evaluate all pending docs |
|
|
| `reviewer.py calculate-score` | Manual weighted score calculation |
|
|
| `reviewer.py route` | Decision routing from score |
|
|
| `reviewer.py log-review` | Log review decision to DB |
|
|
| `reviewer.py load-pending` | Get pending reviews |
|
|
| `reviewer.py history` | Show review history |
|
|
|
|
## Integration
|
|
|
|
| From | Action | To |
|
|
|------|--------|-----|
|
|
| content-repository (raw docs) | Gemini evaluation | → |
|
|
| → | APPROVE | content-distiller |
|
|
| → | DEEP_RESEARCH | web-crawler-orchestrator |
|
|
| → | REJECT | archive (skip distillation) |
|