Files

Andrew Yim b1c2dca080 feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration

- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory
  alongside distilled content, with YAML frontmatter for traceability
- Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer),
  pure archival mode for raw documentation capture
- Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend:
  token-bucket rate limiting, semaphore concurrency, tenacity retries,
  full page metadata extraction (headings, links, schema, OG)
- New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests
  with progress tracking and resume support
- Update crawler selection: docs sites now default to seo-aiohttp
- Update crawl_config.yaml with seo-aiohttp routing rules
- Update pipeline orchestrator with flag resolution and skip paths
- Update README.md and USER-GUIDE.md with raw mode documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 23:26:50 +09:00

3.5 KiB

Raw Blame History

Quality Reviewer

Pre-distillation quality gate using Gemini CLI as an independent evaluator. Assesses raw crawled content before distillation to filter out low-quality sources early. Also supports manual scoring and routing for edge cases.

Trigger Keywords

"review content", "quality check", "QA review", "evaluate sources", "check reference quality"

Skip Condition

This entire stage is skipped when --no-distill is active.

When the pipeline uses --no-distill, quality review is bypassed because there is no distilled content to evaluate. Raw crawled content goes directly to the exporter.

Primary Flow: Gemini Pre-Distillation Gate

[Raw Crawled Content]
       │
       ▼
┌────────────────────┐
│ Gemini CLI Eval    │ → relevance, authority, completeness, freshness, distill_value
└────────────────────┘
       │
       ├── ≥ 0.75 → APPROVE → proceed to distillation
       ├── 0.50-0.74 → DEEP_RESEARCH → re-crawl for better sources
       └── < 0.50 → REJECT → skip distillation entirely

Evaluation Criteria (Gemini)

Criterion	Weight	What It Checks
Relevance	0.25	Does content match the curation topic?
Authority	0.25	Official docs / research paper, or blog spam?
Completeness	0.20	Full article, or nav fragment / error page / stub?
Freshness	0.15	Up-to-date or outdated information?
Distill Value	0.15	Unique info worth summarizing, or redundant?

Workflow

Step 1: Evaluate Single Document

uv run python scripts/reviewer.py gemini-evaluate --doc-id 123 --topic "prompt engineering"

# With auto-logging of decision:
uv run python scripts/reviewer.py gemini-evaluate --doc-id 123 --topic "prompt engineering" --auto-approve

Step 2: Batch Evaluate All Pending

uv run python scripts/reviewer.py gemini-evaluate-pending --topic "prompt engineering" --auto-approve --limit 20

Step 3: Manual Review (Edge Cases)

For documents where Gemini evaluation fails or needs human judgment:

# Calculate score from manual assessment
uv run python scripts/reviewer.py calculate-score --assessment assessment.json

# Route based on score
uv run python scripts/reviewer.py route --score 0.78

# Log review decision
uv run python scripts/reviewer.py log-review \
  --distill-id 123 --decision approve --score 0.85 \
  --feedback "Manually verified"

Step 4: Review History

uv run python scripts/reviewer.py history --distill-id 123

Prerequisites

Gemini CLI: npm install -g @google/gemini-cli
Google auth: gemini (run once interactively to authenticate)
refcurator package installed

Scripts

Command	Purpose
`reviewer.py gemini-evaluate`	Evaluate single doc via Gemini CLI
`reviewer.py gemini-evaluate-pending`	Batch evaluate all pending docs
`reviewer.py calculate-score`	Manual weighted score calculation
`reviewer.py route`	Decision routing from score
`reviewer.py log-review`	Log review decision to DB
`reviewer.py load-pending`	Get pending reviews
`reviewer.py history`	Show review history

Integration

From	Action	To
content-repository (raw docs)	Gemini evaluation	→
→	APPROVE	content-distiller
→	DEEP_RESEARCH	web-crawler-orchestrator
→	REJECT	archive (skip distillation)

3.5 KiB Raw Blame History