feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration
- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory alongside distilled content, with YAML frontmatter for traceability - Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer), pure archival mode for raw documentation capture - Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend: token-bucket rate limiting, semaphore concurrency, tenacity retries, full page metadata extraction (headings, links, schema, OG) - New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests with progress tracking and resume support - Update crawler selection: docs sites now default to seo-aiohttp - Update crawl_config.yaml with seo-aiohttp routing rules - Update pipeline orchestrator with flag resolution and skip paths - Update README.md and USER-GUIDE.md with raw mode documentation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -5,6 +5,12 @@ Pre-distillation quality gate using Gemini CLI as an independent evaluator. Asse
|
||||
## Trigger Keywords
|
||||
"review content", "quality check", "QA review", "evaluate sources", "check reference quality"
|
||||
|
||||
## Skip Condition
|
||||
|
||||
**This entire stage is skipped when `--no-distill` is active.**
|
||||
|
||||
When the pipeline uses `--no-distill`, quality review is bypassed because there is no distilled content to evaluate. Raw crawled content goes directly to the exporter.
|
||||
|
||||
## Primary Flow: Gemini Pre-Distillation Gate
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user