feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration

- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory
  alongside distilled content, with YAML frontmatter for traceability
- Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer),
  pure archival mode for raw documentation capture
- Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend:
  token-bucket rate limiting, semaphore concurrency, tenacity retries,
  full page metadata extraction (headings, links, schema, OG)
- New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests
  with progress tracking and resume support
- Update crawler selection: docs sites now default to seo-aiohttp
- Update crawl_config.yaml with seo-aiohttp routing rules
- Update pipeline orchestrator with flag resolution and skip paths
- Update README.md and USER-GUIDE.md with raw mode documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-12 23:26:50 +09:00
parent f215c11c32
commit b1c2dca080
11 changed files with 977 additions and 52 deletions

View File

@@ -5,6 +5,14 @@ Analyzes and distills raw crawled content into concise reference materials. Clau
## Trigger Keywords
"distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary"
## Skip Condition
**This entire stage is skipped when `--no-distill` is active.**
When the pipeline orchestrator passes `--no-distill`, this stage returns immediately with status `skipped`. The pipeline advances directly from Stage 3 (content-repository) to Stage 6 (markdown-exporter), and raw crawled content becomes the export source instead of distilled content.
No documents are loaded, no distillation occurs, and no records are written to the `distilled_content` table.
## Goals
1. **Compress** — Reduce token count while preserving essential information