feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration

- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory
  alongside distilled content, with YAML frontmatter for traceability
- Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer),
  pure archival mode for raw documentation capture
- Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend:
  token-bucket rate limiting, semaphore concurrency, tenacity retries,
  full page metadata extraction (headings, links, schema, OG)
- New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests
  with progress tracking and resume support
- Update crawler selection: docs sites now default to seo-aiohttp
- Update crawl_config.yaml with seo-aiohttp routing rules
- Update pipeline orchestrator with flag resolution and skip paths
- Update README.md and USER-GUIDE.md with raw mode documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

custom-skills/90-reference-curator/07-pipeline-orchestrator/code/CLAUDE.md

@@ -9,17 +9,20 @@ Coordinates the full 6-skill reference curation workflow with QA loop handling.
 
 ```
 [Input] → discovery → crawler → repository → distiller ◄──┐
-                                                    │     │
-                                              reviewer     │
-                                                    │     │
-                    ┌───────────────────────────────┼─────┤
-                    ▼           ▼                   ▼     │
-                 APPROVE     REJECT          REFACTOR ────┤
-                    │           │                         │
-                    ▼           ▼                DEEP_RESEARCH
-                 export      archive                 │
-                                                     ▼
-                                                  crawler ─┘
+                          │                       │     │
+                          │ (--save-raw)    reviewer     │
+                          │ writes raw/           │     │
+                          │               ┌───────┼─────┤
+                          │               ▼       ▼     │
+                          │            APPROVE  REJECT  REFACTOR ────┤
+                          │               │       │              │
+                          │               ▼       ▼       DEEP_RESEARCH
+                          │            export   archive        │
+                          │               │                    ▼
+                          │               │               crawler ─┘
+                          │               │
+                          └── (--no-distill) ──► export (raw only)
+                               skip stages 4+5
 ```
 
 ## Pipeline State Management
@@ -28,7 +31,7 @@ Coordinates the full 6-skill reference curation workflow with QA loop handling.
 ```bash
 uv run python scripts/pipeline.py init \
   --input "prompt engineering" --type topic \
-  --options '{"max_sources": 10, "auto_approve": true}'
+  --options '{"max_sources": 10, "auto_approve": true, "save_raw": false, "no_distill": false}'
 ```
 
 ### Advance to Next Stage
@@ -99,6 +102,7 @@ If mode == 'topic':
 ```
 → Claude uses Firecrawl MCP tools
 → crawl_mgr.py store-result
+→ If --save-raw or --no-distill: write raw .raw.md files to output
 → pipeline.py advance --stage storing
 ```
 
@@ -108,7 +112,23 @@ If mode == 'topic':
 → pipeline.py advance --stage evaluating
 ```
 
+### Flag Resolution (after Stage 3)
+```
+If --no-distill:
+  → set save_raw = true (implied)
+  → skip Stage 4 (content-distiller)
+  → skip Stage 5 (quality-reviewer)
+  → pipeline.py advance --stage exporting --skip-reason "no-distill"
+  → proceed directly to Stage 6 with raw files as input
+
+If --save-raw (without --no-distill):
+  → raw files already written during Stage 2
+  → continue normal pipeline through stages 4-5
+  → Stage 6 exports both distilled AND raw content
+```
+
 ### Stage 4: Gemini Quality Gate (Pre-Distillation)
+**Skipped when `--no-distill` is active.**
 ```
 → reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve
 → APPROVE: proceed to distillation
@@ -118,6 +138,7 @@ If mode == 'topic':
 ```
 
 ### Stage 5: Content Distiller (Approved Only)
+**Skipped when `--no-distill` is active.**
 ```
 → distiller.py load-pending
 → Claude distills each approved document
@@ -127,11 +148,26 @@ If mode == 'topic':
 
 ### Stage 6: Markdown Exporter
 ```
-→ exporter.py project
-→ exporter.py index
-→ exporter.py crossrefs
-→ exporter.py verify
-→ pipeline.py complete
+If --no-distill:
+  → exporter.py raw --primary (raw files are the main output)
+  → exporter.py index (generate README from raw file metadata)
+  → exporter.py verify
+  → pipeline.py complete
+
+If --save-raw (with distillation):
+  → exporter.py project (normal distilled export)
+  → exporter.py raw (write raw/ subdirectory + bundle)
+  → exporter.py index (includes raw section)
+  → exporter.py crossrefs
+  → exporter.py verify
+  → pipeline.py complete
+
+Default (no raw flags):
+  → exporter.py project
+  → exporter.py index
+  → exporter.py crossrefs
+  → exporter.py verify
+  → pipeline.py complete
 ```
 
 ## Checkpoint Strategy
@@ -139,8 +175,9 @@ If mode == 'topic':
 | Stage | Checkpoint | Resume Point |
 |-------|------------|--------------|
 | discovery | manifest.json created | → crawler |
-| crawl | crawl_result.json | → repository |
-| store | DB records | → distiller |
+| crawl | crawl_result.json + raw files | → repository |
+| store | DB records | → distiller (or skip) |
+| no-distill skip | stages 4-5 marked skipped | → exporter (raw) |
 | distill | distilled_content records | → reviewer |
 | review | review_logs records | → exporter or loop |
 | export | final export complete | Done |
@@ -167,6 +204,8 @@ pipeline:
   max_pages: 50
   auto_approve: false
   approval_threshold: 0.85
+  save_raw: false
+  no_distill: false
 
 qa_loop:
   max_refactor_iterations: 3