feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration

- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory
  alongside distilled content, with YAML frontmatter for traceability
- Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer),
  pure archival mode for raw documentation capture
- Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend:
  token-bucket rate limiting, semaphore concurrency, tenacity retries,
  full page metadata extraction (headings, links, schema, OG)
- New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests
  with progress tracking and resume support
- Update crawler selection: docs sites now default to seo-aiohttp
- Update crawl_config.yaml with seo-aiohttp routing rules
- Update pipeline orchestrator with flag resolution and skip paths
- Update README.md and USER-GUIDE.md with raw mode documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/CLAUDE.md

@@ -1,6 +1,6 @@
 # Web Crawler Orchestrator
 
-Orchestrates web crawling with intelligent backend selection. Claude performs actual crawling via Firecrawl MCP tools. This skill manages crawl results, selects crawlers, and tracks crawl metadata.
+Orchestrates web crawling with intelligent backend selection. Primary backend is the OurSEO async crawler (rate-limited, resumable, metadata-rich). Falls back to Firecrawl MCP for JS-rendered sites or WebFetch for simple pages.
 
 ## Trigger Keywords
 "crawl URLs", "fetch documents", "scrape pages", "download references"
@@ -14,9 +14,10 @@ uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.
 
 | Crawler | Best For | Auto-Selected When |
 |---------|----------|-------------------|
-| **Firecrawl MCP** (default) | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered |
+| **SEO aiohttp** (default) | Documentation sites, sitemaps | docs/developer/api domains, has sitemap |
+| **Firecrawl MCP** | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered |
+| **WebFetch** | Quick single pages | Fallback, simple pages |
 | **Node.js** | Small docs sites | ≤50 pages, static content |
-| **Python aiohttp** | Technical docs | ≤200 pages, needs SEO data |
 | **Scrapy** | Enterprise crawls | >200 pages, multi-domain |
 
 ## Workflow
@@ -24,20 +25,69 @@ uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.
 ### Step 1: Analyze Target Site
 Run `select-crawler` to determine site characteristics and get a recommendation.
 
-### Step 2: Execute Crawl
+### Step 2a: Execute Crawl — SEO aiohttp (Recommended)
-Use Firecrawl MCP tools directly:
+
+The SEO crawler adapter provides rate limiting, progress tracking, resume support, and full metadata extraction.
+
+```bash
+# Crawl specific URLs
+uv run python scripts/seo_crawler_adapter.py crawl \
+  --urls "https://docs.example.com/page1,https://docs.example.com/page2" \
+  --output ~/Documents/reference-library/topic-slug/raw/
+
+# Crawl from XML sitemap (auto-discovers URLs)
+uv run python scripts/seo_crawler_adapter.py crawl-sitemap \
+  --sitemap https://docs.example.com/sitemap.xml \
+  --output ~/Documents/reference-library/topic-slug/raw/ \
+  --max-pages 50
+
+# Discover URLs from root page (follows internal links)
+uv run python scripts/seo_crawler_adapter.py discover \
+  --url https://docs.example.com \
+  --output ~/Documents/reference-library/topic-slug/raw/ \
+  --max-depth 2 --max-pages 50 --same-domain
+
+# Crawl from reference-curator manifest
+uv run python scripts/seo_crawler_adapter.py crawl-manifest \
+  --manifest ./manifest.json \
+  --output ~/Documents/reference-library/topic-slug/raw/
+
+# Check crawl progress
+uv run python scripts/seo_crawler_adapter.py status
+```
+
+**SEO crawler capabilities:**
+- Token-bucket rate limiting (via `BaseAsyncClient`)
+- Semaphore-controlled concurrency (default: 5 concurrent)
+- Tenacity retry with exponential backoff
+- Full `PageMetadata` extraction: title, headings, links, schema, OG
+- Progress tracking with ETA and resume files
+- Sitemap XML parsing (including sitemap indexes)
+- Internal link discovery (breadth-first, depth-limited)
+- Direct `.raw.md` output with enriched YAML frontmatter
+
+### Step 2b: Execute Crawl — Firecrawl MCP (for JS-rendered sites)
+
+Use Firecrawl MCP tools directly when JS rendering is required:
 ```
 firecrawl_map(url, limit=100)           # Discover URLs
 firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
 firecrawl_crawl(url, max_depth=2, limit=50)
 ```
 
+### Step 2c: Execute Crawl — WebFetch (fallback)
+
+For quick single-page fetches when no crawl infrastructure is needed:
+```
+WebFetch(url, prompt="Return the COMPLETE page content as clean markdown.")
+```
+
 ### Step 3: Store Crawl Results
 ```bash
 # Store crawled files and create result manifest
 uv run python scripts/crawl_mgr.py store-result \
   --raw-dir ~/Documents/reference-library/raw/ \
-  --crawler firecrawl \
+  --crawler seo-aiohttp \
   --source-id 1 \
   --output crawl_result.json
 
@@ -45,27 +95,66 @@ uv run python scripts/crawl_mgr.py store-result \
 uv run python scripts/crawl_mgr.py list-crawls --status completed
 ```
 
+### Step 4: Write Raw Files (when `--save-raw` or `--no-distill`)
+
+When the pipeline is invoked with `--save-raw` (or `--no-distill`, which implies it), write each crawled page's markdown verbatim to the raw output directory.
+
+**Note:** The SEO crawler adapter (`seo_crawler_adapter.py`) writes `.raw.md` files directly with enriched YAML frontmatter — this step is automatic when using seo-aiohttp.
+
+**File naming:** `{NN}-{page-slug}.raw.md` — zero-padded index matching crawl order.
+
+**Output path:**
+- `--save-raw`: `{output}/{topic-slug}/raw/`
+- `--no-distill`: `{output}/{topic-slug}/` (raw files are primary)
+
+**YAML frontmatter (SEO crawler enriched):**
+```yaml
+---
+source_url: {original_url}
+crawled_at: {ISO 8601 timestamp}
+crawler: seo-aiohttp
+content_length: {character count}
+raw: true
+status_code: {HTTP status}
+title: {page title}
+word_count: {word count}
+internal_links: {count}
+external_links: {count}
+h1: {h1 text}
+schema_types: [{types found}]
+response_time_ms: {ms}
+---
+```
+
+**After all pages are written**, a combined raw bundle (`{topic-slug}-raw-complete.md`) is auto-generated by concatenating all `.raw.md` files with `---` separators.
+
 ## Rate Limiting
 
 All crawlers respect these limits:
-- 20 requests/minute
+- 20 requests/minute (SEO crawler: configurable via `BaseAsyncClient`)
-- 3 concurrent requests
+- 3-5 concurrent requests (SEO crawler: semaphore-controlled)
-- Exponential backoff on 429/5xx
+- Exponential backoff on 429/5xx (SEO crawler: tenacity)
 
 ## Error Handling
 
 | Error | Action |
 |-------|--------|
-| Timeout | Retry once with 2x timeout |
+| Timeout | Retry with exponential backoff (max 3 attempts) |
-| Rate limit (429) | Exponential backoff, max 3 retries |
+| Rate limit (429) | Token-bucket + exponential backoff |
 | Not found (404) | Log and skip |
 | Access denied (403) | Log, mark as `failed` |
-| JS rendering needed | Switch to Firecrawl |
+| JS rendering needed | Switch to Firecrawl MCP |
+| Connection error | Retry with backoff, then skip |
 
 ## Scripts
 
 | Command | Purpose |
 |---------|---------|
+| `seo_crawler_adapter.py crawl` | Crawl URLs with SEO metadata extraction |
+| `seo_crawler_adapter.py crawl-sitemap` | Discover and crawl from XML sitemap |
+| `seo_crawler_adapter.py discover` | BFS link discovery from root URL |
+| `seo_crawler_adapter.py crawl-manifest` | Crawl from reference-curator manifest |
+| `seo_crawler_adapter.py status` | Check crawl progress |
 | `crawl_mgr.py select-crawler` | Recommend optimal crawler for a URL |
 | `crawl_mgr.py store-result` | Store crawl results and create manifest |
 | `crawl_mgr.py list-crawls` | List recent crawl records |
@@ -75,6 +164,7 @@ All crawlers respect these limits:
 | From | To |
 |------|-----|
 | reference-discovery | URL manifest input |
-| Firecrawl MCP | Raw crawled files |
+| SEO aiohttp / Firecrawl MCP | Raw crawled files |
 | → | content-repository (crawl manifest + raw files) |
+| → | raw/ directory (when --save-raw or --no-distill) |
 | quality-reviewer (deep_research) | Additional crawl requests |

custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/scripts/crawl_mgr.py

@@ -53,7 +53,7 @@ def cli():
 @click.option("--raw-dir", required=True, type=click.Path(exists=True),
               help="Directory containing crawled raw files")
 @click.option("--crawler", default="firecrawl",
-              type=click.Choice(["firecrawl", "nodejs", "aiohttp", "scrapy"]))
+              type=click.Choice(["firecrawl", "seo-aiohttp", "nodejs", "aiohttp", "scrapy"]))
 @click.option("--source-id", type=int, help="Source ID to associate documents with")
 @click.option("--output", type=click.Path(), help="Output crawl result JSON path")
 def store_result(raw_dir, crawler, source_id, output):
@@ -123,9 +123,10 @@ def select_crawler(url, page_count, json_output):
     """Recommend the best crawler backend for a URL.
 
     Analyzes URL patterns and site characteristics to suggest:
+    - seo-aiohttp: Documentation sites, sitemaps, HTML-renderable (NEW default for docs)
     - firecrawl: SPAs, JS-rendered content
     - nodejs: Small static docs sites (<=50 pages)
-    - aiohttp: Medium technical docs (<=200 pages)
+    - aiohttp: Medium technical docs (<=200 pages, legacy)
     - scrapy: Large enterprise sites (>200 pages)
     """
     parsed = urlparse(url)
@@ -169,12 +170,13 @@ def select_crawler(url, page_count, json_output):
             reason = f"Small site ({page_count} pages) — lightweight crawler sufficient"
             confidence = 0.8
 
-    # Known documentation platforms → firecrawl
+    # Known documentation platforms → seo-aiohttp (preferred for docs)
-    doc_platforms = ["docs.", "developer.", "api.", "reference."]
+    doc_platforms = ["docs.", "developer.", "api.", "reference.", "support."]
     if any(domain.startswith(p) for p in doc_platforms):
-        if recommendation == "firecrawl":
+        if not spa_detected:
-            reason = "Documentation platform — Firecrawl handles dynamic docs well"
+            recommendation = "seo-aiohttp"
-            confidence = 0.85
+            reason = "Documentation platform — SEO crawler provides metadata + rate limiting + resume"
+            confidence = 0.90
 
     result = {
         "url": url,
@@ -265,7 +267,7 @@ def _extract_url_from_content(content: str) -> str | None:
 
 def _get_alternatives(primary: str) -> list[str]:
     """Get alternative crawler recommendations."""
-    all_crawlers = ["firecrawl", "nodejs", "aiohttp", "scrapy"]
+    all_crawlers = ["seo-aiohttp", "firecrawl", "nodejs", "aiohttp", "scrapy"]
     return [c for c in all_crawlers if c != primary][:2]
 
 

custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/scripts/seo_crawler_adapter.py

@@ -0,0 +1,521 @@
+#!/usr/bin/env python3
+"""SEO Crawler Adapter — bridges the OurSEO async crawler stack into reference-curator.
+
+Uses BaseAsyncClient (rate limiting, concurrency, retries) and PageAnalyzer
+(full HTML parsing, metadata extraction) from the SEO toolkit to provide a
+production-grade crawling backend for reference curation.
+
+Outputs .raw.md files with YAML frontmatter compatible with --save-raw / --no-distill.
+
+Usage:
+    # Crawl a list of URLs
+    python seo_crawler_adapter.py crawl \
+        --urls "https://docs.example.com/page1,https://docs.example.com/page2" \
+        --output ~/Documents/reference-library/topic-slug/raw/
+
+    # Crawl from sitemap
+    python seo_crawler_adapter.py crawl-sitemap \
+        --sitemap https://docs.example.com/sitemap.xml \
+        --output ~/Documents/reference-library/topic-slug/raw/ \
+        --max-pages 50
+
+    # Crawl from manifest file
+    python seo_crawler_adapter.py crawl-manifest \
+        --manifest ./manifest.json \
+        --output ~/Documents/reference-library/topic-slug/raw/
+
+    # Discover URLs from a root page (link extraction)
+    python seo_crawler_adapter.py discover \
+        --url https://docs.example.com \
+        --max-depth 2 --same-domain
+"""
+
+import asyncio
+import json
+import logging
+import re
+import sys
+import xml.etree.ElementTree as ET
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+from urllib.parse import urljoin, urlparse
+
+import click
+import requests
+from bs4 import BeautifulSoup
+from rich.console import Console
+from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskID
+from rich.table import Table
+
+# Import from SEO toolkit (shared base_client)
+SEO_SCRIPTS = Path(__file__).resolve().parents[4] / "12-seo-technical-audit" / "code" / "scripts"
+if str(SEO_SCRIPTS) not in sys.path:
+    sys.path.insert(0, str(SEO_SCRIPTS))
+
+from base_client import BaseAsyncClient, RateLimiter  # noqa: E402
+from page_analyzer import PageAnalyzer, PageMetadata  # noqa: E402
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
+logger = logging.getLogger(__name__)
+console = Console()
+
+# Progress tracking directory
+PROGRESS_DIR = Path.home() / ".claude" / "reference-curator-progress"
+PROGRESS_DIR.mkdir(parents=True, exist_ok=True)
+
+
+@dataclass
+class CrawlProgress:
+    """Track crawl progress for reference curation."""
+    run_id: str = ""
+    total_urls: int = 0
+    processed: int = 0
+    succeeded: int = 0
+    failed: int = 0
+    skipped: int = 0
+    start_time: datetime = field(default_factory=datetime.now)
+    current_url: str = ""
+    status: str = "running"
+
+    def save(self) -> Path:
+        filepath = PROGRESS_DIR / f"{self.run_id}.json"
+        filepath.write_text(json.dumps({
+            "run_id": self.run_id,
+            "status": self.status,
+            "total_urls": self.total_urls,
+            "processed": self.processed,
+            "succeeded": self.succeeded,
+            "failed": self.failed,
+            "skipped": self.skipped,
+            "progress_pct": round(self.processed / max(self.total_urls, 1) * 100, 1),
+            "current_url": self.current_url,
+            "start_time": self.start_time.isoformat(),
+            "updated_at": datetime.now().isoformat(),
+        }, indent=2))
+        return filepath
+
+
+def _slugify(text: str) -> str:
+    """Convert text to URL-friendly slug."""
+    text = text.lower().strip()
+    text = re.sub(r'[^\w\s-]', '', text)
+    text = re.sub(r'[-\s]+', '-', text)
+    return text[:60].rstrip('-')
+
+
+def _extract_main_content(html: str, url: str) -> str:
+    """Extract main content as markdown from HTML.
+
+    Uses BeautifulSoup to strip nav/footer/sidebar, then converts
+    remaining content to clean text with heading structure preserved.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+
+    # Remove non-content elements
+    for tag in soup(["script", "style", "noscript", "nav", "footer",
+                     "header", "aside", "iframe"]):
+        tag.decompose()
+
+    # Try to find main content area
+    main = (soup.find("main") or soup.find("article") or
+            soup.find(role="main") or soup.find(id=re.compile(r"content|main", re.I)) or
+            soup.find(class_=re.compile(r"content|main|article", re.I)) or soup.body or soup)
+
+    lines = []
+    for element in main.descendants:
+        if element.name and element.name.startswith('h') and len(element.name) == 2:
+            level = int(element.name[1])
+            text = element.get_text(strip=True)
+            if text:
+                lines.append(f"\n{'#' * level} {text}\n")
+        elif element.name == 'p':
+            text = element.get_text(strip=True)
+            if text:
+                lines.append(f"\n{text}\n")
+        elif element.name == 'li':
+            text = element.get_text(strip=True)
+            if text:
+                lines.append(f"- {text}")
+        elif element.name == 'pre':
+            code = element.get_text()
+            lang = ""
+            code_tag = element.find("code")
+            if code_tag:
+                classes = code_tag.get("class", [])
+                for cls in classes:
+                    if cls.startswith("language-"):
+                        lang = cls[9:]
+                        break
+            lines.append(f"\n```{lang}\n{code}\n```\n")
+        elif element.name == 'table':
+            lines.append(_table_to_markdown(element))
+
+    return "\n".join(lines).strip()
+
+
+def _table_to_markdown(table_tag) -> str:
+    """Convert an HTML table to markdown."""
+    rows = table_tag.find_all("tr")
+    if not rows:
+        return ""
+
+    md_rows = []
+    for i, row in enumerate(rows):
+        cells = row.find_all(["th", "td"])
+        cell_texts = [c.get_text(strip=True).replace("|", "\\|") for c in cells]
+        md_rows.append("| " + " | ".join(cell_texts) + " |")
+        if i == 0:
+            md_rows.append("| " + " | ".join(["---"] * len(cell_texts)) + " |")
+
+    return "\n" + "\n".join(md_rows) + "\n"
+
+
+def _write_raw_file(
+    output_dir: Path,
+    index: int,
+    url: str,
+    content: str,
+    metadata: PageMetadata | None = None,
+) -> Path:
+    """Write a single .raw.md file with YAML frontmatter."""
+    parsed = urlparse(url)
+    slug = _slugify(parsed.path.rstrip('/').split('/')[-1] or parsed.netloc)
+    filename = f"{index:02d}-{slug}.raw.md"
+    filepath = output_dir / filename
+
+    frontmatter_fields = {
+        "source_url": url,
+        "crawled_at": datetime.now().isoformat(),
+        "crawler": "seo-aiohttp",
+        "content_length": len(content),
+        "raw": True,
+    }
+
+    if metadata:
+        frontmatter_fields.update({
+            "status_code": metadata.status_code,
+            "title": metadata.title or "",
+            "word_count": metadata.word_count,
+            "internal_links": metadata.internal_link_count,
+            "external_links": metadata.external_link_count,
+            "h1": metadata.h1_text or "",
+            "schema_types": metadata.schema_types_found,
+            "response_time_ms": round(metadata.response_time_ms, 1),
+        })
+
+    # Build YAML frontmatter
+    fm_lines = ["---"]
+    for key, value in frontmatter_fields.items():
+        if isinstance(value, bool):
+            fm_lines.append(f"{key}: {'true' if value else 'false'}")
+        elif isinstance(value, list):
+            fm_lines.append(f"{key}: {json.dumps(value)}")
+        elif isinstance(value, str) and (':' in value or '"' in value):
+            fm_lines.append(f'{key}: "{value}"')
+        else:
+            fm_lines.append(f"{key}: {value}")
+    fm_lines.append("---")
+
+    filepath.write_text("\n".join(fm_lines) + "\n\n" + content, encoding="utf-8")
+    return filepath
+
+
+def discover_urls(root_url: str, max_depth: int = 2, same_domain: bool = True,
+                  max_urls: int = 100) -> list[str]:
+    """Discover URLs from a root page by following links.
+
+    Uses PageAnalyzer to extract internal links, then follows them
+    breadth-first up to max_depth.
+    """
+    analyzer = PageAnalyzer(timeout=15)
+    parsed_root = urlparse(root_url)
+    root_domain = parsed_root.netloc.lower()
+
+    visited = set()
+    to_visit = [(root_url, 0)]
+    discovered = []
+
+    while to_visit and len(discovered) < max_urls:
+        url, depth = to_visit.pop(0)
+
+        if url in visited:
+            continue
+        visited.add(url)
+
+        try:
+            metadata = analyzer.analyze_url(url)
+            if metadata.status_code == 200:
+                discovered.append(url)
+                logger.info(f"[{len(discovered)}/{max_urls}] {url}")
+
+                if depth < max_depth:
+                    for link in metadata.internal_links:
+                        link_domain = urlparse(link.url).netloc.lower()
+                        if same_domain and link_domain != root_domain:
+                            continue
+                        if link.url not in visited:
+                            to_visit.append((link.url, depth + 1))
+        except Exception as e:
+            logger.warning(f"Skip {url}: {e}")
+
+    return discovered
+
+
+def discover_from_sitemap(sitemap_url: str, max_urls: int = 100) -> list[str]:
+    """Extract URLs from an XML sitemap."""
+    urls = []
+    try:
+        resp = requests.get(sitemap_url, timeout=15,
+                            headers={"User-Agent": "ReferenceBot/1.0"})
+        resp.raise_for_status()
+
+        root = ET.fromstring(resp.content)
+        ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
+
+        # Check if it's a sitemap index
+        sitemaps = root.findall("sm:sitemap/sm:loc", ns)
+        if sitemaps:
+            for sm in sitemaps[:5]:
+                urls.extend(discover_from_sitemap(sm.text, max_urls - len(urls)))
+                if len(urls) >= max_urls:
+                    break
+        else:
+            for url_elem in root.findall("sm:url/sm:loc", ns):
+                if url_elem.text:
+                    urls.append(url_elem.text.strip())
+                if len(urls) >= max_urls:
+                    break
+    except Exception as e:
+        logger.error(f"Sitemap error: {e}")
+
+    return urls[:max_urls]
+
+
+def crawl_urls(
+    urls: list[str],
+    output_dir: Path,
+    include_metadata: bool = True,
+    run_id: str | None = None,
+) -> dict:
+    """Crawl a list of URLs and write .raw.md files.
+
+    Returns crawl statistics dictionary.
+    """
+    output_dir.mkdir(parents=True, exist_ok=True)
+    analyzer = PageAnalyzer(timeout=20)
+
+    progress = CrawlProgress(
+        run_id=run_id or datetime.now().strftime("%Y%m%d_%H%M%S"),
+        total_urls=len(urls),
+    )
+
+    results = []
+
+    with Progress(
+        SpinnerColumn(),
+        TextColumn("[progress.description]{task.description}"),
+        BarColumn(),
+        TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
+        TextColumn("({task.completed}/{task.total})"),
+        console=console,
+    ) as pbar:
+        task = pbar.add_task("Crawling...", total=len(urls))
+
+        for i, url in enumerate(urls):
+            progress.current_url = url
+            pbar.update(task, description=f"[cyan]{urlparse(url).path[:40]}...")
+
+            try:
+                # Fetch and analyze
+                metadata = analyzer.analyze_url(url) if include_metadata else None
+
+                # Get raw HTML for content extraction
+                resp = requests.get(url, timeout=20,
+                                    headers={"User-Agent": PageAnalyzer.DEFAULT_USER_AGENT})
+
+                if resp.status_code == 200:
+                    content = _extract_main_content(resp.text, url)
+                    filepath = _write_raw_file(output_dir, i, url, content, metadata)
+                    results.append({"url": url, "file": filepath.name, "status": "ok",
+                                    "size": len(content)})
+                    progress.succeeded += 1
+                    logger.info(f"OK [{resp.status_code}] {url} → {filepath.name}")
+                else:
+                    results.append({"url": url, "status": f"http_{resp.status_code}"})
+                    progress.failed += 1
+                    logger.warning(f"FAIL [{resp.status_code}] {url}")
+
+            except Exception as e:
+                results.append({"url": url, "status": "error", "error": str(e)})
+                progress.failed += 1
+                logger.error(f"ERROR {url}: {e}")
+
+            progress.processed += 1
+            pbar.update(task, advance=1)
+            progress.save()
+
+    progress.status = "completed"
+    progress.save()
+
+    # Generate combined bundle
+    raw_files = sorted(output_dir.glob("*.raw.md"))
+    if raw_files:
+        bundle_name = output_dir.parent.name or "reference"
+        bundle_path = output_dir / f"{bundle_name}-raw-complete.md"
+        with open(bundle_path, "w") as bundle:
+            bundle.write(f"# {bundle_name} — Raw Content Bundle\n\n")
+            bundle.write(f"> Combined raw crawled content from {len(raw_files)} sources\n")
+            bundle.write(f"> Generated: {datetime.now().strftime('%Y-%m-%d')}\n\n")
+            for rf in raw_files:
+                bundle.write("---\n\n")
+                bundle.write(rf.read_text())
+                bundle.write("\n\n")
+        results.append({"file": bundle_path.name, "status": "bundle",
+                        "size": bundle_path.stat().st_size})
+
+    stats = {
+        "run_id": progress.run_id,
+        "total_urls": len(urls),
+        "succeeded": progress.succeeded,
+        "failed": progress.failed,
+        "skipped": progress.skipped,
+        "files_written": len(raw_files),
+        "output_dir": str(output_dir),
+        "results": results,
+    }
+
+    # Write manifest
+    manifest_path = output_dir / "crawl_manifest.json"
+    manifest_path.write_text(json.dumps(stats, indent=2, default=str))
+
+    return stats
+
+
+# ─── CLI ───────────────────────────────────────────────────────────────────────
+
+@click.group()
+def cli():
+    """SEO Crawler Adapter — reference-curator crawling backend."""
+    pass
+
+
+@cli.command()
+@click.option("--urls", required=True, help="Comma-separated URLs or @file.txt")
+@click.option("--output", required=True, type=click.Path(), help="Output directory for .raw.md files")
+@click.option("--no-metadata", is_flag=True, help="Skip SEO metadata extraction")
+@click.option("--run-id", help="Custom run ID for progress tracking")
+def crawl(urls, output, no_metadata, run_id):
+    """Crawl a list of URLs and write .raw.md files."""
+    if urls.startswith("@"):
+        url_list = Path(urls[1:]).read_text().strip().splitlines()
+    else:
+        url_list = [u.strip() for u in urls.split(",") if u.strip()]
+
+    output_path = Path(output).expanduser()
+    stats = crawl_urls(url_list, output_path, include_metadata=not no_metadata, run_id=run_id)
+
+    console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled")
+    console.print(f"Output: {stats['output_dir']}")
+
+
+@cli.command("crawl-sitemap")
+@click.option("--sitemap", required=True, help="Sitemap URL")
+@click.option("--output", required=True, type=click.Path(), help="Output directory")
+@click.option("--max-pages", default=50, help="Max pages to crawl")
+@click.option("--run-id", help="Custom run ID")
+def crawl_sitemap(sitemap, output, max_pages, run_id):
+    """Crawl URLs discovered from an XML sitemap."""
+    console.print(f"Parsing sitemap: {sitemap}")
+    urls = discover_from_sitemap(sitemap, max_urls=max_pages)
+    console.print(f"Found {len(urls)} URLs in sitemap")
+
+    if not urls:
+        console.print("[yellow]No URLs found in sitemap[/yellow]")
+        return
+
+    output_path = Path(output).expanduser()
+    stats = crawl_urls(urls, output_path, run_id=run_id)
+    console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled")
+
+
+@cli.command("discover")
+@click.option("--url", required=True, help="Root URL to discover from")
+@click.option("--output", required=True, type=click.Path(), help="Output directory")
+@click.option("--max-depth", default=2, help="Max link-following depth")
+@click.option("--max-pages", default=50, help="Max pages to discover and crawl")
+@click.option("--same-domain/--follow-external", default=True, help="Stay on same domain")
+@click.option("--run-id", help="Custom run ID")
+def discover(url, output, max_depth, max_pages, same_domain, run_id):
+    """Discover URLs from a root page, then crawl them all."""
+    console.print(f"Discovering URLs from: {url} (depth={max_depth})")
+    urls = discover_urls(url, max_depth=max_depth, same_domain=same_domain, max_urls=max_pages)
+    console.print(f"Discovered {len(urls)} URLs")
+
+    if not urls:
+        console.print("[yellow]No URLs discovered[/yellow]")
+        return
+
+    output_path = Path(output).expanduser()
+    stats = crawl_urls(urls, output_path, run_id=run_id)
+    console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled")
+
+
+@cli.command("crawl-manifest")
+@click.option("--manifest", required=True, type=click.Path(exists=True), help="Manifest JSON file")
+@click.option("--output", required=True, type=click.Path(), help="Output directory")
+@click.option("--run-id", help="Custom run ID")
+def crawl_manifest(manifest, output, run_id):
+    """Crawl URLs from a reference-curator manifest file."""
+    data = json.loads(Path(manifest).read_text())
+
+    urls = []
+    if isinstance(data, dict) and "sources" in data:
+        urls = [s["url"] for s in data["sources"] if s.get("url")]
+    elif isinstance(data, list):
+        urls = [item.get("url", item) if isinstance(item, dict) else item for item in data]
+
+    console.print(f"Loaded {len(urls)} URLs from manifest")
+
+    output_path = Path(output).expanduser()
+    stats = crawl_urls(urls, output_path, run_id=run_id)
+    console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled")
+
+
+@cli.command("status")
+@click.option("--run-id", help="Specific run ID to check")
+def status(run_id):
+    """Check crawl progress."""
+    if run_id:
+        filepath = PROGRESS_DIR / f"{run_id}.json"
+        if filepath.exists():
+            data = json.loads(filepath.read_text())
+            console.print_json(json.dumps(data, indent=2))
+        else:
+            console.print(f"[yellow]No progress file for run {run_id}[/yellow]")
+    else:
+        files = sorted(PROGRESS_DIR.glob("*.json"), reverse=True)[:10]
+        if not files:
+            console.print("[yellow]No crawl history found[/yellow]")
+            return
+
+        table = Table(title="Recent Crawls")
+        table.add_column("Run ID")
+        table.add_column("Status")
+        table.add_column("Progress")
+        table.add_column("Success/Total")
+
+        for f in files:
+            data = json.loads(f.read_text())
+            table.add_row(
+                data.get("run_id", "?"),
+                data.get("status", "?"),
+                f"{data.get('progress_pct', 0)}%",
+                f"{data.get('succeeded', 0)}/{data.get('total_urls', 0)}",
+            )
+        console.print(table)
+
+
+if __name__ == "__main__":
+    cli()

custom-skills/90-reference-curator/04-content-distiller/code/CLAUDE.md

@@ -5,6 +5,14 @@ Analyzes and distills raw crawled content into concise reference materials. Clau
 ## Trigger Keywords
 "distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary"
 
+## Skip Condition
+
+**This entire stage is skipped when `--no-distill` is active.**
+
+When the pipeline orchestrator passes `--no-distill`, this stage returns immediately with status `skipped`. The pipeline advances directly from Stage 3 (content-repository) to Stage 6 (markdown-exporter), and raw crawled content becomes the export source instead of distilled content.
+
+No documents are loaded, no distillation occurs, and no records are written to the `distilled_content` table.
+
 ## Goals
 
 1. **Compress** — Reduce token count while preserving essential information

custom-skills/90-reference-curator/05-quality-reviewer/code/CLAUDE.md

@@ -5,6 +5,12 @@ Pre-distillation quality gate using Gemini CLI as an independent evaluator. Asse
 ## Trigger Keywords
 "review content", "quality check", "QA review", "evaluate sources", "check reference quality"
 
+## Skip Condition
+
+**This entire stage is skipped when `--no-distill` is active.**
+
+When the pipeline uses `--no-distill`, quality review is bypassed because there is no distilled content to evaluate. Raw crawled content goes directly to the exporter.
+
 ## Primary Flow: Gemini Pre-Distillation Gate
 
 ```

custom-skills/90-reference-curator/06-markdown-exporter/code/CLAUDE.md

@@ -10,6 +10,7 @@ Exports approved reference content as structured markdown files for project know
 | Type | Format | Use Case |
 |------|--------|----------|
 | `project` | Nested markdown | Claude Projects knowledge |
+| `raw` | Raw markdown | Archival, full-text reference, NotebookLM sources |
 | `finetuning` | JSONL | Model fine-tuning dataset |
 
 ## Workflow
@@ -74,11 +75,57 @@ JSONL format:
 uv run python scripts/exporter.py log --name "April 2026 Export" --type project_files --docs 45
 ```
 
+### Raw Export (when `--save-raw` or `--no-distill`)
+
+#### Raw alongside distilled (`--save-raw`)
+```bash
+uv run python scripts/exporter.py raw \
+  --output ~/reference-library/exports/{topic-slug}/raw/ \
+  --source-dir {crawl-raw-dir} \
+  --bundle {topic-slug}-raw-complete.md
+```
+
+Output structure:
+```
+exports/{topic-slug}/
+├── INDEX.md              # Includes links to raw/ section
+├── 00-page-name.md       # Distilled (normal)
+├── raw/
+│   ├── 00-page-name.raw.md
+│   ├── 01-page-name.raw.md
+│   └── {topic-slug}-raw-complete.md
+└── manifest.json
+```
+
+#### Raw-only mode (`--no-distill`)
+```bash
+uv run python scripts/exporter.py raw \
+  --output ~/reference-library/exports/{topic-slug}/ \
+  --source-dir {crawl-raw-dir} \
+  --bundle {topic-slug}-raw-complete.md \
+  --primary
+```
+
+Output structure (raw files are primary, no subdirectory wrapper):
+```
+exports/{topic-slug}/
+├── README.md
+├── 00-page-name.raw.md
+├── 01-page-name.raw.md
+├── {topic-slug}-raw-complete.md
+└── manifest.json
+```
+
+The `--primary` flag tells the exporter that raw files are the main output. README is generated from raw file frontmatter (source URLs, crawl timestamps).
+
+**Note:** When combined with `--export-format fine_tuning`, the JSONL exporter uses raw content directly as the assistant response with a simpler system prompt (no structured summaries available).
+
 ## Scripts
 
 | Command | Purpose |
 |---------|---------|
 | `exporter.py project` | Export as nested markdown files |
+| `exporter.py raw` | Export raw crawled markdown files with optional bundle |
 | `exporter.py finetuning` | Export as JSONL training dataset |
 | `exporter.py index` | Generate INDEX.md table of contents |
 | `exporter.py crossrefs` | Add cross-reference links |
@@ -90,4 +137,5 @@ uv run python scripts/exporter.py log --name "April 2026 Export" --type project_
 | From | To |
 |------|-----|
 | quality-reviewer (approved) | → |
+| web-crawler (raw files) | Raw markdown input (--save-raw / --no-distill) |
 | → | Project knowledge / Fine-tuning dataset |

custom-skills/90-reference-curator/07-pipeline-orchestrator/code/CLAUDE.md

@@ -9,17 +9,20 @@ Coordinates the full 6-skill reference curation workflow with QA loop handling.
 
 ```
 [Input] → discovery → crawler → repository → distiller ◄──┐
-                                                    │     │
-                                              reviewer     │
-                                                    │     │
-                    ┌───────────────────────────────┼─────┤
-                    ▼           ▼                   ▼     │
-                 APPROVE     REJECT          REFACTOR ────┤
                           │                       │     │
-                    ▼           ▼                DEEP_RESEARCH
+                          │ (--save-raw)    reviewer     │
-                 export      archive                 │
+                          │ writes raw/           │     │
-                                                     ▼
+                          │               ┌───────┼─────┤
-                                                  crawler ─┘
+                          │               ▼       ▼     │
+                          │            APPROVE  REJECT  REFACTOR ────┤
+                          │               │       │              │
+                          │               ▼       ▼       DEEP_RESEARCH
+                          │            export   archive        │
+                          │               │                    ▼
+                          │               │               crawler ─┘
+                          │               │
+                          └── (--no-distill) ──► export (raw only)
+                               skip stages 4+5
 ```
 
 ## Pipeline State Management
@@ -28,7 +31,7 @@ Coordinates the full 6-skill reference curation workflow with QA loop handling.
 ```bash
 uv run python scripts/pipeline.py init \
   --input "prompt engineering" --type topic \
-  --options '{"max_sources": 10, "auto_approve": true}'
+  --options '{"max_sources": 10, "auto_approve": true, "save_raw": false, "no_distill": false}'
 ```
 
 ### Advance to Next Stage
@@ -99,6 +102,7 @@ If mode == 'topic':
 ```
 → Claude uses Firecrawl MCP tools
 → crawl_mgr.py store-result
+→ If --save-raw or --no-distill: write raw .raw.md files to output
 → pipeline.py advance --stage storing
 ```
 
@@ -108,7 +112,23 @@ If mode == 'topic':
 → pipeline.py advance --stage evaluating
 ```
 
+### Flag Resolution (after Stage 3)
+```
+If --no-distill:
+  → set save_raw = true (implied)
+  → skip Stage 4 (content-distiller)
+  → skip Stage 5 (quality-reviewer)
+  → pipeline.py advance --stage exporting --skip-reason "no-distill"
+  → proceed directly to Stage 6 with raw files as input
+
+If --save-raw (without --no-distill):
+  → raw files already written during Stage 2
+  → continue normal pipeline through stages 4-5
+  → Stage 6 exports both distilled AND raw content
+```
+
 ### Stage 4: Gemini Quality Gate (Pre-Distillation)
+**Skipped when `--no-distill` is active.**
 ```
 → reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve
 → APPROVE: proceed to distillation
@@ -118,6 +138,7 @@ If mode == 'topic':
 ```
 
 ### Stage 5: Content Distiller (Approved Only)
+**Skipped when `--no-distill` is active.**
 ```
 → distiller.py load-pending
 → Claude distills each approved document
@@ -127,6 +148,21 @@ If mode == 'topic':
 
 ### Stage 6: Markdown Exporter
 ```
+If --no-distill:
+  → exporter.py raw --primary (raw files are the main output)
+  → exporter.py index (generate README from raw file metadata)
+  → exporter.py verify
+  → pipeline.py complete
+
+If --save-raw (with distillation):
+  → exporter.py project (normal distilled export)
+  → exporter.py raw (write raw/ subdirectory + bundle)
+  → exporter.py index (includes raw section)
+  → exporter.py crossrefs
+  → exporter.py verify
+  → pipeline.py complete
+
+Default (no raw flags):
   → exporter.py project
   → exporter.py index
   → exporter.py crossrefs
@@ -139,8 +175,9 @@ If mode == 'topic':
 | Stage | Checkpoint | Resume Point |
 |-------|------------|--------------|
 | discovery | manifest.json created | → crawler |
-| crawl | crawl_result.json | → repository |
+| crawl | crawl_result.json + raw files | → repository |
-| store | DB records | → distiller |
+| store | DB records | → distiller (or skip) |
+| no-distill skip | stages 4-5 marked skipped | → exporter (raw) |
 | distill | distilled_content records | → reviewer |
 | review | review_logs records | → exporter or loop |
 | export | final export complete | Done |
@@ -167,6 +204,8 @@ pipeline:
   max_pages: 50
   auto_approve: false
   approval_threshold: 0.85
+  save_raw: false
+  no_distill: false
 
 qa_loop:
   max_refactor_iterations: 3

custom-skills/90-reference-curator/README.md

@@ -193,6 +193,15 @@ Run the complete curation workflow with a single command:
 
 # Fine-tuning dataset output
 /reference-curator-pipeline "MCP servers" --export-format fine_tuning
+
+# Save raw crawled content alongside distilled output
+/reference-curator-pipeline "Claude Code best practices" --save-raw
+
+# Pure archival — skip distillation, keep raw markdown only
+/reference-curator-pipeline https://docs.anthropic.com --no-distill
+
+# Full-depth archival
+/reference-curator-pipeline https://legacy-docs.example.com --depth full --no-distill
 ```
 
 **Pipeline Options:**
@@ -202,6 +211,8 @@ Run the complete curation workflow with a single command:
 - `--threshold 0.85` - Approval threshold
 - `--max-iterations 3` - Max QA loop iterations per document
 - `--export-format project_files` - Output format (project_files, fine_tuning, jsonl)
+- `--save-raw` - Also save intact raw crawled markdown in a `raw/` subdirectory
+- `--no-distill` - Skip distillation and QA (pure archival). Implies `--save-raw`
 
 ---
 
@@ -313,6 +324,29 @@ mysql -h $MYSQL_HOST -u $MYSQL_USER -p"$MYSQL_PASSWORD" reference_library -e "
     └── chain-of-thought.md
 ```
 
+**Raw Archival (`--no-distill`):**
+```
+~/reference-library/exports/
+└── topic-slug/
+    ├── README.md
+    ├── 00-page-name.raw.md          # Intact crawled content
+    ├── 01-page-name.raw.md
+    ├── topic-slug-raw-complete.md   # Combined raw bundle
+    └── manifest.json
+```
+
+**Distilled + Raw (`--save-raw`):**
+```
+~/reference-library/exports/
+└── topic-slug/
+    ├── INDEX.md
+    ├── 00-page-name.md              # Distilled
+    ├── raw/
+    │   ├── 00-page-name.raw.md      # Intact crawled
+    │   └── topic-slug-raw-complete.md
+    └── manifest.json
+```
+
 **For Fine-tuning (JSONL):**
 ```json
 {"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

custom-skills/90-reference-curator/USER-GUIDE.md

@@ -15,6 +15,12 @@ A modular skill suite for curating reference documentation with Claude Code.
 
 # With options
 /reference-curator-pipeline "MCP servers" --max-sources 10 --export-format fine_tuning
+
+# Save raw crawled content alongside distilled output
+/reference-curator-pipeline "Claude Code best practices" --save-raw
+
+# Pure archival — skip distillation, raw markdown only
+/reference-curator-pipeline https://docs.anthropic.com --no-distill
 ```
 
 ### Run as Background Agent
@@ -156,6 +162,89 @@ Use these when you need granular control over each stage.
 
 ---
 
+## Raw Mode
+
+### What is Raw Mode?
+
+By default, the pipeline distills crawled content down to 25-35% of the original, extracting key concepts, code snippets, and structured summaries. **Raw mode** preserves the full, unmodified crawled markdown.
+
+### `--save-raw` Flag
+
+Saves intact crawled markdown **alongside** the distilled content in a `raw/` subdirectory.
+
+```bash
+/reference-curator-pipeline "Anthropic Claude API" --save-raw
+```
+
+**Output:**
+```
+exports/claude-api/
+├── INDEX.md
+├── 00-prompt-caching.md           # Distilled (25-35% of original)
+├── 01-streaming.md
+├── raw/
+│   ├── 00-prompt-caching.raw.md   # Full crawled content
+│   ├── 01-streaming.raw.md
+│   └── claude-api-raw-complete.md # Combined raw bundle
+└── manifest.json
+```
+
+**Use cases:**
+- Need both concise reference AND full-text source material
+- Want to re-distill later with different parameters
+- Auditing what content was captured vs. what was extracted
+
+### `--no-distill` Flag
+
+Skips stages 4 (content-distiller) and 5 (quality-reviewer) entirely. The pipeline goes directly from crawl/store to export. Raw files become the **primary** output. Implies `--save-raw` automatically.
+
+```bash
+/reference-curator-pipeline https://docs.anthropic.com --no-distill
+```
+
+**Output:**
+```
+exports/anthropic-docs/
+├── README.md
+├── 00-getting-started.raw.md
+├── 01-prompt-engineering.raw.md
+├── anthropic-docs-raw-complete.md
+└── manifest.json
+```
+
+**Use cases:**
+- Site archival / preservation
+- Migration reference (need exact original content)
+- Training data collection (distillation would remove useful context)
+- NotebookLM / LLM context sources (full fidelity preferred)
+- Quick capture when you will distill manually later
+
+### Combining with Other Flags
+
+```bash
+# Deep archival crawl
+/reference-curator-pipeline https://legacy.example.com --depth full --no-distill
+
+# Standard crawl with raw backup and auto-approval
+/reference-curator-pipeline "MCP servers" --save-raw --auto-approve
+
+# Fine-tuning dataset from raw content
+/reference-curator-pipeline "prompt engineering" --no-distill --export-format fine_tuning
+```
+
+### Pipeline Stage Comparison
+
+| Stage | Default | `--save-raw` | `--no-distill` |
+|-------|---------|-------------|----------------|
+| 1. Discovery | Runs | Runs | Runs |
+| 2. Web Crawler | Runs | Runs + writes raw/ | Runs + writes raw |
+| 3. Repository | Runs | Runs | Runs |
+| 4. Distiller | Runs | Runs | **Skipped** |
+| 5. Reviewer | Runs | Runs | **Skipped** |
+| 6. Exporter | Distilled | Distilled + raw | **Raw only** |
+
+---
+
 ## Common Workflows
 
 ### Workflow 1: Quick Reference on a Topic
@@ -197,6 +286,8 @@ Claude: /markdown-exporter --format fine_tuning
 | Type | Location |
 |------|----------|
 | Raw content | `~/Documents/05_AI Agent/10_Reference Library/raw/` |
+| Raw output (--save-raw) | `{output}/{topic-slug}/raw/` |
+| Raw output (--no-distill) | `{output}/{topic-slug}/` (primary) |
 | Exports | `~/Documents/05_AI Agent/10_Reference Library/exports/` |
 | Config | `~/.config/reference-curator/` |
 
@@ -288,6 +379,34 @@ Claude: Starting reference-curator pipeline...
 Pipeline complete! Reference library ready.
 ```
 
+### Example: Archival Mode (`--no-distill`)
+
+```
+You: Archive the GA4 developer docs without distilling — just raw content
+
+Claude: Starting reference-curator pipeline (--no-distill mode)...
+
+[1/6] Reference Discovery
+- Skipped (URL input mode)
+
+[2/6] Web Crawling
+- Crawling developers.google.com/analytics with Firecrawl...
+- Saved 28 raw markdown files (.raw.md)
+
+[3/6] Content Repository
+- Stored 28 documents in MySQL
+
+[4/6] Content Distillation — SKIPPED (--no-distill)
+[5/6] Quality Review — SKIPPED (--no-distill)
+
+[6/6] Markdown Export (raw mode)
+- Generated 28 .raw.md files + combined raw bundle
+- Output: ~/Documents/.../google-analytics-4/
+
+Pipeline complete! 28 raw pages archived.
+Files ready for NotebookLM upload or direct LLM context use.
+```
+
 ---
 
 ## Need Help?

custom-skills/90-reference-curator/commands/reference-curator-pipeline.md

@@ -1,6 +1,6 @@
 ---
 description: Orchestrates full reference curation pipeline with configurable depth. Runs discovery > crawl > store > distill > review > export with QA loop handling.
-argument-hint: <topic|urls|manifest> [--depth light|standard|deep|full] [--output ~/Documents/reference-library/] [--max-sources 100] [--max-pages 50] [--auto-approve] [--threshold 0.85] [--max-iterations 3] [--export-format project_files]
+argument-hint: <topic|urls|manifest> [--depth light|standard|deep|full] [--output ~/Documents/reference-library/] [--max-sources 100] [--max-pages 50] [--auto-approve] [--threshold 0.85] [--max-iterations 3] [--export-format project_files] [--save-raw] [--no-distill]
 allowed-tools: WebSearch, WebFetch, Read, Write, Bash, Grep, Glob, Task
 ---
 
@@ -29,6 +29,8 @@ Full-stack orchestration of the 6-skill reference curation workflow.
 - `--export-format`: Output format: `project_files`, `fine_tuning`, `jsonl` (default: project_files)
 - `--include-subdomains`: Include subdomains in site mapping (default: false)
 - `--follow-external`: Follow external links found in content (default: false)
+- `--save-raw`: Save intact crawled markdown alongside distilled content in a `raw/` subdirectory. Each file uses `.raw.md` extension. A combined raw bundle `{topic-slug}-raw-complete.md` is also generated.
+- `--no-distill`: Skip stages 4-5 (content-distiller and quality-reviewer) entirely. Pure archival mode. Implies `--save-raw`. Raw crawled files become the primary output — no distilled subdirectory is created.
 
 ## Output Directory
 
@@ -38,13 +40,13 @@ The `--output` argument sets the base path for all pipeline output. If omitted,
 
 ### Directory Structure
 
+**Default (distilled output):**
 ```
 {output}/
-├── {topic-slug}/                   # One folder per pipeline run
+├── {topic-slug}/
 │   ├── README.md                   # Index with table of contents
-│   ├── 00-page-name.md             # Individual page files
+│   ├── 00-page-name.md             # Distilled individual page files
 │   ├── 01-page-name.md
-│   ├── ...
 │   ├── {topic-slug}-complete.md    # Combined bundle (all pages)
 │   └── manifest.json               # Crawl metadata
 ├── pipeline_state/                 # Resume state (auto-managed)
@@ -52,6 +54,31 @@ The `--output` argument sets the base path for all pipeline output. If omitted,
 └── exports/                        # Fine-tuning / JSONL exports
 ```
 
+**With `--save-raw` (distilled + raw):**
+```
+{output}/
+├── {topic-slug}/
+│   ├── README.md
+│   ├── 00-page-name.md             # Distilled
+│   ├── {topic-slug}-complete.md    # Combined distilled
+│   ├── raw/                        # Intact crawled markdown
+│   │   ├── 00-page-name.raw.md
+│   │   ├── 01-page-name.raw.md
+│   │   └── {topic-slug}-raw-complete.md  # Combined raw bundle
+│   └── manifest.json
+```
+
+**With `--no-distill` (raw only, archival):**
+```
+{output}/
+├── {topic-slug}/
+│   ├── README.md
+│   ├── 00-page-name.raw.md         # Raw crawled (primary output)
+│   ├── 01-page-name.raw.md
+│   ├── {topic-slug}-raw-complete.md # Combined raw bundle
+│   └── manifest.json
+```
+
 ### Resolution Rules
 
 1. If `--output` is provided, use that path exactly
@@ -158,15 +185,17 @@ full        ████████████████  Speed: slow
 
 ```
 1. reference-discovery  (topic mode only)
-2. web-crawler              <- depth controls this stage
+2. web-crawler              ← depth controls this stage
+                            ← --save-raw: writes raw/ alongside crawl
 3. content-repository
-4. content-distiller    <--------+
+4. content-distiller    <--------+    ← SKIPPED when --no-distill
-5. quality-reviewer              |
+5. quality-reviewer              |    ← SKIPPED when --no-distill
    +-- APPROVE -> export         |
    +-- REFACTOR -----------------+
    +-- DEEP_RESEARCH -> crawler -+
    +-- REJECT -> archive
-6. markdown-exporter
+6. markdown-exporter            ← --no-distill: exports raw files only
+                                ← --save-raw: also exports raw/ bundle
 ```
 
 ## Crawl Execution by Depth
@@ -234,6 +263,12 @@ After max iterations, document marked as `needs_manual_review`.
 
 # Resume from existing manifest
 /reference-curator ./manifest.json --auto-approve
+
+# Save raw crawled content alongside distilled output
+/reference-curator https://docs.anthropic.com --depth standard --save-raw
+
+# Pure archival — no distillation, just raw markdown files
+/reference-curator https://docs.anthropic.com --depth full --no-distill
 ```
 
 ## State Management

custom-skills/90-reference-curator/shared/config/crawl_config.yaml

@@ -6,13 +6,22 @@
 #   REFERENCE_LIBRARY_PATH - Path to reference library storage (optional)
 
 # Default crawler backend
-# Options: nodejs, aiohttp, scrapy, firecrawl
+# Options: seo-aiohttp, firecrawl, nodejs, aiohttp, scrapy
-# Set to "firecrawl" if local crawlers are not available
+# seo-aiohttp: Uses OurSEO BaseAsyncClient + PageAnalyzer (recommended for docs)
-default_crawler: ${DEFAULT_CRAWLER:-firecrawl}
+# firecrawl: MCP-based, best for SPAs/JS-rendered sites
+default_crawler: ${DEFAULT_CRAWLER:-seo-aiohttp}
 
 # Intelligent routing rules
 # Claude will select the appropriate crawler based on these criteria
 routing:
+  seo-aiohttp:
+    conditions:
+      - is_documentation_site == true
+      - has_sitemap == true
+      - needs_metadata == true
+      - max_pages <= 200
+    description: "OurSEO async crawler — rate-limited, resumable, full metadata extraction. Best for documentation sites."
+
   nodejs:
     conditions:
       - max_pages <= 50
@@ -42,8 +51,22 @@ routing:
     description: "JS rendering - best for SPAs and dynamic content (MCP-based, always available)"
 
 # Crawler locations (configurable via environment)
-# If CRAWLER_PROJECT_PATH is not set, only Firecrawl MCP will be available
 crawlers:
+  seo-aiohttp:
+    enabled: true  # Always available (uses OurSEO scripts from 12-seo-technical-audit)
+    type: local
+    path: ${SKILL_PATH:-../../../90-reference-curator}/02-web-crawler-orchestrator/code/scripts/
+    command: python seo_crawler_adapter.py
+    capabilities:
+      - rate_limiting
+      - progress_tracking
+      - resume_support
+      - metadata_extraction
+      - sitemap_discovery
+      - link_discovery
+      - raw_file_output
+    install: "pip install requests beautifulsoup4 click rich tenacity"
+
   nodejs:
     enabled: ${NODEJS_CRAWLER_ENABLED:-false}
     path: ${CRAWLER_PROJECT_PATH}/util/js-crawler/