diff --git a/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/CLAUDE.md b/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/CLAUDE.md index 5565071..c6879cb 100644 --- a/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/CLAUDE.md +++ b/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/CLAUDE.md @@ -1,6 +1,6 @@ # Web Crawler Orchestrator -Orchestrates web crawling with intelligent backend selection. Claude performs actual crawling via Firecrawl MCP tools. This skill manages crawl results, selects crawlers, and tracks crawl metadata. +Orchestrates web crawling with intelligent backend selection. Primary backend is the OurSEO async crawler (rate-limited, resumable, metadata-rich). Falls back to Firecrawl MCP for JS-rendered sites or WebFetch for simple pages. ## Trigger Keywords "crawl URLs", "fetch documents", "scrape pages", "download references" @@ -14,9 +14,10 @@ uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic. | Crawler | Best For | Auto-Selected When | |---------|----------|-------------------| -| **Firecrawl MCP** (default) | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered | +| **SEO aiohttp** (default) | Documentation sites, sitemaps | docs/developer/api domains, has sitemap | +| **Firecrawl MCP** | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered | +| **WebFetch** | Quick single pages | Fallback, simple pages | | **Node.js** | Small docs sites | ≤50 pages, static content | -| **Python aiohttp** | Technical docs | ≤200 pages, needs SEO data | | **Scrapy** | Enterprise crawls | >200 pages, multi-domain | ## Workflow @@ -24,20 +25,69 @@ uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic. ### Step 1: Analyze Target Site Run `select-crawler` to determine site characteristics and get a recommendation. -### Step 2: Execute Crawl -Use Firecrawl MCP tools directly: +### Step 2a: Execute Crawl — SEO aiohttp (Recommended) + +The SEO crawler adapter provides rate limiting, progress tracking, resume support, and full metadata extraction. + +```bash +# Crawl specific URLs +uv run python scripts/seo_crawler_adapter.py crawl \ + --urls "https://docs.example.com/page1,https://docs.example.com/page2" \ + --output ~/Documents/reference-library/topic-slug/raw/ + +# Crawl from XML sitemap (auto-discovers URLs) +uv run python scripts/seo_crawler_adapter.py crawl-sitemap \ + --sitemap https://docs.example.com/sitemap.xml \ + --output ~/Documents/reference-library/topic-slug/raw/ \ + --max-pages 50 + +# Discover URLs from root page (follows internal links) +uv run python scripts/seo_crawler_adapter.py discover \ + --url https://docs.example.com \ + --output ~/Documents/reference-library/topic-slug/raw/ \ + --max-depth 2 --max-pages 50 --same-domain + +# Crawl from reference-curator manifest +uv run python scripts/seo_crawler_adapter.py crawl-manifest \ + --manifest ./manifest.json \ + --output ~/Documents/reference-library/topic-slug/raw/ + +# Check crawl progress +uv run python scripts/seo_crawler_adapter.py status +``` + +**SEO crawler capabilities:** +- Token-bucket rate limiting (via `BaseAsyncClient`) +- Semaphore-controlled concurrency (default: 5 concurrent) +- Tenacity retry with exponential backoff +- Full `PageMetadata` extraction: title, headings, links, schema, OG +- Progress tracking with ETA and resume files +- Sitemap XML parsing (including sitemap indexes) +- Internal link discovery (breadth-first, depth-limited) +- Direct `.raw.md` output with enriched YAML frontmatter + +### Step 2b: Execute Crawl — Firecrawl MCP (for JS-rendered sites) + +Use Firecrawl MCP tools directly when JS rendering is required: ``` firecrawl_map(url, limit=100) # Discover URLs firecrawl_scrape(url, formats=["markdown"], only_main_content=true) firecrawl_crawl(url, max_depth=2, limit=50) ``` +### Step 2c: Execute Crawl — WebFetch (fallback) + +For quick single-page fetches when no crawl infrastructure is needed: +``` +WebFetch(url, prompt="Return the COMPLETE page content as clean markdown.") +``` + ### Step 3: Store Crawl Results ```bash # Store crawled files and create result manifest uv run python scripts/crawl_mgr.py store-result \ --raw-dir ~/Documents/reference-library/raw/ \ - --crawler firecrawl \ + --crawler seo-aiohttp \ --source-id 1 \ --output crawl_result.json @@ -45,27 +95,66 @@ uv run python scripts/crawl_mgr.py store-result \ uv run python scripts/crawl_mgr.py list-crawls --status completed ``` +### Step 4: Write Raw Files (when `--save-raw` or `--no-distill`) + +When the pipeline is invoked with `--save-raw` (or `--no-distill`, which implies it), write each crawled page's markdown verbatim to the raw output directory. + +**Note:** The SEO crawler adapter (`seo_crawler_adapter.py`) writes `.raw.md` files directly with enriched YAML frontmatter — this step is automatic when using seo-aiohttp. + +**File naming:** `{NN}-{page-slug}.raw.md` — zero-padded index matching crawl order. + +**Output path:** +- `--save-raw`: `{output}/{topic-slug}/raw/` +- `--no-distill`: `{output}/{topic-slug}/` (raw files are primary) + +**YAML frontmatter (SEO crawler enriched):** +```yaml +--- +source_url: {original_url} +crawled_at: {ISO 8601 timestamp} +crawler: seo-aiohttp +content_length: {character count} +raw: true +status_code: {HTTP status} +title: {page title} +word_count: {word count} +internal_links: {count} +external_links: {count} +h1: {h1 text} +schema_types: [{types found}] +response_time_ms: {ms} +--- +``` + +**After all pages are written**, a combined raw bundle (`{topic-slug}-raw-complete.md`) is auto-generated by concatenating all `.raw.md` files with `---` separators. + ## Rate Limiting All crawlers respect these limits: -- 20 requests/minute -- 3 concurrent requests -- Exponential backoff on 429/5xx +- 20 requests/minute (SEO crawler: configurable via `BaseAsyncClient`) +- 3-5 concurrent requests (SEO crawler: semaphore-controlled) +- Exponential backoff on 429/5xx (SEO crawler: tenacity) ## Error Handling | Error | Action | |-------|--------| -| Timeout | Retry once with 2x timeout | -| Rate limit (429) | Exponential backoff, max 3 retries | +| Timeout | Retry with exponential backoff (max 3 attempts) | +| Rate limit (429) | Token-bucket + exponential backoff | | Not found (404) | Log and skip | | Access denied (403) | Log, mark as `failed` | -| JS rendering needed | Switch to Firecrawl | +| JS rendering needed | Switch to Firecrawl MCP | +| Connection error | Retry with backoff, then skip | ## Scripts | Command | Purpose | |---------|---------| +| `seo_crawler_adapter.py crawl` | Crawl URLs with SEO metadata extraction | +| `seo_crawler_adapter.py crawl-sitemap` | Discover and crawl from XML sitemap | +| `seo_crawler_adapter.py discover` | BFS link discovery from root URL | +| `seo_crawler_adapter.py crawl-manifest` | Crawl from reference-curator manifest | +| `seo_crawler_adapter.py status` | Check crawl progress | | `crawl_mgr.py select-crawler` | Recommend optimal crawler for a URL | | `crawl_mgr.py store-result` | Store crawl results and create manifest | | `crawl_mgr.py list-crawls` | List recent crawl records | @@ -75,6 +164,7 @@ All crawlers respect these limits: | From | To | |------|-----| | reference-discovery | URL manifest input | -| Firecrawl MCP | Raw crawled files | +| SEO aiohttp / Firecrawl MCP | Raw crawled files | | → | content-repository (crawl manifest + raw files) | +| → | raw/ directory (when --save-raw or --no-distill) | | quality-reviewer (deep_research) | Additional crawl requests | diff --git a/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/scripts/crawl_mgr.py b/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/scripts/crawl_mgr.py index 8ec946d..962e2dd 100644 --- a/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/scripts/crawl_mgr.py +++ b/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/scripts/crawl_mgr.py @@ -53,7 +53,7 @@ def cli(): @click.option("--raw-dir", required=True, type=click.Path(exists=True), help="Directory containing crawled raw files") @click.option("--crawler", default="firecrawl", - type=click.Choice(["firecrawl", "nodejs", "aiohttp", "scrapy"])) + type=click.Choice(["firecrawl", "seo-aiohttp", "nodejs", "aiohttp", "scrapy"])) @click.option("--source-id", type=int, help="Source ID to associate documents with") @click.option("--output", type=click.Path(), help="Output crawl result JSON path") def store_result(raw_dir, crawler, source_id, output): @@ -123,9 +123,10 @@ def select_crawler(url, page_count, json_output): """Recommend the best crawler backend for a URL. Analyzes URL patterns and site characteristics to suggest: + - seo-aiohttp: Documentation sites, sitemaps, HTML-renderable (NEW default for docs) - firecrawl: SPAs, JS-rendered content - nodejs: Small static docs sites (<=50 pages) - - aiohttp: Medium technical docs (<=200 pages) + - aiohttp: Medium technical docs (<=200 pages, legacy) - scrapy: Large enterprise sites (>200 pages) """ parsed = urlparse(url) @@ -169,12 +170,13 @@ def select_crawler(url, page_count, json_output): reason = f"Small site ({page_count} pages) — lightweight crawler sufficient" confidence = 0.8 - # Known documentation platforms → firecrawl - doc_platforms = ["docs.", "developer.", "api.", "reference."] + # Known documentation platforms → seo-aiohttp (preferred for docs) + doc_platforms = ["docs.", "developer.", "api.", "reference.", "support."] if any(domain.startswith(p) for p in doc_platforms): - if recommendation == "firecrawl": - reason = "Documentation platform — Firecrawl handles dynamic docs well" - confidence = 0.85 + if not spa_detected: + recommendation = "seo-aiohttp" + reason = "Documentation platform — SEO crawler provides metadata + rate limiting + resume" + confidence = 0.90 result = { "url": url, @@ -265,7 +267,7 @@ def _extract_url_from_content(content: str) -> str | None: def _get_alternatives(primary: str) -> list[str]: """Get alternative crawler recommendations.""" - all_crawlers = ["firecrawl", "nodejs", "aiohttp", "scrapy"] + all_crawlers = ["seo-aiohttp", "firecrawl", "nodejs", "aiohttp", "scrapy"] return [c for c in all_crawlers if c != primary][:2] diff --git a/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/scripts/seo_crawler_adapter.py b/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/scripts/seo_crawler_adapter.py new file mode 100644 index 0000000..bd571d0 --- /dev/null +++ b/custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/scripts/seo_crawler_adapter.py @@ -0,0 +1,521 @@ +#!/usr/bin/env python3 +"""SEO Crawler Adapter — bridges the OurSEO async crawler stack into reference-curator. + +Uses BaseAsyncClient (rate limiting, concurrency, retries) and PageAnalyzer +(full HTML parsing, metadata extraction) from the SEO toolkit to provide a +production-grade crawling backend for reference curation. + +Outputs .raw.md files with YAML frontmatter compatible with --save-raw / --no-distill. + +Usage: + # Crawl a list of URLs + python seo_crawler_adapter.py crawl \ + --urls "https://docs.example.com/page1,https://docs.example.com/page2" \ + --output ~/Documents/reference-library/topic-slug/raw/ + + # Crawl from sitemap + python seo_crawler_adapter.py crawl-sitemap \ + --sitemap https://docs.example.com/sitemap.xml \ + --output ~/Documents/reference-library/topic-slug/raw/ \ + --max-pages 50 + + # Crawl from manifest file + python seo_crawler_adapter.py crawl-manifest \ + --manifest ./manifest.json \ + --output ~/Documents/reference-library/topic-slug/raw/ + + # Discover URLs from a root page (link extraction) + python seo_crawler_adapter.py discover \ + --url https://docs.example.com \ + --max-depth 2 --same-domain +""" + +import asyncio +import json +import logging +import re +import sys +import xml.etree.ElementTree as ET +from dataclasses import dataclass, field +from datetime import datetime +from pathlib import Path +from typing import Any +from urllib.parse import urljoin, urlparse + +import click +import requests +from bs4 import BeautifulSoup +from rich.console import Console +from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskID +from rich.table import Table + +# Import from SEO toolkit (shared base_client) +SEO_SCRIPTS = Path(__file__).resolve().parents[4] / "12-seo-technical-audit" / "code" / "scripts" +if str(SEO_SCRIPTS) not in sys.path: + sys.path.insert(0, str(SEO_SCRIPTS)) + +from base_client import BaseAsyncClient, RateLimiter # noqa: E402 +from page_analyzer import PageAnalyzer, PageMetadata # noqa: E402 + +logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s") +logger = logging.getLogger(__name__) +console = Console() + +# Progress tracking directory +PROGRESS_DIR = Path.home() / ".claude" / "reference-curator-progress" +PROGRESS_DIR.mkdir(parents=True, exist_ok=True) + + +@dataclass +class CrawlProgress: + """Track crawl progress for reference curation.""" + run_id: str = "" + total_urls: int = 0 + processed: int = 0 + succeeded: int = 0 + failed: int = 0 + skipped: int = 0 + start_time: datetime = field(default_factory=datetime.now) + current_url: str = "" + status: str = "running" + + def save(self) -> Path: + filepath = PROGRESS_DIR / f"{self.run_id}.json" + filepath.write_text(json.dumps({ + "run_id": self.run_id, + "status": self.status, + "total_urls": self.total_urls, + "processed": self.processed, + "succeeded": self.succeeded, + "failed": self.failed, + "skipped": self.skipped, + "progress_pct": round(self.processed / max(self.total_urls, 1) * 100, 1), + "current_url": self.current_url, + "start_time": self.start_time.isoformat(), + "updated_at": datetime.now().isoformat(), + }, indent=2)) + return filepath + + +def _slugify(text: str) -> str: + """Convert text to URL-friendly slug.""" + text = text.lower().strip() + text = re.sub(r'[^\w\s-]', '', text) + text = re.sub(r'[-\s]+', '-', text) + return text[:60].rstrip('-') + + +def _extract_main_content(html: str, url: str) -> str: + """Extract main content as markdown from HTML. + + Uses BeautifulSoup to strip nav/footer/sidebar, then converts + remaining content to clean text with heading structure preserved. + """ + soup = BeautifulSoup(html, "html.parser") + + # Remove non-content elements + for tag in soup(["script", "style", "noscript", "nav", "footer", + "header", "aside", "iframe"]): + tag.decompose() + + # Try to find main content area + main = (soup.find("main") or soup.find("article") or + soup.find(role="main") or soup.find(id=re.compile(r"content|main", re.I)) or + soup.find(class_=re.compile(r"content|main|article", re.I)) or soup.body or soup) + + lines = [] + for element in main.descendants: + if element.name and element.name.startswith('h') and len(element.name) == 2: + level = int(element.name[1]) + text = element.get_text(strip=True) + if text: + lines.append(f"\n{'#' * level} {text}\n") + elif element.name == 'p': + text = element.get_text(strip=True) + if text: + lines.append(f"\n{text}\n") + elif element.name == 'li': + text = element.get_text(strip=True) + if text: + lines.append(f"- {text}") + elif element.name == 'pre': + code = element.get_text() + lang = "" + code_tag = element.find("code") + if code_tag: + classes = code_tag.get("class", []) + for cls in classes: + if cls.startswith("language-"): + lang = cls[9:] + break + lines.append(f"\n```{lang}\n{code}\n```\n") + elif element.name == 'table': + lines.append(_table_to_markdown(element)) + + return "\n".join(lines).strip() + + +def _table_to_markdown(table_tag) -> str: + """Convert an HTML table to markdown.""" + rows = table_tag.find_all("tr") + if not rows: + return "" + + md_rows = [] + for i, row in enumerate(rows): + cells = row.find_all(["th", "td"]) + cell_texts = [c.get_text(strip=True).replace("|", "\\|") for c in cells] + md_rows.append("| " + " | ".join(cell_texts) + " |") + if i == 0: + md_rows.append("| " + " | ".join(["---"] * len(cell_texts)) + " |") + + return "\n" + "\n".join(md_rows) + "\n" + + +def _write_raw_file( + output_dir: Path, + index: int, + url: str, + content: str, + metadata: PageMetadata | None = None, +) -> Path: + """Write a single .raw.md file with YAML frontmatter.""" + parsed = urlparse(url) + slug = _slugify(parsed.path.rstrip('/').split('/')[-1] or parsed.netloc) + filename = f"{index:02d}-{slug}.raw.md" + filepath = output_dir / filename + + frontmatter_fields = { + "source_url": url, + "crawled_at": datetime.now().isoformat(), + "crawler": "seo-aiohttp", + "content_length": len(content), + "raw": True, + } + + if metadata: + frontmatter_fields.update({ + "status_code": metadata.status_code, + "title": metadata.title or "", + "word_count": metadata.word_count, + "internal_links": metadata.internal_link_count, + "external_links": metadata.external_link_count, + "h1": metadata.h1_text or "", + "schema_types": metadata.schema_types_found, + "response_time_ms": round(metadata.response_time_ms, 1), + }) + + # Build YAML frontmatter + fm_lines = ["---"] + for key, value in frontmatter_fields.items(): + if isinstance(value, bool): + fm_lines.append(f"{key}: {'true' if value else 'false'}") + elif isinstance(value, list): + fm_lines.append(f"{key}: {json.dumps(value)}") + elif isinstance(value, str) and (':' in value or '"' in value): + fm_lines.append(f'{key}: "{value}"') + else: + fm_lines.append(f"{key}: {value}") + fm_lines.append("---") + + filepath.write_text("\n".join(fm_lines) + "\n\n" + content, encoding="utf-8") + return filepath + + +def discover_urls(root_url: str, max_depth: int = 2, same_domain: bool = True, + max_urls: int = 100) -> list[str]: + """Discover URLs from a root page by following links. + + Uses PageAnalyzer to extract internal links, then follows them + breadth-first up to max_depth. + """ + analyzer = PageAnalyzer(timeout=15) + parsed_root = urlparse(root_url) + root_domain = parsed_root.netloc.lower() + + visited = set() + to_visit = [(root_url, 0)] + discovered = [] + + while to_visit and len(discovered) < max_urls: + url, depth = to_visit.pop(0) + + if url in visited: + continue + visited.add(url) + + try: + metadata = analyzer.analyze_url(url) + if metadata.status_code == 200: + discovered.append(url) + logger.info(f"[{len(discovered)}/{max_urls}] {url}") + + if depth < max_depth: + for link in metadata.internal_links: + link_domain = urlparse(link.url).netloc.lower() + if same_domain and link_domain != root_domain: + continue + if link.url not in visited: + to_visit.append((link.url, depth + 1)) + except Exception as e: + logger.warning(f"Skip {url}: {e}") + + return discovered + + +def discover_from_sitemap(sitemap_url: str, max_urls: int = 100) -> list[str]: + """Extract URLs from an XML sitemap.""" + urls = [] + try: + resp = requests.get(sitemap_url, timeout=15, + headers={"User-Agent": "ReferenceBot/1.0"}) + resp.raise_for_status() + + root = ET.fromstring(resp.content) + ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"} + + # Check if it's a sitemap index + sitemaps = root.findall("sm:sitemap/sm:loc", ns) + if sitemaps: + for sm in sitemaps[:5]: + urls.extend(discover_from_sitemap(sm.text, max_urls - len(urls))) + if len(urls) >= max_urls: + break + else: + for url_elem in root.findall("sm:url/sm:loc", ns): + if url_elem.text: + urls.append(url_elem.text.strip()) + if len(urls) >= max_urls: + break + except Exception as e: + logger.error(f"Sitemap error: {e}") + + return urls[:max_urls] + + +def crawl_urls( + urls: list[str], + output_dir: Path, + include_metadata: bool = True, + run_id: str | None = None, +) -> dict: + """Crawl a list of URLs and write .raw.md files. + + Returns crawl statistics dictionary. + """ + output_dir.mkdir(parents=True, exist_ok=True) + analyzer = PageAnalyzer(timeout=20) + + progress = CrawlProgress( + run_id=run_id or datetime.now().strftime("%Y%m%d_%H%M%S"), + total_urls=len(urls), + ) + + results = [] + + with Progress( + SpinnerColumn(), + TextColumn("[progress.description]{task.description}"), + BarColumn(), + TextColumn("[progress.percentage]{task.percentage:>3.0f}%"), + TextColumn("({task.completed}/{task.total})"), + console=console, + ) as pbar: + task = pbar.add_task("Crawling...", total=len(urls)) + + for i, url in enumerate(urls): + progress.current_url = url + pbar.update(task, description=f"[cyan]{urlparse(url).path[:40]}...") + + try: + # Fetch and analyze + metadata = analyzer.analyze_url(url) if include_metadata else None + + # Get raw HTML for content extraction + resp = requests.get(url, timeout=20, + headers={"User-Agent": PageAnalyzer.DEFAULT_USER_AGENT}) + + if resp.status_code == 200: + content = _extract_main_content(resp.text, url) + filepath = _write_raw_file(output_dir, i, url, content, metadata) + results.append({"url": url, "file": filepath.name, "status": "ok", + "size": len(content)}) + progress.succeeded += 1 + logger.info(f"OK [{resp.status_code}] {url} → {filepath.name}") + else: + results.append({"url": url, "status": f"http_{resp.status_code}"}) + progress.failed += 1 + logger.warning(f"FAIL [{resp.status_code}] {url}") + + except Exception as e: + results.append({"url": url, "status": "error", "error": str(e)}) + progress.failed += 1 + logger.error(f"ERROR {url}: {e}") + + progress.processed += 1 + pbar.update(task, advance=1) + progress.save() + + progress.status = "completed" + progress.save() + + # Generate combined bundle + raw_files = sorted(output_dir.glob("*.raw.md")) + if raw_files: + bundle_name = output_dir.parent.name or "reference" + bundle_path = output_dir / f"{bundle_name}-raw-complete.md" + with open(bundle_path, "w") as bundle: + bundle.write(f"# {bundle_name} — Raw Content Bundle\n\n") + bundle.write(f"> Combined raw crawled content from {len(raw_files)} sources\n") + bundle.write(f"> Generated: {datetime.now().strftime('%Y-%m-%d')}\n\n") + for rf in raw_files: + bundle.write("---\n\n") + bundle.write(rf.read_text()) + bundle.write("\n\n") + results.append({"file": bundle_path.name, "status": "bundle", + "size": bundle_path.stat().st_size}) + + stats = { + "run_id": progress.run_id, + "total_urls": len(urls), + "succeeded": progress.succeeded, + "failed": progress.failed, + "skipped": progress.skipped, + "files_written": len(raw_files), + "output_dir": str(output_dir), + "results": results, + } + + # Write manifest + manifest_path = output_dir / "crawl_manifest.json" + manifest_path.write_text(json.dumps(stats, indent=2, default=str)) + + return stats + + +# ─── CLI ─────────────────────────────────────────────────────────────────────── + +@click.group() +def cli(): + """SEO Crawler Adapter — reference-curator crawling backend.""" + pass + + +@cli.command() +@click.option("--urls", required=True, help="Comma-separated URLs or @file.txt") +@click.option("--output", required=True, type=click.Path(), help="Output directory for .raw.md files") +@click.option("--no-metadata", is_flag=True, help="Skip SEO metadata extraction") +@click.option("--run-id", help="Custom run ID for progress tracking") +def crawl(urls, output, no_metadata, run_id): + """Crawl a list of URLs and write .raw.md files.""" + if urls.startswith("@"): + url_list = Path(urls[1:]).read_text().strip().splitlines() + else: + url_list = [u.strip() for u in urls.split(",") if u.strip()] + + output_path = Path(output).expanduser() + stats = crawl_urls(url_list, output_path, include_metadata=not no_metadata, run_id=run_id) + + console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled") + console.print(f"Output: {stats['output_dir']}") + + +@cli.command("crawl-sitemap") +@click.option("--sitemap", required=True, help="Sitemap URL") +@click.option("--output", required=True, type=click.Path(), help="Output directory") +@click.option("--max-pages", default=50, help="Max pages to crawl") +@click.option("--run-id", help="Custom run ID") +def crawl_sitemap(sitemap, output, max_pages, run_id): + """Crawl URLs discovered from an XML sitemap.""" + console.print(f"Parsing sitemap: {sitemap}") + urls = discover_from_sitemap(sitemap, max_urls=max_pages) + console.print(f"Found {len(urls)} URLs in sitemap") + + if not urls: + console.print("[yellow]No URLs found in sitemap[/yellow]") + return + + output_path = Path(output).expanduser() + stats = crawl_urls(urls, output_path, run_id=run_id) + console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled") + + +@cli.command("discover") +@click.option("--url", required=True, help="Root URL to discover from") +@click.option("--output", required=True, type=click.Path(), help="Output directory") +@click.option("--max-depth", default=2, help="Max link-following depth") +@click.option("--max-pages", default=50, help="Max pages to discover and crawl") +@click.option("--same-domain/--follow-external", default=True, help="Stay on same domain") +@click.option("--run-id", help="Custom run ID") +def discover(url, output, max_depth, max_pages, same_domain, run_id): + """Discover URLs from a root page, then crawl them all.""" + console.print(f"Discovering URLs from: {url} (depth={max_depth})") + urls = discover_urls(url, max_depth=max_depth, same_domain=same_domain, max_urls=max_pages) + console.print(f"Discovered {len(urls)} URLs") + + if not urls: + console.print("[yellow]No URLs discovered[/yellow]") + return + + output_path = Path(output).expanduser() + stats = crawl_urls(urls, output_path, run_id=run_id) + console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled") + + +@cli.command("crawl-manifest") +@click.option("--manifest", required=True, type=click.Path(exists=True), help="Manifest JSON file") +@click.option("--output", required=True, type=click.Path(), help="Output directory") +@click.option("--run-id", help="Custom run ID") +def crawl_manifest(manifest, output, run_id): + """Crawl URLs from a reference-curator manifest file.""" + data = json.loads(Path(manifest).read_text()) + + urls = [] + if isinstance(data, dict) and "sources" in data: + urls = [s["url"] for s in data["sources"] if s.get("url")] + elif isinstance(data, list): + urls = [item.get("url", item) if isinstance(item, dict) else item for item in data] + + console.print(f"Loaded {len(urls)} URLs from manifest") + + output_path = Path(output).expanduser() + stats = crawl_urls(urls, output_path, run_id=run_id) + console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled") + + +@cli.command("status") +@click.option("--run-id", help="Specific run ID to check") +def status(run_id): + """Check crawl progress.""" + if run_id: + filepath = PROGRESS_DIR / f"{run_id}.json" + if filepath.exists(): + data = json.loads(filepath.read_text()) + console.print_json(json.dumps(data, indent=2)) + else: + console.print(f"[yellow]No progress file for run {run_id}[/yellow]") + else: + files = sorted(PROGRESS_DIR.glob("*.json"), reverse=True)[:10] + if not files: + console.print("[yellow]No crawl history found[/yellow]") + return + + table = Table(title="Recent Crawls") + table.add_column("Run ID") + table.add_column("Status") + table.add_column("Progress") + table.add_column("Success/Total") + + for f in files: + data = json.loads(f.read_text()) + table.add_row( + data.get("run_id", "?"), + data.get("status", "?"), + f"{data.get('progress_pct', 0)}%", + f"{data.get('succeeded', 0)}/{data.get('total_urls', 0)}", + ) + console.print(table) + + +if __name__ == "__main__": + cli() diff --git a/custom-skills/90-reference-curator/04-content-distiller/code/CLAUDE.md b/custom-skills/90-reference-curator/04-content-distiller/code/CLAUDE.md index 620190f..e26a9db 100644 --- a/custom-skills/90-reference-curator/04-content-distiller/code/CLAUDE.md +++ b/custom-skills/90-reference-curator/04-content-distiller/code/CLAUDE.md @@ -5,6 +5,14 @@ Analyzes and distills raw crawled content into concise reference materials. Clau ## Trigger Keywords "distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary" +## Skip Condition + +**This entire stage is skipped when `--no-distill` is active.** + +When the pipeline orchestrator passes `--no-distill`, this stage returns immediately with status `skipped`. The pipeline advances directly from Stage 3 (content-repository) to Stage 6 (markdown-exporter), and raw crawled content becomes the export source instead of distilled content. + +No documents are loaded, no distillation occurs, and no records are written to the `distilled_content` table. + ## Goals 1. **Compress** — Reduce token count while preserving essential information diff --git a/custom-skills/90-reference-curator/05-quality-reviewer/code/CLAUDE.md b/custom-skills/90-reference-curator/05-quality-reviewer/code/CLAUDE.md index 2ba3856..a5d842d 100644 --- a/custom-skills/90-reference-curator/05-quality-reviewer/code/CLAUDE.md +++ b/custom-skills/90-reference-curator/05-quality-reviewer/code/CLAUDE.md @@ -5,6 +5,12 @@ Pre-distillation quality gate using Gemini CLI as an independent evaluator. Asse ## Trigger Keywords "review content", "quality check", "QA review", "evaluate sources", "check reference quality" +## Skip Condition + +**This entire stage is skipped when `--no-distill` is active.** + +When the pipeline uses `--no-distill`, quality review is bypassed because there is no distilled content to evaluate. Raw crawled content goes directly to the exporter. + ## Primary Flow: Gemini Pre-Distillation Gate ``` diff --git a/custom-skills/90-reference-curator/06-markdown-exporter/code/CLAUDE.md b/custom-skills/90-reference-curator/06-markdown-exporter/code/CLAUDE.md index df84618..0abec50 100644 --- a/custom-skills/90-reference-curator/06-markdown-exporter/code/CLAUDE.md +++ b/custom-skills/90-reference-curator/06-markdown-exporter/code/CLAUDE.md @@ -10,6 +10,7 @@ Exports approved reference content as structured markdown files for project know | Type | Format | Use Case | |------|--------|----------| | `project` | Nested markdown | Claude Projects knowledge | +| `raw` | Raw markdown | Archival, full-text reference, NotebookLM sources | | `finetuning` | JSONL | Model fine-tuning dataset | ## Workflow @@ -74,11 +75,57 @@ JSONL format: uv run python scripts/exporter.py log --name "April 2026 Export" --type project_files --docs 45 ``` +### Raw Export (when `--save-raw` or `--no-distill`) + +#### Raw alongside distilled (`--save-raw`) +```bash +uv run python scripts/exporter.py raw \ + --output ~/reference-library/exports/{topic-slug}/raw/ \ + --source-dir {crawl-raw-dir} \ + --bundle {topic-slug}-raw-complete.md +``` + +Output structure: +``` +exports/{topic-slug}/ +├── INDEX.md # Includes links to raw/ section +├── 00-page-name.md # Distilled (normal) +├── raw/ +│ ├── 00-page-name.raw.md +│ ├── 01-page-name.raw.md +│ └── {topic-slug}-raw-complete.md +└── manifest.json +``` + +#### Raw-only mode (`--no-distill`) +```bash +uv run python scripts/exporter.py raw \ + --output ~/reference-library/exports/{topic-slug}/ \ + --source-dir {crawl-raw-dir} \ + --bundle {topic-slug}-raw-complete.md \ + --primary +``` + +Output structure (raw files are primary, no subdirectory wrapper): +``` +exports/{topic-slug}/ +├── README.md +├── 00-page-name.raw.md +├── 01-page-name.raw.md +├── {topic-slug}-raw-complete.md +└── manifest.json +``` + +The `--primary` flag tells the exporter that raw files are the main output. README is generated from raw file frontmatter (source URLs, crawl timestamps). + +**Note:** When combined with `--export-format fine_tuning`, the JSONL exporter uses raw content directly as the assistant response with a simpler system prompt (no structured summaries available). + ## Scripts | Command | Purpose | |---------|---------| | `exporter.py project` | Export as nested markdown files | +| `exporter.py raw` | Export raw crawled markdown files with optional bundle | | `exporter.py finetuning` | Export as JSONL training dataset | | `exporter.py index` | Generate INDEX.md table of contents | | `exporter.py crossrefs` | Add cross-reference links | @@ -90,4 +137,5 @@ uv run python scripts/exporter.py log --name "April 2026 Export" --type project_ | From | To | |------|-----| | quality-reviewer (approved) | → | +| web-crawler (raw files) | Raw markdown input (--save-raw / --no-distill) | | → | Project knowledge / Fine-tuning dataset | diff --git a/custom-skills/90-reference-curator/07-pipeline-orchestrator/code/CLAUDE.md b/custom-skills/90-reference-curator/07-pipeline-orchestrator/code/CLAUDE.md index 6b498ce..d22ea5d 100644 --- a/custom-skills/90-reference-curator/07-pipeline-orchestrator/code/CLAUDE.md +++ b/custom-skills/90-reference-curator/07-pipeline-orchestrator/code/CLAUDE.md @@ -9,17 +9,20 @@ Coordinates the full 6-skill reference curation workflow with QA loop handling. ``` [Input] → discovery → crawler → repository → distiller ◄──┐ - │ │ - reviewer │ - │ │ - ┌───────────────────────────────┼─────┤ - ▼ ▼ ▼ │ - APPROVE REJECT REFACTOR ────┤ - │ │ │ - ▼ ▼ DEEP_RESEARCH - export archive │ - ▼ - crawler ─┘ + │ │ │ + │ (--save-raw) reviewer │ + │ writes raw/ │ │ + │ ┌───────┼─────┤ + │ ▼ ▼ │ + │ APPROVE REJECT REFACTOR ────┤ + │ │ │ │ + │ ▼ ▼ DEEP_RESEARCH + │ export archive │ + │ │ ▼ + │ │ crawler ─┘ + │ │ + └── (--no-distill) ──► export (raw only) + skip stages 4+5 ``` ## Pipeline State Management @@ -28,7 +31,7 @@ Coordinates the full 6-skill reference curation workflow with QA loop handling. ```bash uv run python scripts/pipeline.py init \ --input "prompt engineering" --type topic \ - --options '{"max_sources": 10, "auto_approve": true}' + --options '{"max_sources": 10, "auto_approve": true, "save_raw": false, "no_distill": false}' ``` ### Advance to Next Stage @@ -99,6 +102,7 @@ If mode == 'topic': ``` → Claude uses Firecrawl MCP tools → crawl_mgr.py store-result +→ If --save-raw or --no-distill: write raw .raw.md files to output → pipeline.py advance --stage storing ``` @@ -108,7 +112,23 @@ If mode == 'topic': → pipeline.py advance --stage evaluating ``` +### Flag Resolution (after Stage 3) +``` +If --no-distill: + → set save_raw = true (implied) + → skip Stage 4 (content-distiller) + → skip Stage 5 (quality-reviewer) + → pipeline.py advance --stage exporting --skip-reason "no-distill" + → proceed directly to Stage 6 with raw files as input + +If --save-raw (without --no-distill): + → raw files already written during Stage 2 + → continue normal pipeline through stages 4-5 + → Stage 6 exports both distilled AND raw content +``` + ### Stage 4: Gemini Quality Gate (Pre-Distillation) +**Skipped when `--no-distill` is active.** ``` → reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve → APPROVE: proceed to distillation @@ -118,6 +138,7 @@ If mode == 'topic': ``` ### Stage 5: Content Distiller (Approved Only) +**Skipped when `--no-distill` is active.** ``` → distiller.py load-pending → Claude distills each approved document @@ -127,11 +148,26 @@ If mode == 'topic': ### Stage 6: Markdown Exporter ``` -→ exporter.py project -→ exporter.py index -→ exporter.py crossrefs -→ exporter.py verify -→ pipeline.py complete +If --no-distill: + → exporter.py raw --primary (raw files are the main output) + → exporter.py index (generate README from raw file metadata) + → exporter.py verify + → pipeline.py complete + +If --save-raw (with distillation): + → exporter.py project (normal distilled export) + → exporter.py raw (write raw/ subdirectory + bundle) + → exporter.py index (includes raw section) + → exporter.py crossrefs + → exporter.py verify + → pipeline.py complete + +Default (no raw flags): + → exporter.py project + → exporter.py index + → exporter.py crossrefs + → exporter.py verify + → pipeline.py complete ``` ## Checkpoint Strategy @@ -139,8 +175,9 @@ If mode == 'topic': | Stage | Checkpoint | Resume Point | |-------|------------|--------------| | discovery | manifest.json created | → crawler | -| crawl | crawl_result.json | → repository | -| store | DB records | → distiller | +| crawl | crawl_result.json + raw files | → repository | +| store | DB records | → distiller (or skip) | +| no-distill skip | stages 4-5 marked skipped | → exporter (raw) | | distill | distilled_content records | → reviewer | | review | review_logs records | → exporter or loop | | export | final export complete | Done | @@ -167,6 +204,8 @@ pipeline: max_pages: 50 auto_approve: false approval_threshold: 0.85 + save_raw: false + no_distill: false qa_loop: max_refactor_iterations: 3 diff --git a/custom-skills/90-reference-curator/README.md b/custom-skills/90-reference-curator/README.md index 3bf145d..1a27209 100644 --- a/custom-skills/90-reference-curator/README.md +++ b/custom-skills/90-reference-curator/README.md @@ -193,6 +193,15 @@ Run the complete curation workflow with a single command: # Fine-tuning dataset output /reference-curator-pipeline "MCP servers" --export-format fine_tuning + +# Save raw crawled content alongside distilled output +/reference-curator-pipeline "Claude Code best practices" --save-raw + +# Pure archival — skip distillation, keep raw markdown only +/reference-curator-pipeline https://docs.anthropic.com --no-distill + +# Full-depth archival +/reference-curator-pipeline https://legacy-docs.example.com --depth full --no-distill ``` **Pipeline Options:** @@ -202,6 +211,8 @@ Run the complete curation workflow with a single command: - `--threshold 0.85` - Approval threshold - `--max-iterations 3` - Max QA loop iterations per document - `--export-format project_files` - Output format (project_files, fine_tuning, jsonl) +- `--save-raw` - Also save intact raw crawled markdown in a `raw/` subdirectory +- `--no-distill` - Skip distillation and QA (pure archival). Implies `--save-raw` --- @@ -313,6 +324,29 @@ mysql -h $MYSQL_HOST -u $MYSQL_USER -p"$MYSQL_PASSWORD" reference_library -e " └── chain-of-thought.md ``` +**Raw Archival (`--no-distill`):** +``` +~/reference-library/exports/ +└── topic-slug/ + ├── README.md + ├── 00-page-name.raw.md # Intact crawled content + ├── 01-page-name.raw.md + ├── topic-slug-raw-complete.md # Combined raw bundle + └── manifest.json +``` + +**Distilled + Raw (`--save-raw`):** +``` +~/reference-library/exports/ +└── topic-slug/ + ├── INDEX.md + ├── 00-page-name.md # Distilled + ├── raw/ + │ ├── 00-page-name.raw.md # Intact crawled + │ └── topic-slug-raw-complete.md + └── manifest.json +``` + **For Fine-tuning (JSONL):** ```json {"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]} diff --git a/custom-skills/90-reference-curator/USER-GUIDE.md b/custom-skills/90-reference-curator/USER-GUIDE.md index 33b937d..d6bb523 100644 --- a/custom-skills/90-reference-curator/USER-GUIDE.md +++ b/custom-skills/90-reference-curator/USER-GUIDE.md @@ -15,6 +15,12 @@ A modular skill suite for curating reference documentation with Claude Code. # With options /reference-curator-pipeline "MCP servers" --max-sources 10 --export-format fine_tuning + +# Save raw crawled content alongside distilled output +/reference-curator-pipeline "Claude Code best practices" --save-raw + +# Pure archival — skip distillation, raw markdown only +/reference-curator-pipeline https://docs.anthropic.com --no-distill ``` ### Run as Background Agent @@ -156,6 +162,89 @@ Use these when you need granular control over each stage. --- +## Raw Mode + +### What is Raw Mode? + +By default, the pipeline distills crawled content down to 25-35% of the original, extracting key concepts, code snippets, and structured summaries. **Raw mode** preserves the full, unmodified crawled markdown. + +### `--save-raw` Flag + +Saves intact crawled markdown **alongside** the distilled content in a `raw/` subdirectory. + +```bash +/reference-curator-pipeline "Anthropic Claude API" --save-raw +``` + +**Output:** +``` +exports/claude-api/ +├── INDEX.md +├── 00-prompt-caching.md # Distilled (25-35% of original) +├── 01-streaming.md +├── raw/ +│ ├── 00-prompt-caching.raw.md # Full crawled content +│ ├── 01-streaming.raw.md +│ └── claude-api-raw-complete.md # Combined raw bundle +└── manifest.json +``` + +**Use cases:** +- Need both concise reference AND full-text source material +- Want to re-distill later with different parameters +- Auditing what content was captured vs. what was extracted + +### `--no-distill` Flag + +Skips stages 4 (content-distiller) and 5 (quality-reviewer) entirely. The pipeline goes directly from crawl/store to export. Raw files become the **primary** output. Implies `--save-raw` automatically. + +```bash +/reference-curator-pipeline https://docs.anthropic.com --no-distill +``` + +**Output:** +``` +exports/anthropic-docs/ +├── README.md +├── 00-getting-started.raw.md +├── 01-prompt-engineering.raw.md +├── anthropic-docs-raw-complete.md +└── manifest.json +``` + +**Use cases:** +- Site archival / preservation +- Migration reference (need exact original content) +- Training data collection (distillation would remove useful context) +- NotebookLM / LLM context sources (full fidelity preferred) +- Quick capture when you will distill manually later + +### Combining with Other Flags + +```bash +# Deep archival crawl +/reference-curator-pipeline https://legacy.example.com --depth full --no-distill + +# Standard crawl with raw backup and auto-approval +/reference-curator-pipeline "MCP servers" --save-raw --auto-approve + +# Fine-tuning dataset from raw content +/reference-curator-pipeline "prompt engineering" --no-distill --export-format fine_tuning +``` + +### Pipeline Stage Comparison + +| Stage | Default | `--save-raw` | `--no-distill` | +|-------|---------|-------------|----------------| +| 1. Discovery | Runs | Runs | Runs | +| 2. Web Crawler | Runs | Runs + writes raw/ | Runs + writes raw | +| 3. Repository | Runs | Runs | Runs | +| 4. Distiller | Runs | Runs | **Skipped** | +| 5. Reviewer | Runs | Runs | **Skipped** | +| 6. Exporter | Distilled | Distilled + raw | **Raw only** | + +--- + ## Common Workflows ### Workflow 1: Quick Reference on a Topic @@ -197,6 +286,8 @@ Claude: /markdown-exporter --format fine_tuning | Type | Location | |------|----------| | Raw content | `~/Documents/05_AI Agent/10_Reference Library/raw/` | +| Raw output (--save-raw) | `{output}/{topic-slug}/raw/` | +| Raw output (--no-distill) | `{output}/{topic-slug}/` (primary) | | Exports | `~/Documents/05_AI Agent/10_Reference Library/exports/` | | Config | `~/.config/reference-curator/` | @@ -288,6 +379,34 @@ Claude: Starting reference-curator pipeline... Pipeline complete! Reference library ready. ``` +### Example: Archival Mode (`--no-distill`) + +``` +You: Archive the GA4 developer docs without distilling — just raw content + +Claude: Starting reference-curator pipeline (--no-distill mode)... + +[1/6] Reference Discovery +- Skipped (URL input mode) + +[2/6] Web Crawling +- Crawling developers.google.com/analytics with Firecrawl... +- Saved 28 raw markdown files (.raw.md) + +[3/6] Content Repository +- Stored 28 documents in MySQL + +[4/6] Content Distillation — SKIPPED (--no-distill) +[5/6] Quality Review — SKIPPED (--no-distill) + +[6/6] Markdown Export (raw mode) +- Generated 28 .raw.md files + combined raw bundle +- Output: ~/Documents/.../google-analytics-4/ + +Pipeline complete! 28 raw pages archived. +Files ready for NotebookLM upload or direct LLM context use. +``` + --- ## Need Help? diff --git a/custom-skills/90-reference-curator/commands/reference-curator-pipeline.md b/custom-skills/90-reference-curator/commands/reference-curator-pipeline.md index f7424e6..b8b5f03 100644 --- a/custom-skills/90-reference-curator/commands/reference-curator-pipeline.md +++ b/custom-skills/90-reference-curator/commands/reference-curator-pipeline.md @@ -1,6 +1,6 @@ --- description: Orchestrates full reference curation pipeline with configurable depth. Runs discovery > crawl > store > distill > review > export with QA loop handling. -argument-hint: [--depth light|standard|deep|full] [--output ~/Documents/reference-library/] [--max-sources 100] [--max-pages 50] [--auto-approve] [--threshold 0.85] [--max-iterations 3] [--export-format project_files] +argument-hint: [--depth light|standard|deep|full] [--output ~/Documents/reference-library/] [--max-sources 100] [--max-pages 50] [--auto-approve] [--threshold 0.85] [--max-iterations 3] [--export-format project_files] [--save-raw] [--no-distill] allowed-tools: WebSearch, WebFetch, Read, Write, Bash, Grep, Glob, Task --- @@ -29,6 +29,8 @@ Full-stack orchestration of the 6-skill reference curation workflow. - `--export-format`: Output format: `project_files`, `fine_tuning`, `jsonl` (default: project_files) - `--include-subdomains`: Include subdomains in site mapping (default: false) - `--follow-external`: Follow external links found in content (default: false) +- `--save-raw`: Save intact crawled markdown alongside distilled content in a `raw/` subdirectory. Each file uses `.raw.md` extension. A combined raw bundle `{topic-slug}-raw-complete.md` is also generated. +- `--no-distill`: Skip stages 4-5 (content-distiller and quality-reviewer) entirely. Pure archival mode. Implies `--save-raw`. Raw crawled files become the primary output — no distilled subdirectory is created. ## Output Directory @@ -38,13 +40,13 @@ The `--output` argument sets the base path for all pipeline output. If omitted, ### Directory Structure +**Default (distilled output):** ``` {output}/ -├── {topic-slug}/ # One folder per pipeline run +├── {topic-slug}/ │ ├── README.md # Index with table of contents -│ ├── 00-page-name.md # Individual page files +│ ├── 00-page-name.md # Distilled individual page files │ ├── 01-page-name.md -│ ├── ... │ ├── {topic-slug}-complete.md # Combined bundle (all pages) │ └── manifest.json # Crawl metadata ├── pipeline_state/ # Resume state (auto-managed) @@ -52,6 +54,31 @@ The `--output` argument sets the base path for all pipeline output. If omitted, └── exports/ # Fine-tuning / JSONL exports ``` +**With `--save-raw` (distilled + raw):** +``` +{output}/ +├── {topic-slug}/ +│ ├── README.md +│ ├── 00-page-name.md # Distilled +│ ├── {topic-slug}-complete.md # Combined distilled +│ ├── raw/ # Intact crawled markdown +│ │ ├── 00-page-name.raw.md +│ │ ├── 01-page-name.raw.md +│ │ └── {topic-slug}-raw-complete.md # Combined raw bundle +│ └── manifest.json +``` + +**With `--no-distill` (raw only, archival):** +``` +{output}/ +├── {topic-slug}/ +│ ├── README.md +│ ├── 00-page-name.raw.md # Raw crawled (primary output) +│ ├── 01-page-name.raw.md +│ ├── {topic-slug}-raw-complete.md # Combined raw bundle +│ └── manifest.json +``` + ### Resolution Rules 1. If `--output` is provided, use that path exactly @@ -158,15 +185,17 @@ full ████████████████ Speed: slow ``` 1. reference-discovery (topic mode only) -2. web-crawler <- depth controls this stage +2. web-crawler ← depth controls this stage + ← --save-raw: writes raw/ alongside crawl 3. content-repository -4. content-distiller <--------+ -5. quality-reviewer | +4. content-distiller <--------+ ← SKIPPED when --no-distill +5. quality-reviewer | ← SKIPPED when --no-distill +-- APPROVE -> export | +-- REFACTOR -----------------+ +-- DEEP_RESEARCH -> crawler -+ +-- REJECT -> archive -6. markdown-exporter +6. markdown-exporter ← --no-distill: exports raw files only + ← --save-raw: also exports raw/ bundle ``` ## Crawl Execution by Depth @@ -234,6 +263,12 @@ After max iterations, document marked as `needs_manual_review`. # Resume from existing manifest /reference-curator ./manifest.json --auto-approve + +# Save raw crawled content alongside distilled output +/reference-curator https://docs.anthropic.com --depth standard --save-raw + +# Pure archival — no distillation, just raw markdown files +/reference-curator https://docs.anthropic.com --depth full --no-distill ``` ## State Management diff --git a/custom-skills/90-reference-curator/shared/config/crawl_config.yaml b/custom-skills/90-reference-curator/shared/config/crawl_config.yaml index 19b9e9d..39ba3b1 100644 --- a/custom-skills/90-reference-curator/shared/config/crawl_config.yaml +++ b/custom-skills/90-reference-curator/shared/config/crawl_config.yaml @@ -6,13 +6,22 @@ # REFERENCE_LIBRARY_PATH - Path to reference library storage (optional) # Default crawler backend -# Options: nodejs, aiohttp, scrapy, firecrawl -# Set to "firecrawl" if local crawlers are not available -default_crawler: ${DEFAULT_CRAWLER:-firecrawl} +# Options: seo-aiohttp, firecrawl, nodejs, aiohttp, scrapy +# seo-aiohttp: Uses OurSEO BaseAsyncClient + PageAnalyzer (recommended for docs) +# firecrawl: MCP-based, best for SPAs/JS-rendered sites +default_crawler: ${DEFAULT_CRAWLER:-seo-aiohttp} # Intelligent routing rules # Claude will select the appropriate crawler based on these criteria routing: + seo-aiohttp: + conditions: + - is_documentation_site == true + - has_sitemap == true + - needs_metadata == true + - max_pages <= 200 + description: "OurSEO async crawler — rate-limited, resumable, full metadata extraction. Best for documentation sites." + nodejs: conditions: - max_pages <= 50 @@ -42,8 +51,22 @@ routing: description: "JS rendering - best for SPAs and dynamic content (MCP-based, always available)" # Crawler locations (configurable via environment) -# If CRAWLER_PROJECT_PATH is not set, only Firecrawl MCP will be available crawlers: + seo-aiohttp: + enabled: true # Always available (uses OurSEO scripts from 12-seo-technical-audit) + type: local + path: ${SKILL_PATH:-../../../90-reference-curator}/02-web-crawler-orchestrator/code/scripts/ + command: python seo_crawler_adapter.py + capabilities: + - rate_limiting + - progress_tracking + - resume_support + - metadata_extraction + - sitemap_discovery + - link_discovery + - raw_file_output + install: "pip install requests beautifulsoup4 click rich tenacity" + nodejs: enabled: ${NODEJS_CRAWLER_ENABLED:-false} path: ${CRAWLER_PROJECT_PATH}/util/js-crawler/