feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration
- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory alongside distilled content, with YAML frontmatter for traceability - Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer), pure archival mode for raw documentation capture - Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend: token-bucket rate limiting, semaphore concurrency, tenacity retries, full page metadata extraction (headings, links, schema, OG) - New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests with progress tracking and resume support - Update crawler selection: docs sites now default to seo-aiohttp - Update crawl_config.yaml with seo-aiohttp routing rules - Update pipeline orchestrator with flag resolution and skip paths - Update README.md and USER-GUIDE.md with raw mode documentation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Web Crawler Orchestrator
|
||||
|
||||
Orchestrates web crawling with intelligent backend selection. Claude performs actual crawling via Firecrawl MCP tools. This skill manages crawl results, selects crawlers, and tracks crawl metadata.
|
||||
Orchestrates web crawling with intelligent backend selection. Primary backend is the OurSEO async crawler (rate-limited, resumable, metadata-rich). Falls back to Firecrawl MCP for JS-rendered sites or WebFetch for simple pages.
|
||||
|
||||
## Trigger Keywords
|
||||
"crawl URLs", "fetch documents", "scrape pages", "download references"
|
||||
@@ -14,9 +14,10 @@ uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.
|
||||
|
||||
| Crawler | Best For | Auto-Selected When |
|
||||
|---------|----------|-------------------|
|
||||
| **Firecrawl MCP** (default) | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered |
|
||||
| **SEO aiohttp** (default) | Documentation sites, sitemaps | docs/developer/api domains, has sitemap |
|
||||
| **Firecrawl MCP** | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered |
|
||||
| **WebFetch** | Quick single pages | Fallback, simple pages |
|
||||
| **Node.js** | Small docs sites | ≤50 pages, static content |
|
||||
| **Python aiohttp** | Technical docs | ≤200 pages, needs SEO data |
|
||||
| **Scrapy** | Enterprise crawls | >200 pages, multi-domain |
|
||||
|
||||
## Workflow
|
||||
@@ -24,20 +25,69 @@ uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.
|
||||
### Step 1: Analyze Target Site
|
||||
Run `select-crawler` to determine site characteristics and get a recommendation.
|
||||
|
||||
### Step 2: Execute Crawl
|
||||
Use Firecrawl MCP tools directly:
|
||||
### Step 2a: Execute Crawl — SEO aiohttp (Recommended)
|
||||
|
||||
The SEO crawler adapter provides rate limiting, progress tracking, resume support, and full metadata extraction.
|
||||
|
||||
```bash
|
||||
# Crawl specific URLs
|
||||
uv run python scripts/seo_crawler_adapter.py crawl \
|
||||
--urls "https://docs.example.com/page1,https://docs.example.com/page2" \
|
||||
--output ~/Documents/reference-library/topic-slug/raw/
|
||||
|
||||
# Crawl from XML sitemap (auto-discovers URLs)
|
||||
uv run python scripts/seo_crawler_adapter.py crawl-sitemap \
|
||||
--sitemap https://docs.example.com/sitemap.xml \
|
||||
--output ~/Documents/reference-library/topic-slug/raw/ \
|
||||
--max-pages 50
|
||||
|
||||
# Discover URLs from root page (follows internal links)
|
||||
uv run python scripts/seo_crawler_adapter.py discover \
|
||||
--url https://docs.example.com \
|
||||
--output ~/Documents/reference-library/topic-slug/raw/ \
|
||||
--max-depth 2 --max-pages 50 --same-domain
|
||||
|
||||
# Crawl from reference-curator manifest
|
||||
uv run python scripts/seo_crawler_adapter.py crawl-manifest \
|
||||
--manifest ./manifest.json \
|
||||
--output ~/Documents/reference-library/topic-slug/raw/
|
||||
|
||||
# Check crawl progress
|
||||
uv run python scripts/seo_crawler_adapter.py status
|
||||
```
|
||||
|
||||
**SEO crawler capabilities:**
|
||||
- Token-bucket rate limiting (via `BaseAsyncClient`)
|
||||
- Semaphore-controlled concurrency (default: 5 concurrent)
|
||||
- Tenacity retry with exponential backoff
|
||||
- Full `PageMetadata` extraction: title, headings, links, schema, OG
|
||||
- Progress tracking with ETA and resume files
|
||||
- Sitemap XML parsing (including sitemap indexes)
|
||||
- Internal link discovery (breadth-first, depth-limited)
|
||||
- Direct `.raw.md` output with enriched YAML frontmatter
|
||||
|
||||
### Step 2b: Execute Crawl — Firecrawl MCP (for JS-rendered sites)
|
||||
|
||||
Use Firecrawl MCP tools directly when JS rendering is required:
|
||||
```
|
||||
firecrawl_map(url, limit=100) # Discover URLs
|
||||
firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
|
||||
firecrawl_crawl(url, max_depth=2, limit=50)
|
||||
```
|
||||
|
||||
### Step 2c: Execute Crawl — WebFetch (fallback)
|
||||
|
||||
For quick single-page fetches when no crawl infrastructure is needed:
|
||||
```
|
||||
WebFetch(url, prompt="Return the COMPLETE page content as clean markdown.")
|
||||
```
|
||||
|
||||
### Step 3: Store Crawl Results
|
||||
```bash
|
||||
# Store crawled files and create result manifest
|
||||
uv run python scripts/crawl_mgr.py store-result \
|
||||
--raw-dir ~/Documents/reference-library/raw/ \
|
||||
--crawler firecrawl \
|
||||
--crawler seo-aiohttp \
|
||||
--source-id 1 \
|
||||
--output crawl_result.json
|
||||
|
||||
@@ -45,27 +95,66 @@ uv run python scripts/crawl_mgr.py store-result \
|
||||
uv run python scripts/crawl_mgr.py list-crawls --status completed
|
||||
```
|
||||
|
||||
### Step 4: Write Raw Files (when `--save-raw` or `--no-distill`)
|
||||
|
||||
When the pipeline is invoked with `--save-raw` (or `--no-distill`, which implies it), write each crawled page's markdown verbatim to the raw output directory.
|
||||
|
||||
**Note:** The SEO crawler adapter (`seo_crawler_adapter.py`) writes `.raw.md` files directly with enriched YAML frontmatter — this step is automatic when using seo-aiohttp.
|
||||
|
||||
**File naming:** `{NN}-{page-slug}.raw.md` — zero-padded index matching crawl order.
|
||||
|
||||
**Output path:**
|
||||
- `--save-raw`: `{output}/{topic-slug}/raw/`
|
||||
- `--no-distill`: `{output}/{topic-slug}/` (raw files are primary)
|
||||
|
||||
**YAML frontmatter (SEO crawler enriched):**
|
||||
```yaml
|
||||
---
|
||||
source_url: {original_url}
|
||||
crawled_at: {ISO 8601 timestamp}
|
||||
crawler: seo-aiohttp
|
||||
content_length: {character count}
|
||||
raw: true
|
||||
status_code: {HTTP status}
|
||||
title: {page title}
|
||||
word_count: {word count}
|
||||
internal_links: {count}
|
||||
external_links: {count}
|
||||
h1: {h1 text}
|
||||
schema_types: [{types found}]
|
||||
response_time_ms: {ms}
|
||||
---
|
||||
```
|
||||
|
||||
**After all pages are written**, a combined raw bundle (`{topic-slug}-raw-complete.md`) is auto-generated by concatenating all `.raw.md` files with `---` separators.
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
All crawlers respect these limits:
|
||||
- 20 requests/minute
|
||||
- 3 concurrent requests
|
||||
- Exponential backoff on 429/5xx
|
||||
- 20 requests/minute (SEO crawler: configurable via `BaseAsyncClient`)
|
||||
- 3-5 concurrent requests (SEO crawler: semaphore-controlled)
|
||||
- Exponential backoff on 429/5xx (SEO crawler: tenacity)
|
||||
|
||||
## Error Handling
|
||||
|
||||
| Error | Action |
|
||||
|-------|--------|
|
||||
| Timeout | Retry once with 2x timeout |
|
||||
| Rate limit (429) | Exponential backoff, max 3 retries |
|
||||
| Timeout | Retry with exponential backoff (max 3 attempts) |
|
||||
| Rate limit (429) | Token-bucket + exponential backoff |
|
||||
| Not found (404) | Log and skip |
|
||||
| Access denied (403) | Log, mark as `failed` |
|
||||
| JS rendering needed | Switch to Firecrawl |
|
||||
| JS rendering needed | Switch to Firecrawl MCP |
|
||||
| Connection error | Retry with backoff, then skip |
|
||||
|
||||
## Scripts
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `seo_crawler_adapter.py crawl` | Crawl URLs with SEO metadata extraction |
|
||||
| `seo_crawler_adapter.py crawl-sitemap` | Discover and crawl from XML sitemap |
|
||||
| `seo_crawler_adapter.py discover` | BFS link discovery from root URL |
|
||||
| `seo_crawler_adapter.py crawl-manifest` | Crawl from reference-curator manifest |
|
||||
| `seo_crawler_adapter.py status` | Check crawl progress |
|
||||
| `crawl_mgr.py select-crawler` | Recommend optimal crawler for a URL |
|
||||
| `crawl_mgr.py store-result` | Store crawl results and create manifest |
|
||||
| `crawl_mgr.py list-crawls` | List recent crawl records |
|
||||
@@ -75,6 +164,7 @@ All crawlers respect these limits:
|
||||
| From | To |
|
||||
|------|-----|
|
||||
| reference-discovery | URL manifest input |
|
||||
| Firecrawl MCP | Raw crawled files |
|
||||
| SEO aiohttp / Firecrawl MCP | Raw crawled files |
|
||||
| → | content-repository (crawl manifest + raw files) |
|
||||
| → | raw/ directory (when --save-raw or --no-distill) |
|
||||
| quality-reviewer (deep_research) | Additional crawl requests |
|
||||
|
||||
@@ -53,7 +53,7 @@ def cli():
|
||||
@click.option("--raw-dir", required=True, type=click.Path(exists=True),
|
||||
help="Directory containing crawled raw files")
|
||||
@click.option("--crawler", default="firecrawl",
|
||||
type=click.Choice(["firecrawl", "nodejs", "aiohttp", "scrapy"]))
|
||||
type=click.Choice(["firecrawl", "seo-aiohttp", "nodejs", "aiohttp", "scrapy"]))
|
||||
@click.option("--source-id", type=int, help="Source ID to associate documents with")
|
||||
@click.option("--output", type=click.Path(), help="Output crawl result JSON path")
|
||||
def store_result(raw_dir, crawler, source_id, output):
|
||||
@@ -123,9 +123,10 @@ def select_crawler(url, page_count, json_output):
|
||||
"""Recommend the best crawler backend for a URL.
|
||||
|
||||
Analyzes URL patterns and site characteristics to suggest:
|
||||
- seo-aiohttp: Documentation sites, sitemaps, HTML-renderable (NEW default for docs)
|
||||
- firecrawl: SPAs, JS-rendered content
|
||||
- nodejs: Small static docs sites (<=50 pages)
|
||||
- aiohttp: Medium technical docs (<=200 pages)
|
||||
- aiohttp: Medium technical docs (<=200 pages, legacy)
|
||||
- scrapy: Large enterprise sites (>200 pages)
|
||||
"""
|
||||
parsed = urlparse(url)
|
||||
@@ -169,12 +170,13 @@ def select_crawler(url, page_count, json_output):
|
||||
reason = f"Small site ({page_count} pages) — lightweight crawler sufficient"
|
||||
confidence = 0.8
|
||||
|
||||
# Known documentation platforms → firecrawl
|
||||
doc_platforms = ["docs.", "developer.", "api.", "reference."]
|
||||
# Known documentation platforms → seo-aiohttp (preferred for docs)
|
||||
doc_platforms = ["docs.", "developer.", "api.", "reference.", "support."]
|
||||
if any(domain.startswith(p) for p in doc_platforms):
|
||||
if recommendation == "firecrawl":
|
||||
reason = "Documentation platform — Firecrawl handles dynamic docs well"
|
||||
confidence = 0.85
|
||||
if not spa_detected:
|
||||
recommendation = "seo-aiohttp"
|
||||
reason = "Documentation platform — SEO crawler provides metadata + rate limiting + resume"
|
||||
confidence = 0.90
|
||||
|
||||
result = {
|
||||
"url": url,
|
||||
@@ -265,7 +267,7 @@ def _extract_url_from_content(content: str) -> str | None:
|
||||
|
||||
def _get_alternatives(primary: str) -> list[str]:
|
||||
"""Get alternative crawler recommendations."""
|
||||
all_crawlers = ["firecrawl", "nodejs", "aiohttp", "scrapy"]
|
||||
all_crawlers = ["seo-aiohttp", "firecrawl", "nodejs", "aiohttp", "scrapy"]
|
||||
return [c for c in all_crawlers if c != primary][:2]
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,521 @@
|
||||
#!/usr/bin/env python3
|
||||
"""SEO Crawler Adapter — bridges the OurSEO async crawler stack into reference-curator.
|
||||
|
||||
Uses BaseAsyncClient (rate limiting, concurrency, retries) and PageAnalyzer
|
||||
(full HTML parsing, metadata extraction) from the SEO toolkit to provide a
|
||||
production-grade crawling backend for reference curation.
|
||||
|
||||
Outputs .raw.md files with YAML frontmatter compatible with --save-raw / --no-distill.
|
||||
|
||||
Usage:
|
||||
# Crawl a list of URLs
|
||||
python seo_crawler_adapter.py crawl \
|
||||
--urls "https://docs.example.com/page1,https://docs.example.com/page2" \
|
||||
--output ~/Documents/reference-library/topic-slug/raw/
|
||||
|
||||
# Crawl from sitemap
|
||||
python seo_crawler_adapter.py crawl-sitemap \
|
||||
--sitemap https://docs.example.com/sitemap.xml \
|
||||
--output ~/Documents/reference-library/topic-slug/raw/ \
|
||||
--max-pages 50
|
||||
|
||||
# Crawl from manifest file
|
||||
python seo_crawler_adapter.py crawl-manifest \
|
||||
--manifest ./manifest.json \
|
||||
--output ~/Documents/reference-library/topic-slug/raw/
|
||||
|
||||
# Discover URLs from a root page (link extraction)
|
||||
python seo_crawler_adapter.py discover \
|
||||
--url https://docs.example.com \
|
||||
--max-depth 2 --same-domain
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
import click
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
from rich.console import Console
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskID
|
||||
from rich.table import Table
|
||||
|
||||
# Import from SEO toolkit (shared base_client)
|
||||
SEO_SCRIPTS = Path(__file__).resolve().parents[4] / "12-seo-technical-audit" / "code" / "scripts"
|
||||
if str(SEO_SCRIPTS) not in sys.path:
|
||||
sys.path.insert(0, str(SEO_SCRIPTS))
|
||||
|
||||
from base_client import BaseAsyncClient, RateLimiter # noqa: E402
|
||||
from page_analyzer import PageAnalyzer, PageMetadata # noqa: E402
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
|
||||
logger = logging.getLogger(__name__)
|
||||
console = Console()
|
||||
|
||||
# Progress tracking directory
|
||||
PROGRESS_DIR = Path.home() / ".claude" / "reference-curator-progress"
|
||||
PROGRESS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
@dataclass
|
||||
class CrawlProgress:
|
||||
"""Track crawl progress for reference curation."""
|
||||
run_id: str = ""
|
||||
total_urls: int = 0
|
||||
processed: int = 0
|
||||
succeeded: int = 0
|
||||
failed: int = 0
|
||||
skipped: int = 0
|
||||
start_time: datetime = field(default_factory=datetime.now)
|
||||
current_url: str = ""
|
||||
status: str = "running"
|
||||
|
||||
def save(self) -> Path:
|
||||
filepath = PROGRESS_DIR / f"{self.run_id}.json"
|
||||
filepath.write_text(json.dumps({
|
||||
"run_id": self.run_id,
|
||||
"status": self.status,
|
||||
"total_urls": self.total_urls,
|
||||
"processed": self.processed,
|
||||
"succeeded": self.succeeded,
|
||||
"failed": self.failed,
|
||||
"skipped": self.skipped,
|
||||
"progress_pct": round(self.processed / max(self.total_urls, 1) * 100, 1),
|
||||
"current_url": self.current_url,
|
||||
"start_time": self.start_time.isoformat(),
|
||||
"updated_at": datetime.now().isoformat(),
|
||||
}, indent=2))
|
||||
return filepath
|
||||
|
||||
|
||||
def _slugify(text: str) -> str:
|
||||
"""Convert text to URL-friendly slug."""
|
||||
text = text.lower().strip()
|
||||
text = re.sub(r'[^\w\s-]', '', text)
|
||||
text = re.sub(r'[-\s]+', '-', text)
|
||||
return text[:60].rstrip('-')
|
||||
|
||||
|
||||
def _extract_main_content(html: str, url: str) -> str:
|
||||
"""Extract main content as markdown from HTML.
|
||||
|
||||
Uses BeautifulSoup to strip nav/footer/sidebar, then converts
|
||||
remaining content to clean text with heading structure preserved.
|
||||
"""
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
|
||||
# Remove non-content elements
|
||||
for tag in soup(["script", "style", "noscript", "nav", "footer",
|
||||
"header", "aside", "iframe"]):
|
||||
tag.decompose()
|
||||
|
||||
# Try to find main content area
|
||||
main = (soup.find("main") or soup.find("article") or
|
||||
soup.find(role="main") or soup.find(id=re.compile(r"content|main", re.I)) or
|
||||
soup.find(class_=re.compile(r"content|main|article", re.I)) or soup.body or soup)
|
||||
|
||||
lines = []
|
||||
for element in main.descendants:
|
||||
if element.name and element.name.startswith('h') and len(element.name) == 2:
|
||||
level = int(element.name[1])
|
||||
text = element.get_text(strip=True)
|
||||
if text:
|
||||
lines.append(f"\n{'#' * level} {text}\n")
|
||||
elif element.name == 'p':
|
||||
text = element.get_text(strip=True)
|
||||
if text:
|
||||
lines.append(f"\n{text}\n")
|
||||
elif element.name == 'li':
|
||||
text = element.get_text(strip=True)
|
||||
if text:
|
||||
lines.append(f"- {text}")
|
||||
elif element.name == 'pre':
|
||||
code = element.get_text()
|
||||
lang = ""
|
||||
code_tag = element.find("code")
|
||||
if code_tag:
|
||||
classes = code_tag.get("class", [])
|
||||
for cls in classes:
|
||||
if cls.startswith("language-"):
|
||||
lang = cls[9:]
|
||||
break
|
||||
lines.append(f"\n```{lang}\n{code}\n```\n")
|
||||
elif element.name == 'table':
|
||||
lines.append(_table_to_markdown(element))
|
||||
|
||||
return "\n".join(lines).strip()
|
||||
|
||||
|
||||
def _table_to_markdown(table_tag) -> str:
|
||||
"""Convert an HTML table to markdown."""
|
||||
rows = table_tag.find_all("tr")
|
||||
if not rows:
|
||||
return ""
|
||||
|
||||
md_rows = []
|
||||
for i, row in enumerate(rows):
|
||||
cells = row.find_all(["th", "td"])
|
||||
cell_texts = [c.get_text(strip=True).replace("|", "\\|") for c in cells]
|
||||
md_rows.append("| " + " | ".join(cell_texts) + " |")
|
||||
if i == 0:
|
||||
md_rows.append("| " + " | ".join(["---"] * len(cell_texts)) + " |")
|
||||
|
||||
return "\n" + "\n".join(md_rows) + "\n"
|
||||
|
||||
|
||||
def _write_raw_file(
|
||||
output_dir: Path,
|
||||
index: int,
|
||||
url: str,
|
||||
content: str,
|
||||
metadata: PageMetadata | None = None,
|
||||
) -> Path:
|
||||
"""Write a single .raw.md file with YAML frontmatter."""
|
||||
parsed = urlparse(url)
|
||||
slug = _slugify(parsed.path.rstrip('/').split('/')[-1] or parsed.netloc)
|
||||
filename = f"{index:02d}-{slug}.raw.md"
|
||||
filepath = output_dir / filename
|
||||
|
||||
frontmatter_fields = {
|
||||
"source_url": url,
|
||||
"crawled_at": datetime.now().isoformat(),
|
||||
"crawler": "seo-aiohttp",
|
||||
"content_length": len(content),
|
||||
"raw": True,
|
||||
}
|
||||
|
||||
if metadata:
|
||||
frontmatter_fields.update({
|
||||
"status_code": metadata.status_code,
|
||||
"title": metadata.title or "",
|
||||
"word_count": metadata.word_count,
|
||||
"internal_links": metadata.internal_link_count,
|
||||
"external_links": metadata.external_link_count,
|
||||
"h1": metadata.h1_text or "",
|
||||
"schema_types": metadata.schema_types_found,
|
||||
"response_time_ms": round(metadata.response_time_ms, 1),
|
||||
})
|
||||
|
||||
# Build YAML frontmatter
|
||||
fm_lines = ["---"]
|
||||
for key, value in frontmatter_fields.items():
|
||||
if isinstance(value, bool):
|
||||
fm_lines.append(f"{key}: {'true' if value else 'false'}")
|
||||
elif isinstance(value, list):
|
||||
fm_lines.append(f"{key}: {json.dumps(value)}")
|
||||
elif isinstance(value, str) and (':' in value or '"' in value):
|
||||
fm_lines.append(f'{key}: "{value}"')
|
||||
else:
|
||||
fm_lines.append(f"{key}: {value}")
|
||||
fm_lines.append("---")
|
||||
|
||||
filepath.write_text("\n".join(fm_lines) + "\n\n" + content, encoding="utf-8")
|
||||
return filepath
|
||||
|
||||
|
||||
def discover_urls(root_url: str, max_depth: int = 2, same_domain: bool = True,
|
||||
max_urls: int = 100) -> list[str]:
|
||||
"""Discover URLs from a root page by following links.
|
||||
|
||||
Uses PageAnalyzer to extract internal links, then follows them
|
||||
breadth-first up to max_depth.
|
||||
"""
|
||||
analyzer = PageAnalyzer(timeout=15)
|
||||
parsed_root = urlparse(root_url)
|
||||
root_domain = parsed_root.netloc.lower()
|
||||
|
||||
visited = set()
|
||||
to_visit = [(root_url, 0)]
|
||||
discovered = []
|
||||
|
||||
while to_visit and len(discovered) < max_urls:
|
||||
url, depth = to_visit.pop(0)
|
||||
|
||||
if url in visited:
|
||||
continue
|
||||
visited.add(url)
|
||||
|
||||
try:
|
||||
metadata = analyzer.analyze_url(url)
|
||||
if metadata.status_code == 200:
|
||||
discovered.append(url)
|
||||
logger.info(f"[{len(discovered)}/{max_urls}] {url}")
|
||||
|
||||
if depth < max_depth:
|
||||
for link in metadata.internal_links:
|
||||
link_domain = urlparse(link.url).netloc.lower()
|
||||
if same_domain and link_domain != root_domain:
|
||||
continue
|
||||
if link.url not in visited:
|
||||
to_visit.append((link.url, depth + 1))
|
||||
except Exception as e:
|
||||
logger.warning(f"Skip {url}: {e}")
|
||||
|
||||
return discovered
|
||||
|
||||
|
||||
def discover_from_sitemap(sitemap_url: str, max_urls: int = 100) -> list[str]:
|
||||
"""Extract URLs from an XML sitemap."""
|
||||
urls = []
|
||||
try:
|
||||
resp = requests.get(sitemap_url, timeout=15,
|
||||
headers={"User-Agent": "ReferenceBot/1.0"})
|
||||
resp.raise_for_status()
|
||||
|
||||
root = ET.fromstring(resp.content)
|
||||
ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
|
||||
|
||||
# Check if it's a sitemap index
|
||||
sitemaps = root.findall("sm:sitemap/sm:loc", ns)
|
||||
if sitemaps:
|
||||
for sm in sitemaps[:5]:
|
||||
urls.extend(discover_from_sitemap(sm.text, max_urls - len(urls)))
|
||||
if len(urls) >= max_urls:
|
||||
break
|
||||
else:
|
||||
for url_elem in root.findall("sm:url/sm:loc", ns):
|
||||
if url_elem.text:
|
||||
urls.append(url_elem.text.strip())
|
||||
if len(urls) >= max_urls:
|
||||
break
|
||||
except Exception as e:
|
||||
logger.error(f"Sitemap error: {e}")
|
||||
|
||||
return urls[:max_urls]
|
||||
|
||||
|
||||
def crawl_urls(
|
||||
urls: list[str],
|
||||
output_dir: Path,
|
||||
include_metadata: bool = True,
|
||||
run_id: str | None = None,
|
||||
) -> dict:
|
||||
"""Crawl a list of URLs and write .raw.md files.
|
||||
|
||||
Returns crawl statistics dictionary.
|
||||
"""
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
analyzer = PageAnalyzer(timeout=20)
|
||||
|
||||
progress = CrawlProgress(
|
||||
run_id=run_id or datetime.now().strftime("%Y%m%d_%H%M%S"),
|
||||
total_urls=len(urls),
|
||||
)
|
||||
|
||||
results = []
|
||||
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
|
||||
TextColumn("({task.completed}/{task.total})"),
|
||||
console=console,
|
||||
) as pbar:
|
||||
task = pbar.add_task("Crawling...", total=len(urls))
|
||||
|
||||
for i, url in enumerate(urls):
|
||||
progress.current_url = url
|
||||
pbar.update(task, description=f"[cyan]{urlparse(url).path[:40]}...")
|
||||
|
||||
try:
|
||||
# Fetch and analyze
|
||||
metadata = analyzer.analyze_url(url) if include_metadata else None
|
||||
|
||||
# Get raw HTML for content extraction
|
||||
resp = requests.get(url, timeout=20,
|
||||
headers={"User-Agent": PageAnalyzer.DEFAULT_USER_AGENT})
|
||||
|
||||
if resp.status_code == 200:
|
||||
content = _extract_main_content(resp.text, url)
|
||||
filepath = _write_raw_file(output_dir, i, url, content, metadata)
|
||||
results.append({"url": url, "file": filepath.name, "status": "ok",
|
||||
"size": len(content)})
|
||||
progress.succeeded += 1
|
||||
logger.info(f"OK [{resp.status_code}] {url} → {filepath.name}")
|
||||
else:
|
||||
results.append({"url": url, "status": f"http_{resp.status_code}"})
|
||||
progress.failed += 1
|
||||
logger.warning(f"FAIL [{resp.status_code}] {url}")
|
||||
|
||||
except Exception as e:
|
||||
results.append({"url": url, "status": "error", "error": str(e)})
|
||||
progress.failed += 1
|
||||
logger.error(f"ERROR {url}: {e}")
|
||||
|
||||
progress.processed += 1
|
||||
pbar.update(task, advance=1)
|
||||
progress.save()
|
||||
|
||||
progress.status = "completed"
|
||||
progress.save()
|
||||
|
||||
# Generate combined bundle
|
||||
raw_files = sorted(output_dir.glob("*.raw.md"))
|
||||
if raw_files:
|
||||
bundle_name = output_dir.parent.name or "reference"
|
||||
bundle_path = output_dir / f"{bundle_name}-raw-complete.md"
|
||||
with open(bundle_path, "w") as bundle:
|
||||
bundle.write(f"# {bundle_name} — Raw Content Bundle\n\n")
|
||||
bundle.write(f"> Combined raw crawled content from {len(raw_files)} sources\n")
|
||||
bundle.write(f"> Generated: {datetime.now().strftime('%Y-%m-%d')}\n\n")
|
||||
for rf in raw_files:
|
||||
bundle.write("---\n\n")
|
||||
bundle.write(rf.read_text())
|
||||
bundle.write("\n\n")
|
||||
results.append({"file": bundle_path.name, "status": "bundle",
|
||||
"size": bundle_path.stat().st_size})
|
||||
|
||||
stats = {
|
||||
"run_id": progress.run_id,
|
||||
"total_urls": len(urls),
|
||||
"succeeded": progress.succeeded,
|
||||
"failed": progress.failed,
|
||||
"skipped": progress.skipped,
|
||||
"files_written": len(raw_files),
|
||||
"output_dir": str(output_dir),
|
||||
"results": results,
|
||||
}
|
||||
|
||||
# Write manifest
|
||||
manifest_path = output_dir / "crawl_manifest.json"
|
||||
manifest_path.write_text(json.dumps(stats, indent=2, default=str))
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
# ─── CLI ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
@click.group()
|
||||
def cli():
|
||||
"""SEO Crawler Adapter — reference-curator crawling backend."""
|
||||
pass
|
||||
|
||||
|
||||
@cli.command()
|
||||
@click.option("--urls", required=True, help="Comma-separated URLs or @file.txt")
|
||||
@click.option("--output", required=True, type=click.Path(), help="Output directory for .raw.md files")
|
||||
@click.option("--no-metadata", is_flag=True, help="Skip SEO metadata extraction")
|
||||
@click.option("--run-id", help="Custom run ID for progress tracking")
|
||||
def crawl(urls, output, no_metadata, run_id):
|
||||
"""Crawl a list of URLs and write .raw.md files."""
|
||||
if urls.startswith("@"):
|
||||
url_list = Path(urls[1:]).read_text().strip().splitlines()
|
||||
else:
|
||||
url_list = [u.strip() for u in urls.split(",") if u.strip()]
|
||||
|
||||
output_path = Path(output).expanduser()
|
||||
stats = crawl_urls(url_list, output_path, include_metadata=not no_metadata, run_id=run_id)
|
||||
|
||||
console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled")
|
||||
console.print(f"Output: {stats['output_dir']}")
|
||||
|
||||
|
||||
@cli.command("crawl-sitemap")
|
||||
@click.option("--sitemap", required=True, help="Sitemap URL")
|
||||
@click.option("--output", required=True, type=click.Path(), help="Output directory")
|
||||
@click.option("--max-pages", default=50, help="Max pages to crawl")
|
||||
@click.option("--run-id", help="Custom run ID")
|
||||
def crawl_sitemap(sitemap, output, max_pages, run_id):
|
||||
"""Crawl URLs discovered from an XML sitemap."""
|
||||
console.print(f"Parsing sitemap: {sitemap}")
|
||||
urls = discover_from_sitemap(sitemap, max_urls=max_pages)
|
||||
console.print(f"Found {len(urls)} URLs in sitemap")
|
||||
|
||||
if not urls:
|
||||
console.print("[yellow]No URLs found in sitemap[/yellow]")
|
||||
return
|
||||
|
||||
output_path = Path(output).expanduser()
|
||||
stats = crawl_urls(urls, output_path, run_id=run_id)
|
||||
console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled")
|
||||
|
||||
|
||||
@cli.command("discover")
|
||||
@click.option("--url", required=True, help="Root URL to discover from")
|
||||
@click.option("--output", required=True, type=click.Path(), help="Output directory")
|
||||
@click.option("--max-depth", default=2, help="Max link-following depth")
|
||||
@click.option("--max-pages", default=50, help="Max pages to discover and crawl")
|
||||
@click.option("--same-domain/--follow-external", default=True, help="Stay on same domain")
|
||||
@click.option("--run-id", help="Custom run ID")
|
||||
def discover(url, output, max_depth, max_pages, same_domain, run_id):
|
||||
"""Discover URLs from a root page, then crawl them all."""
|
||||
console.print(f"Discovering URLs from: {url} (depth={max_depth})")
|
||||
urls = discover_urls(url, max_depth=max_depth, same_domain=same_domain, max_urls=max_pages)
|
||||
console.print(f"Discovered {len(urls)} URLs")
|
||||
|
||||
if not urls:
|
||||
console.print("[yellow]No URLs discovered[/yellow]")
|
||||
return
|
||||
|
||||
output_path = Path(output).expanduser()
|
||||
stats = crawl_urls(urls, output_path, run_id=run_id)
|
||||
console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled")
|
||||
|
||||
|
||||
@cli.command("crawl-manifest")
|
||||
@click.option("--manifest", required=True, type=click.Path(exists=True), help="Manifest JSON file")
|
||||
@click.option("--output", required=True, type=click.Path(), help="Output directory")
|
||||
@click.option("--run-id", help="Custom run ID")
|
||||
def crawl_manifest(manifest, output, run_id):
|
||||
"""Crawl URLs from a reference-curator manifest file."""
|
||||
data = json.loads(Path(manifest).read_text())
|
||||
|
||||
urls = []
|
||||
if isinstance(data, dict) and "sources" in data:
|
||||
urls = [s["url"] for s in data["sources"] if s.get("url")]
|
||||
elif isinstance(data, list):
|
||||
urls = [item.get("url", item) if isinstance(item, dict) else item for item in data]
|
||||
|
||||
console.print(f"Loaded {len(urls)} URLs from manifest")
|
||||
|
||||
output_path = Path(output).expanduser()
|
||||
stats = crawl_urls(urls, output_path, run_id=run_id)
|
||||
console.print(f"\n[green]Done![/green] {stats['succeeded']}/{stats['total_urls']} pages crawled")
|
||||
|
||||
|
||||
@cli.command("status")
|
||||
@click.option("--run-id", help="Specific run ID to check")
|
||||
def status(run_id):
|
||||
"""Check crawl progress."""
|
||||
if run_id:
|
||||
filepath = PROGRESS_DIR / f"{run_id}.json"
|
||||
if filepath.exists():
|
||||
data = json.loads(filepath.read_text())
|
||||
console.print_json(json.dumps(data, indent=2))
|
||||
else:
|
||||
console.print(f"[yellow]No progress file for run {run_id}[/yellow]")
|
||||
else:
|
||||
files = sorted(PROGRESS_DIR.glob("*.json"), reverse=True)[:10]
|
||||
if not files:
|
||||
console.print("[yellow]No crawl history found[/yellow]")
|
||||
return
|
||||
|
||||
table = Table(title="Recent Crawls")
|
||||
table.add_column("Run ID")
|
||||
table.add_column("Status")
|
||||
table.add_column("Progress")
|
||||
table.add_column("Success/Total")
|
||||
|
||||
for f in files:
|
||||
data = json.loads(f.read_text())
|
||||
table.add_row(
|
||||
data.get("run_id", "?"),
|
||||
data.get("status", "?"),
|
||||
f"{data.get('progress_pct', 0)}%",
|
||||
f"{data.get('succeeded', 0)}/{data.get('total_urls', 0)}",
|
||||
)
|
||||
console.print(table)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
cli()
|
||||
@@ -5,6 +5,14 @@ Analyzes and distills raw crawled content into concise reference materials. Clau
|
||||
## Trigger Keywords
|
||||
"distill content", "summarize document", "extract key concepts", "process raw content", "create reference summary"
|
||||
|
||||
## Skip Condition
|
||||
|
||||
**This entire stage is skipped when `--no-distill` is active.**
|
||||
|
||||
When the pipeline orchestrator passes `--no-distill`, this stage returns immediately with status `skipped`. The pipeline advances directly from Stage 3 (content-repository) to Stage 6 (markdown-exporter), and raw crawled content becomes the export source instead of distilled content.
|
||||
|
||||
No documents are loaded, no distillation occurs, and no records are written to the `distilled_content` table.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Compress** — Reduce token count while preserving essential information
|
||||
|
||||
@@ -5,6 +5,12 @@ Pre-distillation quality gate using Gemini CLI as an independent evaluator. Asse
|
||||
## Trigger Keywords
|
||||
"review content", "quality check", "QA review", "evaluate sources", "check reference quality"
|
||||
|
||||
## Skip Condition
|
||||
|
||||
**This entire stage is skipped when `--no-distill` is active.**
|
||||
|
||||
When the pipeline uses `--no-distill`, quality review is bypassed because there is no distilled content to evaluate. Raw crawled content goes directly to the exporter.
|
||||
|
||||
## Primary Flow: Gemini Pre-Distillation Gate
|
||||
|
||||
```
|
||||
|
||||
@@ -10,6 +10,7 @@ Exports approved reference content as structured markdown files for project know
|
||||
| Type | Format | Use Case |
|
||||
|------|--------|----------|
|
||||
| `project` | Nested markdown | Claude Projects knowledge |
|
||||
| `raw` | Raw markdown | Archival, full-text reference, NotebookLM sources |
|
||||
| `finetuning` | JSONL | Model fine-tuning dataset |
|
||||
|
||||
## Workflow
|
||||
@@ -74,11 +75,57 @@ JSONL format:
|
||||
uv run python scripts/exporter.py log --name "April 2026 Export" --type project_files --docs 45
|
||||
```
|
||||
|
||||
### Raw Export (when `--save-raw` or `--no-distill`)
|
||||
|
||||
#### Raw alongside distilled (`--save-raw`)
|
||||
```bash
|
||||
uv run python scripts/exporter.py raw \
|
||||
--output ~/reference-library/exports/{topic-slug}/raw/ \
|
||||
--source-dir {crawl-raw-dir} \
|
||||
--bundle {topic-slug}-raw-complete.md
|
||||
```
|
||||
|
||||
Output structure:
|
||||
```
|
||||
exports/{topic-slug}/
|
||||
├── INDEX.md # Includes links to raw/ section
|
||||
├── 00-page-name.md # Distilled (normal)
|
||||
├── raw/
|
||||
│ ├── 00-page-name.raw.md
|
||||
│ ├── 01-page-name.raw.md
|
||||
│ └── {topic-slug}-raw-complete.md
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
#### Raw-only mode (`--no-distill`)
|
||||
```bash
|
||||
uv run python scripts/exporter.py raw \
|
||||
--output ~/reference-library/exports/{topic-slug}/ \
|
||||
--source-dir {crawl-raw-dir} \
|
||||
--bundle {topic-slug}-raw-complete.md \
|
||||
--primary
|
||||
```
|
||||
|
||||
Output structure (raw files are primary, no subdirectory wrapper):
|
||||
```
|
||||
exports/{topic-slug}/
|
||||
├── README.md
|
||||
├── 00-page-name.raw.md
|
||||
├── 01-page-name.raw.md
|
||||
├── {topic-slug}-raw-complete.md
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
The `--primary` flag tells the exporter that raw files are the main output. README is generated from raw file frontmatter (source URLs, crawl timestamps).
|
||||
|
||||
**Note:** When combined with `--export-format fine_tuning`, the JSONL exporter uses raw content directly as the assistant response with a simpler system prompt (no structured summaries available).
|
||||
|
||||
## Scripts
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `exporter.py project` | Export as nested markdown files |
|
||||
| `exporter.py raw` | Export raw crawled markdown files with optional bundle |
|
||||
| `exporter.py finetuning` | Export as JSONL training dataset |
|
||||
| `exporter.py index` | Generate INDEX.md table of contents |
|
||||
| `exporter.py crossrefs` | Add cross-reference links |
|
||||
@@ -90,4 +137,5 @@ uv run python scripts/exporter.py log --name "April 2026 Export" --type project_
|
||||
| From | To |
|
||||
|------|-----|
|
||||
| quality-reviewer (approved) | → |
|
||||
| web-crawler (raw files) | Raw markdown input (--save-raw / --no-distill) |
|
||||
| → | Project knowledge / Fine-tuning dataset |
|
||||
|
||||
@@ -9,17 +9,20 @@ Coordinates the full 6-skill reference curation workflow with QA loop handling.
|
||||
|
||||
```
|
||||
[Input] → discovery → crawler → repository → distiller ◄──┐
|
||||
│ │
|
||||
reviewer │
|
||||
│ │
|
||||
┌───────────────────────────────┼─────┤
|
||||
▼ ▼ ▼ │
|
||||
APPROVE REJECT REFACTOR ────┤
|
||||
│ │ │
|
||||
▼ ▼ DEEP_RESEARCH
|
||||
export archive │
|
||||
▼
|
||||
crawler ─┘
|
||||
│ (--save-raw) reviewer │
|
||||
│ writes raw/ │ │
|
||||
│ ┌───────┼─────┤
|
||||
│ ▼ ▼ │
|
||||
│ APPROVE REJECT REFACTOR ────┤
|
||||
│ │ │ │
|
||||
│ ▼ ▼ DEEP_RESEARCH
|
||||
│ export archive │
|
||||
│ │ ▼
|
||||
│ │ crawler ─┘
|
||||
│ │
|
||||
└── (--no-distill) ──► export (raw only)
|
||||
skip stages 4+5
|
||||
```
|
||||
|
||||
## Pipeline State Management
|
||||
@@ -28,7 +31,7 @@ Coordinates the full 6-skill reference curation workflow with QA loop handling.
|
||||
```bash
|
||||
uv run python scripts/pipeline.py init \
|
||||
--input "prompt engineering" --type topic \
|
||||
--options '{"max_sources": 10, "auto_approve": true}'
|
||||
--options '{"max_sources": 10, "auto_approve": true, "save_raw": false, "no_distill": false}'
|
||||
```
|
||||
|
||||
### Advance to Next Stage
|
||||
@@ -99,6 +102,7 @@ If mode == 'topic':
|
||||
```
|
||||
→ Claude uses Firecrawl MCP tools
|
||||
→ crawl_mgr.py store-result
|
||||
→ If --save-raw or --no-distill: write raw .raw.md files to output
|
||||
→ pipeline.py advance --stage storing
|
||||
```
|
||||
|
||||
@@ -108,7 +112,23 @@ If mode == 'topic':
|
||||
→ pipeline.py advance --stage evaluating
|
||||
```
|
||||
|
||||
### Flag Resolution (after Stage 3)
|
||||
```
|
||||
If --no-distill:
|
||||
→ set save_raw = true (implied)
|
||||
→ skip Stage 4 (content-distiller)
|
||||
→ skip Stage 5 (quality-reviewer)
|
||||
→ pipeline.py advance --stage exporting --skip-reason "no-distill"
|
||||
→ proceed directly to Stage 6 with raw files as input
|
||||
|
||||
If --save-raw (without --no-distill):
|
||||
→ raw files already written during Stage 2
|
||||
→ continue normal pipeline through stages 4-5
|
||||
→ Stage 6 exports both distilled AND raw content
|
||||
```
|
||||
|
||||
### Stage 4: Gemini Quality Gate (Pre-Distillation)
|
||||
**Skipped when `--no-distill` is active.**
|
||||
```
|
||||
→ reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve
|
||||
→ APPROVE: proceed to distillation
|
||||
@@ -118,6 +138,7 @@ If mode == 'topic':
|
||||
```
|
||||
|
||||
### Stage 5: Content Distiller (Approved Only)
|
||||
**Skipped when `--no-distill` is active.**
|
||||
```
|
||||
→ distiller.py load-pending
|
||||
→ Claude distills each approved document
|
||||
@@ -127,6 +148,21 @@ If mode == 'topic':
|
||||
|
||||
### Stage 6: Markdown Exporter
|
||||
```
|
||||
If --no-distill:
|
||||
→ exporter.py raw --primary (raw files are the main output)
|
||||
→ exporter.py index (generate README from raw file metadata)
|
||||
→ exporter.py verify
|
||||
→ pipeline.py complete
|
||||
|
||||
If --save-raw (with distillation):
|
||||
→ exporter.py project (normal distilled export)
|
||||
→ exporter.py raw (write raw/ subdirectory + bundle)
|
||||
→ exporter.py index (includes raw section)
|
||||
→ exporter.py crossrefs
|
||||
→ exporter.py verify
|
||||
→ pipeline.py complete
|
||||
|
||||
Default (no raw flags):
|
||||
→ exporter.py project
|
||||
→ exporter.py index
|
||||
→ exporter.py crossrefs
|
||||
@@ -139,8 +175,9 @@ If mode == 'topic':
|
||||
| Stage | Checkpoint | Resume Point |
|
||||
|-------|------------|--------------|
|
||||
| discovery | manifest.json created | → crawler |
|
||||
| crawl | crawl_result.json | → repository |
|
||||
| store | DB records | → distiller |
|
||||
| crawl | crawl_result.json + raw files | → repository |
|
||||
| store | DB records | → distiller (or skip) |
|
||||
| no-distill skip | stages 4-5 marked skipped | → exporter (raw) |
|
||||
| distill | distilled_content records | → reviewer |
|
||||
| review | review_logs records | → exporter or loop |
|
||||
| export | final export complete | Done |
|
||||
@@ -167,6 +204,8 @@ pipeline:
|
||||
max_pages: 50
|
||||
auto_approve: false
|
||||
approval_threshold: 0.85
|
||||
save_raw: false
|
||||
no_distill: false
|
||||
|
||||
qa_loop:
|
||||
max_refactor_iterations: 3
|
||||
|
||||
@@ -193,6 +193,15 @@ Run the complete curation workflow with a single command:
|
||||
|
||||
# Fine-tuning dataset output
|
||||
/reference-curator-pipeline "MCP servers" --export-format fine_tuning
|
||||
|
||||
# Save raw crawled content alongside distilled output
|
||||
/reference-curator-pipeline "Claude Code best practices" --save-raw
|
||||
|
||||
# Pure archival — skip distillation, keep raw markdown only
|
||||
/reference-curator-pipeline https://docs.anthropic.com --no-distill
|
||||
|
||||
# Full-depth archival
|
||||
/reference-curator-pipeline https://legacy-docs.example.com --depth full --no-distill
|
||||
```
|
||||
|
||||
**Pipeline Options:**
|
||||
@@ -202,6 +211,8 @@ Run the complete curation workflow with a single command:
|
||||
- `--threshold 0.85` - Approval threshold
|
||||
- `--max-iterations 3` - Max QA loop iterations per document
|
||||
- `--export-format project_files` - Output format (project_files, fine_tuning, jsonl)
|
||||
- `--save-raw` - Also save intact raw crawled markdown in a `raw/` subdirectory
|
||||
- `--no-distill` - Skip distillation and QA (pure archival). Implies `--save-raw`
|
||||
|
||||
---
|
||||
|
||||
@@ -313,6 +324,29 @@ mysql -h $MYSQL_HOST -u $MYSQL_USER -p"$MYSQL_PASSWORD" reference_library -e "
|
||||
└── chain-of-thought.md
|
||||
```
|
||||
|
||||
**Raw Archival (`--no-distill`):**
|
||||
```
|
||||
~/reference-library/exports/
|
||||
└── topic-slug/
|
||||
├── README.md
|
||||
├── 00-page-name.raw.md # Intact crawled content
|
||||
├── 01-page-name.raw.md
|
||||
├── topic-slug-raw-complete.md # Combined raw bundle
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
**Distilled + Raw (`--save-raw`):**
|
||||
```
|
||||
~/reference-library/exports/
|
||||
└── topic-slug/
|
||||
├── INDEX.md
|
||||
├── 00-page-name.md # Distilled
|
||||
├── raw/
|
||||
│ ├── 00-page-name.raw.md # Intact crawled
|
||||
│ └── topic-slug-raw-complete.md
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
**For Fine-tuning (JSONL):**
|
||||
```json
|
||||
{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
|
||||
|
||||
@@ -15,6 +15,12 @@ A modular skill suite for curating reference documentation with Claude Code.
|
||||
|
||||
# With options
|
||||
/reference-curator-pipeline "MCP servers" --max-sources 10 --export-format fine_tuning
|
||||
|
||||
# Save raw crawled content alongside distilled output
|
||||
/reference-curator-pipeline "Claude Code best practices" --save-raw
|
||||
|
||||
# Pure archival — skip distillation, raw markdown only
|
||||
/reference-curator-pipeline https://docs.anthropic.com --no-distill
|
||||
```
|
||||
|
||||
### Run as Background Agent
|
||||
@@ -156,6 +162,89 @@ Use these when you need granular control over each stage.
|
||||
|
||||
---
|
||||
|
||||
## Raw Mode
|
||||
|
||||
### What is Raw Mode?
|
||||
|
||||
By default, the pipeline distills crawled content down to 25-35% of the original, extracting key concepts, code snippets, and structured summaries. **Raw mode** preserves the full, unmodified crawled markdown.
|
||||
|
||||
### `--save-raw` Flag
|
||||
|
||||
Saves intact crawled markdown **alongside** the distilled content in a `raw/` subdirectory.
|
||||
|
||||
```bash
|
||||
/reference-curator-pipeline "Anthropic Claude API" --save-raw
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
exports/claude-api/
|
||||
├── INDEX.md
|
||||
├── 00-prompt-caching.md # Distilled (25-35% of original)
|
||||
├── 01-streaming.md
|
||||
├── raw/
|
||||
│ ├── 00-prompt-caching.raw.md # Full crawled content
|
||||
│ ├── 01-streaming.raw.md
|
||||
│ └── claude-api-raw-complete.md # Combined raw bundle
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
- Need both concise reference AND full-text source material
|
||||
- Want to re-distill later with different parameters
|
||||
- Auditing what content was captured vs. what was extracted
|
||||
|
||||
### `--no-distill` Flag
|
||||
|
||||
Skips stages 4 (content-distiller) and 5 (quality-reviewer) entirely. The pipeline goes directly from crawl/store to export. Raw files become the **primary** output. Implies `--save-raw` automatically.
|
||||
|
||||
```bash
|
||||
/reference-curator-pipeline https://docs.anthropic.com --no-distill
|
||||
```
|
||||
|
||||
**Output:**
|
||||
```
|
||||
exports/anthropic-docs/
|
||||
├── README.md
|
||||
├── 00-getting-started.raw.md
|
||||
├── 01-prompt-engineering.raw.md
|
||||
├── anthropic-docs-raw-complete.md
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
- Site archival / preservation
|
||||
- Migration reference (need exact original content)
|
||||
- Training data collection (distillation would remove useful context)
|
||||
- NotebookLM / LLM context sources (full fidelity preferred)
|
||||
- Quick capture when you will distill manually later
|
||||
|
||||
### Combining with Other Flags
|
||||
|
||||
```bash
|
||||
# Deep archival crawl
|
||||
/reference-curator-pipeline https://legacy.example.com --depth full --no-distill
|
||||
|
||||
# Standard crawl with raw backup and auto-approval
|
||||
/reference-curator-pipeline "MCP servers" --save-raw --auto-approve
|
||||
|
||||
# Fine-tuning dataset from raw content
|
||||
/reference-curator-pipeline "prompt engineering" --no-distill --export-format fine_tuning
|
||||
```
|
||||
|
||||
### Pipeline Stage Comparison
|
||||
|
||||
| Stage | Default | `--save-raw` | `--no-distill` |
|
||||
|-------|---------|-------------|----------------|
|
||||
| 1. Discovery | Runs | Runs | Runs |
|
||||
| 2. Web Crawler | Runs | Runs + writes raw/ | Runs + writes raw |
|
||||
| 3. Repository | Runs | Runs | Runs |
|
||||
| 4. Distiller | Runs | Runs | **Skipped** |
|
||||
| 5. Reviewer | Runs | Runs | **Skipped** |
|
||||
| 6. Exporter | Distilled | Distilled + raw | **Raw only** |
|
||||
|
||||
---
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Workflow 1: Quick Reference on a Topic
|
||||
@@ -197,6 +286,8 @@ Claude: /markdown-exporter --format fine_tuning
|
||||
| Type | Location |
|
||||
|------|----------|
|
||||
| Raw content | `~/Documents/05_AI Agent/10_Reference Library/raw/` |
|
||||
| Raw output (--save-raw) | `{output}/{topic-slug}/raw/` |
|
||||
| Raw output (--no-distill) | `{output}/{topic-slug}/` (primary) |
|
||||
| Exports | `~/Documents/05_AI Agent/10_Reference Library/exports/` |
|
||||
| Config | `~/.config/reference-curator/` |
|
||||
|
||||
@@ -288,6 +379,34 @@ Claude: Starting reference-curator pipeline...
|
||||
Pipeline complete! Reference library ready.
|
||||
```
|
||||
|
||||
### Example: Archival Mode (`--no-distill`)
|
||||
|
||||
```
|
||||
You: Archive the GA4 developer docs without distilling — just raw content
|
||||
|
||||
Claude: Starting reference-curator pipeline (--no-distill mode)...
|
||||
|
||||
[1/6] Reference Discovery
|
||||
- Skipped (URL input mode)
|
||||
|
||||
[2/6] Web Crawling
|
||||
- Crawling developers.google.com/analytics with Firecrawl...
|
||||
- Saved 28 raw markdown files (.raw.md)
|
||||
|
||||
[3/6] Content Repository
|
||||
- Stored 28 documents in MySQL
|
||||
|
||||
[4/6] Content Distillation — SKIPPED (--no-distill)
|
||||
[5/6] Quality Review — SKIPPED (--no-distill)
|
||||
|
||||
[6/6] Markdown Export (raw mode)
|
||||
- Generated 28 .raw.md files + combined raw bundle
|
||||
- Output: ~/Documents/.../google-analytics-4/
|
||||
|
||||
Pipeline complete! 28 raw pages archived.
|
||||
Files ready for NotebookLM upload or direct LLM context use.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Need Help?
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
description: Orchestrates full reference curation pipeline with configurable depth. Runs discovery > crawl > store > distill > review > export with QA loop handling.
|
||||
argument-hint: <topic|urls|manifest> [--depth light|standard|deep|full] [--output ~/Documents/reference-library/] [--max-sources 100] [--max-pages 50] [--auto-approve] [--threshold 0.85] [--max-iterations 3] [--export-format project_files]
|
||||
argument-hint: <topic|urls|manifest> [--depth light|standard|deep|full] [--output ~/Documents/reference-library/] [--max-sources 100] [--max-pages 50] [--auto-approve] [--threshold 0.85] [--max-iterations 3] [--export-format project_files] [--save-raw] [--no-distill]
|
||||
allowed-tools: WebSearch, WebFetch, Read, Write, Bash, Grep, Glob, Task
|
||||
---
|
||||
|
||||
@@ -29,6 +29,8 @@ Full-stack orchestration of the 6-skill reference curation workflow.
|
||||
- `--export-format`: Output format: `project_files`, `fine_tuning`, `jsonl` (default: project_files)
|
||||
- `--include-subdomains`: Include subdomains in site mapping (default: false)
|
||||
- `--follow-external`: Follow external links found in content (default: false)
|
||||
- `--save-raw`: Save intact crawled markdown alongside distilled content in a `raw/` subdirectory. Each file uses `.raw.md` extension. A combined raw bundle `{topic-slug}-raw-complete.md` is also generated.
|
||||
- `--no-distill`: Skip stages 4-5 (content-distiller and quality-reviewer) entirely. Pure archival mode. Implies `--save-raw`. Raw crawled files become the primary output — no distilled subdirectory is created.
|
||||
|
||||
## Output Directory
|
||||
|
||||
@@ -38,13 +40,13 @@ The `--output` argument sets the base path for all pipeline output. If omitted,
|
||||
|
||||
### Directory Structure
|
||||
|
||||
**Default (distilled output):**
|
||||
```
|
||||
{output}/
|
||||
├── {topic-slug}/ # One folder per pipeline run
|
||||
├── {topic-slug}/
|
||||
│ ├── README.md # Index with table of contents
|
||||
│ ├── 00-page-name.md # Individual page files
|
||||
│ ├── 00-page-name.md # Distilled individual page files
|
||||
│ ├── 01-page-name.md
|
||||
│ ├── ...
|
||||
│ ├── {topic-slug}-complete.md # Combined bundle (all pages)
|
||||
│ └── manifest.json # Crawl metadata
|
||||
├── pipeline_state/ # Resume state (auto-managed)
|
||||
@@ -52,6 +54,31 @@ The `--output` argument sets the base path for all pipeline output. If omitted,
|
||||
└── exports/ # Fine-tuning / JSONL exports
|
||||
```
|
||||
|
||||
**With `--save-raw` (distilled + raw):**
|
||||
```
|
||||
{output}/
|
||||
├── {topic-slug}/
|
||||
│ ├── README.md
|
||||
│ ├── 00-page-name.md # Distilled
|
||||
│ ├── {topic-slug}-complete.md # Combined distilled
|
||||
│ ├── raw/ # Intact crawled markdown
|
||||
│ │ ├── 00-page-name.raw.md
|
||||
│ │ ├── 01-page-name.raw.md
|
||||
│ │ └── {topic-slug}-raw-complete.md # Combined raw bundle
|
||||
│ └── manifest.json
|
||||
```
|
||||
|
||||
**With `--no-distill` (raw only, archival):**
|
||||
```
|
||||
{output}/
|
||||
├── {topic-slug}/
|
||||
│ ├── README.md
|
||||
│ ├── 00-page-name.raw.md # Raw crawled (primary output)
|
||||
│ ├── 01-page-name.raw.md
|
||||
│ ├── {topic-slug}-raw-complete.md # Combined raw bundle
|
||||
│ └── manifest.json
|
||||
```
|
||||
|
||||
### Resolution Rules
|
||||
|
||||
1. If `--output` is provided, use that path exactly
|
||||
@@ -158,15 +185,17 @@ full ████████████████ Speed: slow
|
||||
|
||||
```
|
||||
1. reference-discovery (topic mode only)
|
||||
2. web-crawler <- depth controls this stage
|
||||
2. web-crawler ← depth controls this stage
|
||||
← --save-raw: writes raw/ alongside crawl
|
||||
3. content-repository
|
||||
4. content-distiller <--------+
|
||||
5. quality-reviewer |
|
||||
4. content-distiller <--------+ ← SKIPPED when --no-distill
|
||||
5. quality-reviewer | ← SKIPPED when --no-distill
|
||||
+-- APPROVE -> export |
|
||||
+-- REFACTOR -----------------+
|
||||
+-- DEEP_RESEARCH -> crawler -+
|
||||
+-- REJECT -> archive
|
||||
6. markdown-exporter
|
||||
6. markdown-exporter ← --no-distill: exports raw files only
|
||||
← --save-raw: also exports raw/ bundle
|
||||
```
|
||||
|
||||
## Crawl Execution by Depth
|
||||
@@ -234,6 +263,12 @@ After max iterations, document marked as `needs_manual_review`.
|
||||
|
||||
# Resume from existing manifest
|
||||
/reference-curator ./manifest.json --auto-approve
|
||||
|
||||
# Save raw crawled content alongside distilled output
|
||||
/reference-curator https://docs.anthropic.com --depth standard --save-raw
|
||||
|
||||
# Pure archival — no distillation, just raw markdown files
|
||||
/reference-curator https://docs.anthropic.com --depth full --no-distill
|
||||
```
|
||||
|
||||
## State Management
|
||||
|
||||
@@ -6,13 +6,22 @@
|
||||
# REFERENCE_LIBRARY_PATH - Path to reference library storage (optional)
|
||||
|
||||
# Default crawler backend
|
||||
# Options: nodejs, aiohttp, scrapy, firecrawl
|
||||
# Set to "firecrawl" if local crawlers are not available
|
||||
default_crawler: ${DEFAULT_CRAWLER:-firecrawl}
|
||||
# Options: seo-aiohttp, firecrawl, nodejs, aiohttp, scrapy
|
||||
# seo-aiohttp: Uses OurSEO BaseAsyncClient + PageAnalyzer (recommended for docs)
|
||||
# firecrawl: MCP-based, best for SPAs/JS-rendered sites
|
||||
default_crawler: ${DEFAULT_CRAWLER:-seo-aiohttp}
|
||||
|
||||
# Intelligent routing rules
|
||||
# Claude will select the appropriate crawler based on these criteria
|
||||
routing:
|
||||
seo-aiohttp:
|
||||
conditions:
|
||||
- is_documentation_site == true
|
||||
- has_sitemap == true
|
||||
- needs_metadata == true
|
||||
- max_pages <= 200
|
||||
description: "OurSEO async crawler — rate-limited, resumable, full metadata extraction. Best for documentation sites."
|
||||
|
||||
nodejs:
|
||||
conditions:
|
||||
- max_pages <= 50
|
||||
@@ -42,8 +51,22 @@ routing:
|
||||
description: "JS rendering - best for SPAs and dynamic content (MCP-based, always available)"
|
||||
|
||||
# Crawler locations (configurable via environment)
|
||||
# If CRAWLER_PROJECT_PATH is not set, only Firecrawl MCP will be available
|
||||
crawlers:
|
||||
seo-aiohttp:
|
||||
enabled: true # Always available (uses OurSEO scripts from 12-seo-technical-audit)
|
||||
type: local
|
||||
path: ${SKILL_PATH:-../../../90-reference-curator}/02-web-crawler-orchestrator/code/scripts/
|
||||
command: python seo_crawler_adapter.py
|
||||
capabilities:
|
||||
- rate_limiting
|
||||
- progress_tracking
|
||||
- resume_support
|
||||
- metadata_extraction
|
||||
- sitemap_discovery
|
||||
- link_discovery
|
||||
- raw_file_output
|
||||
install: "pip install requests beautifulsoup4 click rich tenacity"
|
||||
|
||||
nodejs:
|
||||
enabled: ${NODEJS_CRAWLER_ENABLED:-false}
|
||||
path: ${CRAWLER_PROJECT_PATH}/util/js-crawler/
|
||||
|
||||
Reference in New Issue
Block a user