ourdigital/our-claude-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration

Browse Source 
- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory
  alongside distilled content, with YAML frontmatter for traceability
- Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer),
  pure archival mode for raw documentation capture
- Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend:
  token-bucket rate limiting, semaphore concurrency, tenacity retries,
  full page metadata extraction (headings, links, schema, OG)
- New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests
  with progress tracking and resume support
- Update crawler selection: docs sites now default to seo-aiohttp
- Update crawl_config.yaml with seo-aiohttp routing rules
- Update pipeline orchestrator with flag resolution and skip paths
- Update README.md and USER-GUIDE.md with raw mode documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Yim
2026-04-12 23:26:50 +09:00
parent f215c11c32
commit b1c2dca080
11 changed files with 977 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

116
custom-skills/90-reference-curator/02-web-crawler-orchestrator/code/CLAUDE.md
View File

@@ -1,6 +1,6 @@
# Web Crawler Orchestrator
Orchestrates web crawling with intelligent backend selection. Claude performs actual crawling via Firecrawl MCP tools. This skill manages crawl results, selects crawlers, and tracks crawl metadata.
Orchestrates web crawling with intelligent backend selection. Primary backend is the OurSEO async crawler (rate-limited, resumable, metadata-rich). Falls back to Firecrawl MCP for JS-rendered sites or WebFetch for simple pages.
## Trigger Keywords
"crawl URLs", "fetch documents", "scrape pages", "download references"
@@ -14,9 +14,10 @@ uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.
| Crawler | Best For | Auto-Selected When |
|---------|----------|-------------------|
| **Firecrawl MCP** (default) | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered |
| **SEO aiohttp** (default) | Documentation sites, sitemaps | docs/developer/api domains, has sitemap |
| **Firecrawl MCP** | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered |
| **WebFetch** | Quick single pages | Fallback, simple pages |
| **Node.js** | Small docs sites | ≤50 pages, static content |
| **Python aiohttp** | Technical docs | ≤200 pages, needs SEO data |
| **Scrapy** | Enterprise crawls | >200 pages, multi-domain |
## Workflow
@@ -24,20 +25,69 @@ uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.
### Step 1: Analyze Target Site
Run `select-crawler` to determine site characteristics and get a recommendation.
### Step 2: Execute Crawl
Use Firecrawl MCP tools directly:
### Step 2a: Execute Crawl — SEO aiohttp (Recommended)
The SEO crawler adapter provides rate limiting, progress tracking, resume support, and full metadata extraction.
```bash
# Crawl specific URLs
uv run python scripts/seo_crawler_adapter.py crawl \
--urls "https://docs.example.com/page1,https://docs.example.com/page2" \
--output ~/Documents/reference-library/topic-slug/raw/
# Crawl from XML sitemap (auto-discovers URLs)
uv run python scripts/seo_crawler_adapter.py crawl-sitemap \
--sitemap https://docs.example.com/sitemap.xml \
--output ~/Documents/reference-library/topic-slug/raw/ \
--max-pages 50
# Discover URLs from root page (follows internal links)
uv run python scripts/seo_crawler_adapter.py discover \
--url https://docs.example.com \
--output ~/Documents/reference-library/topic-slug/raw/ \
--max-depth 2 --max-pages 50 --same-domain
# Crawl from reference-curator manifest
uv run python scripts/seo_crawler_adapter.py crawl-manifest \
--manifest ./manifest.json \
--output ~/Documents/reference-library/topic-slug/raw/
# Check crawl progress
uv run python scripts/seo_crawler_adapter.py status
```
**SEO crawler capabilities:**
- Token-bucket rate limiting (via `BaseAsyncClient`)
- Semaphore-controlled concurrency (default: 5 concurrent)
- Tenacity retry with exponential backoff
- Full `PageMetadata` extraction: title, headings, links, schema, OG
- Progress tracking with ETA and resume files
- Sitemap XML parsing (including sitemap indexes)
- Internal link discovery (breadth-first, depth-limited)
- Direct `.raw.md` output with enriched YAML frontmatter
### Step 2b: Execute Crawl — Firecrawl MCP (for JS-rendered sites)
Use Firecrawl MCP tools directly when JS rendering is required:
```
firecrawl_map(url, limit=100) # Discover URLs
firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
firecrawl_crawl(url, max_depth=2, limit=50)
```
### Step 2c: Execute Crawl — WebFetch (fallback)
For quick single-page fetches when no crawl infrastructure is needed:
```
WebFetch(url, prompt="Return the COMPLETE page content as clean markdown.")
```
### Step 3: Store Crawl Results
```bash
# Store crawled files and create result manifest
uv run python scripts/crawl_mgr.py store-result \
--raw-dir ~/Documents/reference-library/raw/ \
--crawler firecrawl \
--crawler seo-aiohttp \
--source-id 1 \
--output crawl_result.json
@@ -45,27 +95,66 @@ uv run python scripts/crawl_mgr.py store-result \
uv run python scripts/crawl_mgr.py list-crawls --status completed
```
### Step 4: Write Raw Files (when `--save-raw` or `--no-distill`)
When the pipeline is invoked with `--save-raw` (or `--no-distill`, which implies it), write each crawled page's markdown verbatim to the raw output directory.
**Note:** The SEO crawler adapter (`seo_crawler_adapter.py`) writes `.raw.md` files directly with enriched YAML frontmatter — this step is automatic when using seo-aiohttp.
**File naming:** `{NN}-{page-slug}.raw.md` — zero-padded index matching crawl order.
**Output path:**
- `--save-raw`: `{output}/{topic-slug}/raw/`
- `--no-distill`: `{output}/{topic-slug}/` (raw files are primary)
**YAML frontmatter (SEO crawler enriched):**
```yaml
---
source_url: {original_url}
crawled_at: {ISO 8601 timestamp}
crawler: seo-aiohttp
content_length: {character count}
raw: true
status_code: {HTTP status}
title: {page title}
word_count: {word count}
internal_links: {count}
external_links: {count}
h1: {h1 text}
schema_types: [{types found}]
response_time_ms: {ms}
---
```
**After all pages are written**, a combined raw bundle (`{topic-slug}-raw-complete.md`) is auto-generated by concatenating all `.raw.md` files with `---` separators.
## Rate Limiting
All crawlers respect these limits:
- 20 requests/minute
- 3 concurrent requests
- Exponential backoff on 429/5xx
- 20 requests/minute (SEO crawler: configurable via `BaseAsyncClient`)
- 3-5 concurrent requests (SEO crawler: semaphore-controlled)
- Exponential backoff on 429/5xx (SEO crawler: tenacity)
## Error Handling
| Error | Action |
|-------|--------|
| Timeout | Retry once with 2x timeout |
| Rate limit (429) | Exponential backoff, max 3 retries |
| Timeout | Retry with exponential backoff (max 3 attempts) |
| Rate limit (429) | Token-bucket + exponential backoff |
| Not found (404) | Log and skip |
| Access denied (403) | Log, mark as `failed` |
| JS rendering needed | Switch to Firecrawl |
| JS rendering needed | Switch to Firecrawl MCP |
| Connection error | Retry with backoff, then skip |
## Scripts
| Command | Purpose |
|---------|---------|
| `seo_crawler_adapter.py crawl` | Crawl URLs with SEO metadata extraction |
| `seo_crawler_adapter.py crawl-sitemap` | Discover and crawl from XML sitemap |
| `seo_crawler_adapter.py discover` | BFS link discovery from root URL |
| `seo_crawler_adapter.py crawl-manifest` | Crawl from reference-curator manifest |
| `seo_crawler_adapter.py status` | Check crawl progress |
| `crawl_mgr.py select-crawler` | Recommend optimal crawler for a URL |
| `crawl_mgr.py store-result` | Store crawl results and create manifest |
| `crawl_mgr.py list-crawls` | List recent crawl records |
@@ -75,6 +164,7 @@ All crawlers respect these limits:
| From | To |
|------|-----|
| reference-discovery | URL manifest input |
| Firecrawl MCP | Raw crawled files |
| SEO aiohttp / Firecrawl MCP | Raw crawled files |
| → | content-repository (crawl manifest + raw files) |
| → | raw/ directory (when --save-raw or --no-distill) |
| quality-reviewer (deep_research) | Additional crawl requests |