Files

Andrew Yim f215c11c32 feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 18:22:28 +09:00

2.5 KiB

Raw Blame History

Web Crawler Orchestrator

Orchestrates web crawling with intelligent backend selection. Claude performs actual crawling via Firecrawl MCP tools. This skill manages crawl results, selects crawlers, and tracks crawl metadata.

Trigger Keywords

"crawl URLs", "fetch documents", "scrape pages", "download references"

Intelligent Crawler Selection

# Get crawler recommendation for a URL
uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.com"

Crawler	Best For	Auto-Selected When
Firecrawl MCP (default)	Dynamic sites, SPAs	React/Vue/Angular, JS-rendered
Node.js	Small docs sites	≤50 pages, static content
Python aiohttp	Technical docs	≤200 pages, needs SEO data
Scrapy	Enterprise crawls	>200 pages, multi-domain

Workflow

Step 1: Analyze Target Site

Run select-crawler to determine site characteristics and get a recommendation.

Step 2: Execute Crawl

Use Firecrawl MCP tools directly:

firecrawl_map(url, limit=100)           # Discover URLs
firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
firecrawl_crawl(url, max_depth=2, limit=50)

Step 3: Store Crawl Results

# Store crawled files and create result manifest
uv run python scripts/crawl_mgr.py store-result \
  --raw-dir ~/Documents/reference-library/raw/ \
  --crawler firecrawl \
  --source-id 1 \
  --output crawl_result.json

# List recent crawls
uv run python scripts/crawl_mgr.py list-crawls --status completed

Rate Limiting

All crawlers respect these limits:

20 requests/minute
3 concurrent requests
Exponential backoff on 429/5xx

Error Handling

Error	Action
Timeout	Retry once with 2x timeout
Rate limit (429)	Exponential backoff, max 3 retries
Not found (404)	Log and skip
Access denied (403)	Log, mark as `failed`
JS rendering needed	Switch to Firecrawl

Scripts

Command	Purpose
`crawl_mgr.py select-crawler`	Recommend optimal crawler for a URL
`crawl_mgr.py store-result`	Store crawl results and create manifest
`crawl_mgr.py list-crawls`	List recent crawl records

Integration

From	To
reference-discovery	URL manifest input
Firecrawl MCP	Raw crawled files
→	content-repository (crawl manifest + raw files)
quality-reviewer (deep_research)	Additional crawl requests

2.5 KiB Raw Blame History