feat(reference-curator): implement Python scripts + Gemini quality gate
Build the refcurator shared Python package and 7 CLI scripts that were previously specification-only. Add Gemini CLI as an independent pre-distillation quality evaluator, replacing the circular Claude-self-review pattern. Key changes: - shared/lib/src/refcurator/: 7-module package (config, db, models, utils, manifest, gemini) with PyMySQL + JSON file dual backend - 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer, exporter, pipeline — each with subcommands for data management - Gemini quality gate: evaluates raw content BEFORE distillation using 5 criteria (relevance, authority, completeness, freshness, distill_value) - Pipeline reordered: discovery → crawl → store → evaluate → distill → export - Bug fixes from Codex adversarial review: - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries - Exporter uses MAX(review_id) to prevent shipping stale approvals - Distiller updates existing rows on refactor instead of forking - Updated all 7 CLAUDE.md directives with real script references - install.sh updated with refcurator package install step 51/51 E2E tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,115 +1,48 @@
|
||||
# Web Crawler Orchestrator
|
||||
|
||||
Orchestrates web crawling with intelligent backend selection. Automatically chooses the best crawler based on site characteristics.
|
||||
Orchestrates web crawling with intelligent backend selection. Claude performs actual crawling via Firecrawl MCP tools. This skill manages crawl results, selects crawlers, and tracks crawl metadata.
|
||||
|
||||
## Trigger Keywords
|
||||
"crawl URLs", "fetch documents", "scrape pages", "download references"
|
||||
|
||||
## Intelligent Crawler Selection
|
||||
|
||||
Claude automatically selects the optimal crawler based on the request:
|
||||
```bash
|
||||
# Get crawler recommendation for a URL
|
||||
uv run python scripts/crawl_mgr.py select-crawler --url "https://docs.anthropic.com"
|
||||
```
|
||||
|
||||
| Crawler | Best For | Auto-Selected When |
|
||||
|---------|----------|-------------------|
|
||||
| **Node.js** (default) | Small docs sites | ≤50 pages, static content |
|
||||
| **Firecrawl MCP** (default) | Dynamic sites, SPAs | React/Vue/Angular, JS-rendered |
|
||||
| **Node.js** | Small docs sites | ≤50 pages, static content |
|
||||
| **Python aiohttp** | Technical docs | ≤200 pages, needs SEO data |
|
||||
| **Scrapy** | Enterprise crawls | >200 pages, multi-domain |
|
||||
| **Firecrawl MCP** | Dynamic sites | SPAs, JS-rendered content |
|
||||
|
||||
### Decision Flow
|
||||
|
||||
```
|
||||
[Crawl Request]
|
||||
│
|
||||
├─ Is it SPA/React/Vue/Angular? → Firecrawl MCP
|
||||
│
|
||||
├─ >200 pages or multi-domain? → Scrapy
|
||||
│
|
||||
├─ Needs SEO extraction? → Python aiohttp
|
||||
│
|
||||
└─ Default (small site) → Node.js
|
||||
```
|
||||
|
||||
## Crawler Backends
|
||||
|
||||
### Node.js (Default)
|
||||
Fast, lightweight crawler for small documentation sites.
|
||||
```bash
|
||||
cd ~/Project/our-seo-agent/util/js-crawler
|
||||
node src/crawler.js <URL> --max-pages 50
|
||||
```
|
||||
|
||||
### Python aiohttp
|
||||
Async crawler with full SEO extraction.
|
||||
```bash
|
||||
cd ~/Project/our-seo-agent
|
||||
python -m seo_agent.crawler --url <URL> --max-pages 100
|
||||
```
|
||||
|
||||
### Scrapy
|
||||
Enterprise-grade crawler with pipelines.
|
||||
```bash
|
||||
cd ~/Project/our-seo-agent
|
||||
scrapy crawl seo_spider -a start_url=<URL> -a max_pages=500
|
||||
```
|
||||
|
||||
### Firecrawl MCP
|
||||
Use MCP tools for JavaScript-heavy sites:
|
||||
```
|
||||
firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
|
||||
firecrawl_crawl(url, max_depth=2, limit=50)
|
||||
firecrawl_map(url, limit=100) # Discover URLs first
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
### Step 1: Analyze Target Site
|
||||
Determine site characteristics:
|
||||
- Is it a SPA? (React, Vue, Angular, Next.js)
|
||||
- How many pages expected?
|
||||
- Does it need JavaScript rendering?
|
||||
- Is SEO data extraction needed?
|
||||
Run `select-crawler` to determine site characteristics and get a recommendation.
|
||||
|
||||
### Step 2: Select Crawler
|
||||
Based on analysis, select the appropriate backend.
|
||||
### Step 2: Execute Crawl
|
||||
Use Firecrawl MCP tools directly:
|
||||
```
|
||||
firecrawl_map(url, limit=100) # Discover URLs
|
||||
firecrawl_scrape(url, formats=["markdown"], only_main_content=true)
|
||||
firecrawl_crawl(url, max_depth=2, limit=50)
|
||||
```
|
||||
|
||||
### Step 3: Load URL Manifest
|
||||
### Step 3: Store Crawl Results
|
||||
```bash
|
||||
# From reference-discovery output
|
||||
cat manifest.json | jq '.urls[].url'
|
||||
```
|
||||
# Store crawled files and create result manifest
|
||||
uv run python scripts/crawl_mgr.py store-result \
|
||||
--raw-dir ~/Documents/reference-library/raw/ \
|
||||
--crawler firecrawl \
|
||||
--source-id 1 \
|
||||
--output crawl_result.json
|
||||
|
||||
### Step 4: Execute Crawl
|
||||
|
||||
**For Node.js:**
|
||||
```bash
|
||||
cd ~/Project/our-seo-agent/util/js-crawler
|
||||
for url in $(cat urls.txt); do
|
||||
node src/crawler.js "$url" --max-pages 50
|
||||
sleep 2
|
||||
done
|
||||
```
|
||||
|
||||
**For Firecrawl MCP (Claude Desktop/Code):**
|
||||
Use the firecrawl MCP tools directly in conversation.
|
||||
|
||||
### Step 5: Save Raw Content
|
||||
```
|
||||
~/reference-library/raw/
|
||||
└── 2025/01/
|
||||
├── a1b2c3d4.md
|
||||
└── b2c3d4e5.md
|
||||
```
|
||||
|
||||
### Step 6: Generate Crawl Manifest
|
||||
```json
|
||||
{
|
||||
"crawl_date": "2025-01-28T12:00:00",
|
||||
"crawler_used": "nodejs",
|
||||
"total_crawled": 45,
|
||||
"total_failed": 5,
|
||||
"documents": [...]
|
||||
}
|
||||
# List recent crawls
|
||||
uv run python scripts/crawl_mgr.py list-crawls --status completed
|
||||
```
|
||||
|
||||
## Rate Limiting
|
||||
@@ -129,31 +62,19 @@ All crawlers respect these limits:
|
||||
| Access denied (403) | Log, mark as `failed` |
|
||||
| JS rendering needed | Switch to Firecrawl |
|
||||
|
||||
## Site Type Detection
|
||||
|
||||
Indicators for automatic routing:
|
||||
|
||||
**SPA (→ Firecrawl):**
|
||||
- URL contains `#/` or uses hash routing
|
||||
- Page source shows React/Vue/Angular markers
|
||||
- Content loads dynamically after initial load
|
||||
|
||||
**Static docs (→ Node.js/aiohttp):**
|
||||
- Built with Hugo, Jekyll, MkDocs, Docusaurus, GitBook
|
||||
- Clean HTML structure
|
||||
- Server-side rendered
|
||||
|
||||
## Scripts
|
||||
|
||||
- `scripts/select_crawler.py` - Intelligent crawler selection
|
||||
- `scripts/crawl_with_nodejs.sh` - Node.js wrapper
|
||||
- `scripts/crawl_with_aiohttp.sh` - Python wrapper
|
||||
- `scripts/crawl_with_firecrawl.py` - Firecrawl MCP wrapper
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `crawl_mgr.py select-crawler` | Recommend optimal crawler for a URL |
|
||||
| `crawl_mgr.py store-result` | Store crawl results and create manifest |
|
||||
| `crawl_mgr.py list-crawls` | List recent crawl records |
|
||||
|
||||
## Integration
|
||||
|
||||
| From | To |
|
||||
|------|-----|
|
||||
| reference-discovery | URL manifest input |
|
||||
| Firecrawl MCP | Raw crawled files |
|
||||
| → | content-repository (crawl manifest + raw files) |
|
||||
| quality-reviewer (deep_research) | Additional crawl requests |
|
||||
|
||||
Reference in New Issue
Block a user