- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory
alongside distilled content, with YAML frontmatter for traceability
- Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer),
pure archival mode for raw documentation capture
- Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend:
token-bucket rate limiting, semaphore concurrency, tenacity retries,
full page metadata extraction (headings, links, schema, OG)
- New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests
with progress tracking and resume support
- Update crawler selection: docs sites now default to seo-aiohttp
- Update crawl_config.yaml with seo-aiohttp routing rules
- Update pipeline orchestrator with flag resolution and skip paths
- Update README.md and USER-GUIDE.md with raw mode documentation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pipeline Orchestrator:
- Add 07-pipeline-orchestrator skill with code/CLAUDE.md and desktop/SKILL.md
- Add /reference-curator-pipeline slash command for full workflow automation
- Add pipeline_runs and pipeline_iteration_tracker tables to schema.sql
- Add v_pipeline_status and v_pipeline_iterations views
- Add pipeline_config.yaml configuration template
- Update AGENTS.md with Reference Curator Skills section
- Update claude-project files with pipeline documentation
Skill Format Refactoring:
- Extract YAML frontmatter from SKILL.md files to separate skill.yaml
- Add tools/ directories with MCP tool documentation
- Update SKILL-FORMAT-REQUIREMENTS.md with new structure
- Add migrate-skill-structure.py script for format conversion
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add claude-project/ folder with skill files formatted for upload to
Claude.ai Projects (web interface):
- reference-curator-complete.md: All 6 skills consolidated
- INDEX.md: Overview and workflow documentation
- Individual skill files (01-06) without YAML frontmatter
Add --claude-ai option to install.sh:
- Lists available files for upload
- Optionally copies to custom destination directory
- Provides upload instructions for Claude.ai
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>