feat(reference-curator): Add pipeline orchestrator and refactor skill format
Pipeline Orchestrator: - Add 07-pipeline-orchestrator skill with code/CLAUDE.md and desktop/SKILL.md - Add /reference-curator-pipeline slash command for full workflow automation - Add pipeline_runs and pipeline_iteration_tracker tables to schema.sql - Add v_pipeline_status and v_pipeline_iterations views - Add pipeline_config.yaml configuration template - Update AGENTS.md with Reference Curator Skills section - Update claude-project files with pipeline documentation Skill Format Refactoring: - Extract YAML frontmatter from SKILL.md files to separate skill.yaml - Add tools/ directories with MCP tool documentation - Update SKILL-FORMAT-REQUIREMENTS.md with new structure - Add migrate-skill-structure.py script for format conversion Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,11 +1,27 @@
|
||||
# Reference Curator - Claude.ai Project Knowledge
|
||||
|
||||
This project knowledge enables Claude to curate, process, and export reference documentation through 6 modular skills.
|
||||
This project knowledge enables Claude to curate, process, and export reference documentation through 7 modular skills.
|
||||
|
||||
## Quick Start - Pipeline Orchestrator
|
||||
|
||||
Run the full curation workflow with a single command:
|
||||
|
||||
```
|
||||
# Full pipeline from topic
|
||||
curate references on "Claude Code best practices"
|
||||
|
||||
# From URLs (skip discovery)
|
||||
curate these URLs: https://docs.anthropic.com/en/docs/prompt-caching
|
||||
|
||||
# With auto-approve
|
||||
curate references on "MCP servers" with auto-approve
|
||||
```
|
||||
|
||||
## Skills Overview
|
||||
|
||||
| Skill | Purpose | Trigger Phrases |
|
||||
|-------|---------|-----------------|
|
||||
| **pipeline-orchestrator** | Full 6-skill workflow with QA loops | "curate references", "run full pipeline", "automate curation" |
|
||||
| **reference-discovery** | Search & validate authoritative sources | "find references", "search documentation", "discover sources" |
|
||||
| **web-crawler** | Multi-backend crawling orchestration | "crawl URL", "fetch documents", "scrape pages" |
|
||||
| **content-repository** | MySQL storage management | "store content", "save to database", "check duplicates" |
|
||||
@@ -16,37 +32,43 @@ This project knowledge enables Claude to curate, process, and export reference d
|
||||
## Workflow
|
||||
|
||||
```
|
||||
[Topic Input]
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ reference-discovery │ → Search & validate sources
|
||||
└─────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────┐
|
||||
│ pipeline-orchestrator │ (Coordinates all stages)
|
||||
└───────────────────────────┘
|
||||
│
|
||||
┌───────────────────┼───────────────────┐
|
||||
▼ ▼ ▼
|
||||
[Topic Input] [URL Input] [Manifest Input]
|
||||
│ │ │
|
||||
▼ │ │
|
||||
┌─────────────────────┐ │ │
|
||||
│ reference-discovery │ ◄───┴───────────────────┘
|
||||
└─────────────────────┘ (skip if URLs/manifest)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ web-crawler │ → Crawl (Firecrawl/Node.js/aiohttp/Scrapy)
|
||||
└─────────────────────┘
|
||||
│
|
||||
▼
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ content-repository │ → Store in MySQL
|
||||
└─────────────────────┘
|
||||
│
|
||||
▼
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ content-distiller │ → Summarize & extract
|
||||
└─────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ quality-reviewer │ → QA loop
|
||||
└─────────────────────┘
|
||||
│
|
||||
├── REFACTOR → content-distiller
|
||||
├── DEEP_RESEARCH → web-crawler
|
||||
│
|
||||
▼ APPROVE
|
||||
│ content-distiller │ → Summarize & extract ◄────┐
|
||||
└─────────────────────┘ │
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────────┐ │
|
||||
│ quality-reviewer │ → QA loop │
|
||||
└─────────────────────┘ │
|
||||
│ │
|
||||
├── REFACTOR (max 3) ───────────────────────┤
|
||||
├── DEEP_RESEARCH (max 2) → crawler ────────┘
|
||||
│
|
||||
▼ APPROVE
|
||||
┌─────────────────────┐
|
||||
│ markdown-exporter │ → Project files / Fine-tuning
|
||||
└─────────────────────┘
|
||||
@@ -74,16 +96,28 @@ This project knowledge enables Claude to curate, process, and export reference d
|
||||
## Files in This Project
|
||||
|
||||
- `INDEX.md` - This overview file
|
||||
- `reference-curator-complete.md` - All 6 skills in one file
|
||||
- `reference-curator-complete.md` - All 7 skills in one file (recommended)
|
||||
- `01-reference-discovery.md` - Source discovery skill
|
||||
- `02-web-crawler.md` - Crawling orchestration skill
|
||||
- `03-content-repository.md` - Database storage skill
|
||||
- `04-content-distiller.md` - Content summarization skill
|
||||
- `05-quality-reviewer.md` - QA review skill
|
||||
- `06-markdown-exporter.md` - Export skill
|
||||
- `07-pipeline-orchestrator.md` - Full pipeline orchestration
|
||||
|
||||
## Usage
|
||||
|
||||
Upload all files to a Claude.ai Project, or upload only the skills you need.
|
||||
|
||||
For the complete experience, upload `reference-curator-complete.md` which contains all skills in one file.
|
||||
|
||||
## Pipeline Orchestrator Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| max_sources | 10 | Max sources to discover |
|
||||
| max_pages | 50 | Max pages per source |
|
||||
| auto_approve | false | Auto-approve above threshold |
|
||||
| threshold | 0.85 | Approval threshold |
|
||||
| max_iterations | 3 | Max QA loop iterations |
|
||||
| export_format | project_files | Output format |
|
||||
|
||||
Reference in New Issue
Block a user