feat(reference-curator): Add pipeline orchestrator and refactor skill format
Pipeline Orchestrator: - Add 07-pipeline-orchestrator skill with code/CLAUDE.md and desktop/SKILL.md - Add /reference-curator-pipeline slash command for full workflow automation - Add pipeline_runs and pipeline_iteration_tracker tables to schema.sql - Add v_pipeline_status and v_pipeline_iterations views - Add pipeline_config.yaml configuration template - Update AGENTS.md with Reference Curator Skills section - Update claude-project files with pipeline documentation Skill Format Refactoring: - Extract YAML frontmatter from SKILL.md files to separate skill.yaml - Add tools/ directories with MCP tool documentation - Update SKILL-FORMAT-REQUIREMENTS.md with new structure - Add migrate-skill-structure.py script for format conversion Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,175 @@
|
||||
# Pipeline Orchestrator
|
||||
|
||||
Coordinates the full 6-skill reference curation workflow with automated QA loop handling.
|
||||
|
||||
## Trigger Phrases
|
||||
|
||||
- "curate references on [topic]"
|
||||
- "run full curation pipeline"
|
||||
- "automate reference curation"
|
||||
- "curate these URLs: [url1, url2]"
|
||||
|
||||
## Input Modes
|
||||
|
||||
| Mode | Example | Pipeline Start |
|
||||
|------|---------|----------------|
|
||||
| **Topic** | "curate references on Claude system prompts" | Stage 1 (discovery) |
|
||||
| **URLs** | "curate these URLs: https://docs.anthropic.com/..." | Stage 2 (crawler) |
|
||||
| **Manifest** | "resume curation from manifest.json" | Stage 2 (crawler) |
|
||||
|
||||
## Pipeline Stages
|
||||
|
||||
```
|
||||
1. reference-discovery (topic mode only)
|
||||
│
|
||||
▼
|
||||
2. web-crawler-orchestrator
|
||||
│
|
||||
▼
|
||||
3. content-repository
|
||||
│
|
||||
▼
|
||||
4. content-distiller ◄─────────────┐
|
||||
│ │
|
||||
▼ │
|
||||
5. quality-reviewer │
|
||||
│ │
|
||||
├── APPROVE → Stage 6 │
|
||||
├── REFACTOR ───────────────┤
|
||||
├── DEEP_RESEARCH → Stage 2 ┘
|
||||
└── REJECT → Archive
|
||||
│
|
||||
▼
|
||||
6. markdown-exporter
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| max_sources | 10 | Maximum sources to discover (topic mode) |
|
||||
| max_pages | 50 | Maximum pages per source to crawl |
|
||||
| auto_approve | false | Auto-approve scores above threshold |
|
||||
| threshold | 0.85 | Quality score threshold for approval |
|
||||
| max_iterations | 3 | Maximum QA loop iterations per document |
|
||||
| export_format | project_files | Output format (project_files, fine_tuning, jsonl) |
|
||||
|
||||
## QA Loop Handling
|
||||
|
||||
The orchestrator automatically handles QA decisions:
|
||||
|
||||
| Decision | Action | Iteration Limit |
|
||||
|----------|--------|-----------------|
|
||||
| **APPROVE** | Proceed to export | - |
|
||||
| **REFACTOR** | Re-distill with feedback | 3 iterations |
|
||||
| **DEEP_RESEARCH** | Crawl more sources, re-distill | 2 iterations |
|
||||
| **REJECT** | Archive with reason | - |
|
||||
|
||||
After reaching iteration limits, documents are marked `needs_manual_review`.
|
||||
|
||||
## State Management
|
||||
|
||||
### With Database
|
||||
|
||||
Pipeline state is tracked in `pipeline_runs` table:
|
||||
- Run ID, input type, current stage
|
||||
- Statistics (crawled, distilled, approved, etc.)
|
||||
- Error handling and resume capability
|
||||
|
||||
### File-Based Fallback
|
||||
|
||||
State saved to `~/reference-library/pipeline_state/run_XXX/`:
|
||||
- `state.json` - Current stage and statistics
|
||||
- `manifest.json` - Discovered sources
|
||||
- `review_log.json` - QA decisions
|
||||
|
||||
## Progress Tracking
|
||||
|
||||
The orchestrator reports progress at each stage:
|
||||
|
||||
```
|
||||
[Pipeline] Stage 1/6: Discovery - Found 8 sources
|
||||
[Pipeline] Stage 2/6: Crawling - 45/50 pages complete
|
||||
[Pipeline] Stage 3/6: Storing - 45 documents saved
|
||||
[Pipeline] Stage 4/6: Distilling - 45 documents processed
|
||||
[Pipeline] Stage 5/6: Reviewing - 40 approved, 3 refactored, 2 rejected
|
||||
[Pipeline] Stage 6/6: Exporting - 40 documents exported
|
||||
[Pipeline] Complete! See ~/reference-library/exports/
|
||||
```
|
||||
|
||||
## Output Summary
|
||||
|
||||
On completion, returns detailed summary:
|
||||
|
||||
```
|
||||
Pipeline Complete:
|
||||
- Sources discovered: 5
|
||||
- Pages crawled: 45
|
||||
- Documents stored: 45
|
||||
- Approved: 40
|
||||
- Refactored: 8
|
||||
- Deep researched: 2
|
||||
- Rejected: 3
|
||||
- Needs manual review: 2
|
||||
|
||||
Exports saved to: ~/reference-library/exports/
|
||||
Format: project_files
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
If a stage fails:
|
||||
1. State is checkpointed
|
||||
2. Error is logged with details
|
||||
3. Pipeline can be resumed from last successful stage
|
||||
|
||||
## Resume Pipeline
|
||||
|
||||
To resume a paused or failed pipeline:
|
||||
- Provide the manifest.json or run state file
|
||||
- Pipeline continues from last checkpoint
|
||||
|
||||
## Example Usage
|
||||
|
||||
### Full Pipeline from Topic
|
||||
|
||||
```
|
||||
User: Curate references on Claude Code best practices with 5 sources
|
||||
|
||||
Claude: I'll run the full curation pipeline for "Claude Code best practices":
|
||||
|
||||
[Pipeline] Stage 1/6: Discovering sources...
|
||||
Found 5 authoritative sources (3 official, 2 community)
|
||||
|
||||
[Pipeline] Stage 2/6: Crawling 5 sources...
|
||||
Crawled 45 pages total
|
||||
|
||||
[Pipeline] Stage 3/6: Storing documents...
|
||||
45 documents saved to repository
|
||||
|
||||
[Pipeline] Stage 4/6: Distilling content...
|
||||
45 documents processed
|
||||
|
||||
[Pipeline] Stage 5/6: Quality review...
|
||||
- 38 approved
|
||||
- 5 refactored (re-distilling...)
|
||||
- 2 rejected (low quality)
|
||||
|
||||
[Pipeline] Stage 6/6: Exporting...
|
||||
40 documents exported to ~/reference-library/exports/
|
||||
|
||||
Pipeline complete! 40 reference documents ready for use.
|
||||
```
|
||||
|
||||
### From Specific URLs
|
||||
|
||||
```
|
||||
User: Curate these URLs with auto-approve:
|
||||
- https://docs.anthropic.com/en/docs/prompt-caching
|
||||
- https://docs.anthropic.com/en/docs/build-with-claude/tool-use
|
||||
|
||||
Claude: Running pipeline from URLs (skipping discovery)...
|
||||
|
||||
[Pipeline] Stage 2/6: Crawling 2 sources...
|
||||
[...continues with remaining stages...]
|
||||
```
|
||||
Reference in New Issue
Block a user