feat(reference-curator): implement Python scripts + Gemini quality gate
Build the refcurator shared Python package and 7 CLI scripts that were previously specification-only. Add Gemini CLI as an independent pre-distillation quality evaluator, replacing the circular Claude-self-review pattern. Key changes: - shared/lib/src/refcurator/: 7-module package (config, db, models, utils, manifest, gemini) with PyMySQL + JSON file dual backend - 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer, exporter, pipeline — each with subcommands for data management - Gemini quality gate: evaluates raw content BEFORE distillation using 5 criteria (relevance, authority, completeness, freshness, distill_value) - Pipeline reordered: discovery → crawl → store → evaluate → distill → export - Bug fixes from Codex adversarial review: - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries - Exporter uses MAX(review_id) to prevent shipping stale approvals - Distiller updates existing rows on refactor instead of forking - Updated all 7 CLAUDE.md directives with real script references - install.sh updated with refcurator package install step 51/51 E2E tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Pipeline Orchestrator
|
||||
|
||||
Coordinates the full 6-skill reference curation workflow with QA loop handling.
|
||||
Coordinates the full 6-skill reference curation workflow with QA loop handling. Manages pipeline state (init, advance, pause, resume, complete) while Claude orchestrates the actual stage execution.
|
||||
|
||||
## Trigger Keywords
|
||||
"curate references", "full pipeline", "run curation", "reference-curator-pipeline"
|
||||
@@ -22,257 +22,144 @@ Coordinates the full 6-skill reference curation workflow with QA loop handling.
|
||||
crawler ─┘
|
||||
```
|
||||
|
||||
## Input Detection
|
||||
## Pipeline State Management
|
||||
|
||||
Parse input to determine mode:
|
||||
|
||||
```python
|
||||
def detect_input_mode(input_value):
|
||||
if input_value.endswith('.json') and os.path.exists(input_value):
|
||||
return 'manifest'
|
||||
elif input_value.startswith('http://') or input_value.startswith('https://'):
|
||||
return 'urls'
|
||||
else:
|
||||
return 'topic'
|
||||
### Initialize a Run
|
||||
```bash
|
||||
uv run python scripts/pipeline.py init \
|
||||
--input "prompt engineering" --type topic \
|
||||
--options '{"max_sources": 10, "auto_approve": true}'
|
||||
```
|
||||
|
||||
## Pipeline Execution
|
||||
### Advance to Next Stage
|
||||
```bash
|
||||
uv run python scripts/pipeline.py advance \
|
||||
--run-id 1 --stage crawling \
|
||||
--stats '{"sources_discovered": 8}'
|
||||
```
|
||||
|
||||
### Stage 1: Reference Discovery (Topic Mode Only)
|
||||
### Pause on Error
|
||||
```bash
|
||||
uv run python scripts/pipeline.py pause \
|
||||
--run-id 1 --error "Crawl timeout on page 45" --stage crawling
|
||||
```
|
||||
|
||||
### Resume from Pause
|
||||
```bash
|
||||
uv run python scripts/pipeline.py resume --run-id 1
|
||||
```
|
||||
|
||||
### Complete Pipeline
|
||||
```bash
|
||||
uv run python scripts/pipeline.py complete \
|
||||
--run-id 1 --export-path ~/reference-library/exports/ --export-count 40
|
||||
```
|
||||
|
||||
### Check Status
|
||||
```bash
|
||||
uv run python scripts/pipeline.py status --run-id 1
|
||||
uv run python scripts/pipeline.py status --all
|
||||
```
|
||||
|
||||
## QA Loop Tracking
|
||||
|
||||
```bash
|
||||
# Skip if input mode is 'urls' or 'manifest'
|
||||
if mode == 'topic':
|
||||
/reference-discovery "$TOPIC" --max-sources $MAX_SOURCES
|
||||
# Output: manifest.json
|
||||
# Track a refactor iteration for a document
|
||||
uv run python scripts/pipeline.py track-iteration \
|
||||
--run-id 1 --doc-id 42 --action refactor
|
||||
|
||||
# Track a deep research iteration
|
||||
uv run python scripts/pipeline.py track-iteration \
|
||||
--run-id 1 --doc-id 42 --action deep_research
|
||||
```
|
||||
|
||||
Returns one of:
|
||||
- `re_distill` — proceed with refactor
|
||||
- `re_crawl_and_distill` — proceed with deep research
|
||||
- `needs_manual_review` — max iterations exceeded
|
||||
|
||||
| Decision | Max Iterations |
|
||||
|----------|----------------|
|
||||
| REFACTOR | 3 |
|
||||
| DEEP_RESEARCH | 2 |
|
||||
| Combined total | 5 |
|
||||
|
||||
## Pipeline Execution Flow
|
||||
|
||||
### Stage 1: Reference Discovery (Topic Mode Only)
|
||||
```
|
||||
If mode == 'topic':
|
||||
→ Claude runs WebSearch
|
||||
→ discover.py create-manifest
|
||||
→ discover.py dedup
|
||||
→ pipeline.py advance --stage crawling
|
||||
```
|
||||
|
||||
### Stage 2: Web Crawler
|
||||
|
||||
```bash
|
||||
# From manifest or URLs
|
||||
/web-crawler $INPUT --max-pages $MAX_PAGES
|
||||
# Output: crawled files in ~/reference-library/raw/
|
||||
```
|
||||
→ Claude uses Firecrawl MCP tools
|
||||
→ crawl_mgr.py store-result
|
||||
→ pipeline.py advance --stage storing
|
||||
```
|
||||
|
||||
### Stage 3: Content Repository
|
||||
|
||||
```bash
|
||||
/content-repository store
|
||||
# Output: documents stored in MySQL or file-based storage
|
||||
```
|
||||
→ repo.py store (for each crawled doc)
|
||||
→ pipeline.py advance --stage evaluating
|
||||
```
|
||||
|
||||
### Stage 4: Content Distiller
|
||||
|
||||
```bash
|
||||
/content-distiller all-pending
|
||||
# Output: distilled content records
|
||||
### Stage 4: Gemini Quality Gate (Pre-Distillation)
|
||||
```
|
||||
→ reviewer.py gemini-evaluate-pending --topic "$TOPIC" --auto-approve
|
||||
→ APPROVE: proceed to distillation
|
||||
→ DEEP_RESEARCH: pipeline.py track-iteration → crawler (re-crawl)
|
||||
→ REJECT: skip document entirely
|
||||
→ pipeline.py advance --stage distilling
|
||||
```
|
||||
|
||||
### Stage 5: Quality Reviewer
|
||||
|
||||
```bash
|
||||
if auto_approve:
|
||||
/quality-reviewer all-pending --auto-approve --threshold $THRESHOLD
|
||||
else:
|
||||
/quality-reviewer all-pending
|
||||
### Stage 5: Content Distiller (Approved Only)
|
||||
```
|
||||
→ distiller.py load-pending
|
||||
→ Claude distills each approved document
|
||||
→ distiller.py store
|
||||
→ pipeline.py advance --stage exporting
|
||||
```
|
||||
|
||||
Handle QA decisions:
|
||||
- **APPROVE**: Add to export queue
|
||||
- **REFACTOR**: Re-run distiller with feedback (track iteration count)
|
||||
- **DEEP_RESEARCH**: Run crawler for additional sources, then distill
|
||||
- **REJECT**: Archive with reason
|
||||
|
||||
### Stage 6: Markdown Exporter
|
||||
|
||||
```bash
|
||||
/markdown-exporter $EXPORT_FORMAT
|
||||
# Output: files in ~/reference-library/exports/
|
||||
```
|
||||
|
||||
## State Management
|
||||
|
||||
### Initialize Pipeline State
|
||||
|
||||
```python
|
||||
def init_pipeline_state(run_id, input_value, options):
|
||||
state = {
|
||||
"run_id": run_id,
|
||||
"run_type": detect_input_mode(input_value),
|
||||
"input_value": input_value,
|
||||
"status": "running",
|
||||
"current_stage": "discovery",
|
||||
"options": options,
|
||||
"stats": {
|
||||
"sources_discovered": 0,
|
||||
"pages_crawled": 0,
|
||||
"documents_stored": 0,
|
||||
"documents_distilled": 0,
|
||||
"approved": 0,
|
||||
"refactored": 0,
|
||||
"deep_researched": 0,
|
||||
"rejected": 0,
|
||||
"needs_manual_review": 0
|
||||
},
|
||||
"started_at": datetime.now().isoformat()
|
||||
}
|
||||
save_state(run_id, state)
|
||||
return state
|
||||
```
|
||||
|
||||
### MySQL State (Preferred)
|
||||
|
||||
```sql
|
||||
INSERT INTO pipeline_runs (run_type, input_value, options)
|
||||
VALUES ('topic', 'Claude system prompts', '{"max_sources": 10}');
|
||||
```
|
||||
|
||||
### File-Based Fallback
|
||||
|
||||
```
|
||||
~/reference-library/pipeline_state/run_XXX/
|
||||
├── state.json # Current stage and stats
|
||||
├── manifest.json # Discovered sources
|
||||
├── crawl_results.json # Crawled document paths
|
||||
├── review_log.json # QA decisions per document
|
||||
└── errors.log # Any errors encountered
|
||||
```
|
||||
|
||||
## QA Loop Logic
|
||||
|
||||
```python
|
||||
MAX_REFACTOR_ITERATIONS = 3
|
||||
MAX_DEEP_RESEARCH_ITERATIONS = 2
|
||||
MAX_TOTAL_ITERATIONS = 5
|
||||
|
||||
def handle_qa_decision(doc_id, decision, iteration_counts):
|
||||
refactor_count = iteration_counts.get('refactor', 0)
|
||||
research_count = iteration_counts.get('deep_research', 0)
|
||||
total = refactor_count + research_count
|
||||
|
||||
if total >= MAX_TOTAL_ITERATIONS:
|
||||
return 'needs_manual_review'
|
||||
|
||||
if decision == 'refactor':
|
||||
if refactor_count >= MAX_REFACTOR_ITERATIONS:
|
||||
return 'needs_manual_review'
|
||||
iteration_counts['refactor'] = refactor_count + 1
|
||||
return 're_distill'
|
||||
|
||||
if decision == 'deep_research':
|
||||
if research_count >= MAX_DEEP_RESEARCH_ITERATIONS:
|
||||
return 'needs_manual_review'
|
||||
iteration_counts['deep_research'] = research_count + 1
|
||||
return 're_crawl_and_distill'
|
||||
|
||||
return decision # approve or reject
|
||||
→ exporter.py project
|
||||
→ exporter.py index
|
||||
→ exporter.py crossrefs
|
||||
→ exporter.py verify
|
||||
→ pipeline.py complete
|
||||
```
|
||||
|
||||
## Checkpoint Strategy
|
||||
|
||||
Save checkpoint after each stage completes:
|
||||
|
||||
| Stage | Checkpoint | Resume Point |
|
||||
|-------|------------|--------------|
|
||||
| discovery | `manifest.json` created | → crawler |
|
||||
| crawl | `crawl_results.json` | → repository |
|
||||
| store | DB records or file list | → distiller |
|
||||
| discovery | manifest.json created | → crawler |
|
||||
| crawl | crawl_result.json | → repository |
|
||||
| store | DB records | → distiller |
|
||||
| distill | distilled_content records | → reviewer |
|
||||
| review | review_logs records | → exporter or loop |
|
||||
| export | final export complete | Done |
|
||||
|
||||
## Progress Reporting
|
||||
## Scripts
|
||||
|
||||
Report progress to user at key checkpoints:
|
||||
|
||||
```
|
||||
[Pipeline] Stage 1/6: Discovery - Found 8 sources
|
||||
[Pipeline] Stage 2/6: Crawling - 45/50 pages complete
|
||||
[Pipeline] Stage 3/6: Storing - 45 documents saved
|
||||
[Pipeline] Stage 4/6: Distilling - 45 documents processed
|
||||
[Pipeline] Stage 5/6: Reviewing - 40 approved, 3 refactored, 2 rejected
|
||||
[Pipeline] Stage 6/6: Exporting - 40 documents exported
|
||||
[Pipeline] Complete! See ~/reference-library/exports/
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```python
|
||||
def handle_stage_error(stage, error, state):
|
||||
state['status'] = 'paused'
|
||||
state['error_message'] = str(error)
|
||||
state['error_stage'] = stage
|
||||
save_state(state['run_id'], state)
|
||||
|
||||
# Log to errors.log
|
||||
log_error(state['run_id'], stage, error)
|
||||
|
||||
# Report to user
|
||||
return f"Pipeline paused at {stage}: {error}. Resume with run_id {state['run_id']}"
|
||||
```
|
||||
|
||||
## Resume Pipeline
|
||||
|
||||
```python
|
||||
def resume_pipeline(run_id):
|
||||
state = load_state(run_id)
|
||||
|
||||
if state['status'] != 'paused':
|
||||
return f"Pipeline {run_id} is {state['status']}, cannot resume"
|
||||
|
||||
stage = state['current_stage']
|
||||
state['status'] = 'running'
|
||||
state['error_message'] = None
|
||||
save_state(run_id, state)
|
||||
|
||||
# Resume from failed stage
|
||||
return execute_from_stage(stage, state)
|
||||
```
|
||||
|
||||
## Output Summary
|
||||
|
||||
On completion, generate summary:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_id": 123,
|
||||
"status": "completed",
|
||||
"duration_minutes": 15,
|
||||
"stats": {
|
||||
"sources_discovered": 5,
|
||||
"pages_crawled": 45,
|
||||
"documents_stored": 45,
|
||||
"documents_distilled": 45,
|
||||
"approved": 40,
|
||||
"refactored": 8,
|
||||
"deep_researched": 2,
|
||||
"rejected": 3,
|
||||
"needs_manual_review": 2
|
||||
},
|
||||
"exports": {
|
||||
"format": "project_files",
|
||||
"path": "~/reference-library/exports/",
|
||||
"document_count": 40
|
||||
},
|
||||
"errors": []
|
||||
}
|
||||
```
|
||||
|
||||
## Integration Points
|
||||
|
||||
| Skill | Called By | Provides |
|
||||
|-------|-----------|----------|
|
||||
| reference-discovery | Orchestrator | manifest.json |
|
||||
| web-crawler | Orchestrator | Raw crawled files |
|
||||
| content-repository | Orchestrator | Stored documents |
|
||||
| content-distiller | Orchestrator, QA loop | Distilled content |
|
||||
| quality-reviewer | Orchestrator | QA decisions |
|
||||
| markdown-exporter | Orchestrator | Final exports |
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `pipeline.py init` | Initialize a new pipeline run |
|
||||
| `pipeline.py advance` | Advance to next stage |
|
||||
| `pipeline.py pause` | Pause on error |
|
||||
| `pipeline.py resume` | Resume from pause |
|
||||
| `pipeline.py complete` | Mark pipeline complete |
|
||||
| `pipeline.py status` | Show run status |
|
||||
| `pipeline.py track-iteration` | Track QA loop iterations |
|
||||
|
||||
## Configuration
|
||||
|
||||
Read from `~/.config/reference-curator/pipeline_config.yaml`:
|
||||
Reads from `~/.config/reference-curator/pipeline_config.yaml`:
|
||||
|
||||
```yaml
|
||||
pipeline:
|
||||
@@ -288,9 +175,4 @@ qa_loop:
|
||||
|
||||
export:
|
||||
default_format: project_files
|
||||
include_rejected: false
|
||||
|
||||
state:
|
||||
backend: mysql # or 'file'
|
||||
state_directory: ~/reference-library/pipeline_state/
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user