our-claude-skills/custom-skills/90-reference-curator/USER-GUIDE.md

Reference Curator - User Guide

A modular skill suite for curating reference documentation with Claude Code.

Quick Start

One-Shot Full Pipeline (Recommended)

# Research a topic and generate reference library
/reference-curator-pipeline "Claude Code best practices" --auto-approve

# From specific URLs
/reference-curator-pipeline https://docs.anthropic.com/en/docs/agents

# With options
/reference-curator-pipeline "MCP servers" --max-sources 10 --export-format fine_tuning

# Save raw crawled content alongside distilled output
/reference-curator-pipeline "Claude Code best practices" --save-raw

# Pure archival — skip distillation, raw markdown only
/reference-curator-pipeline https://docs.anthropic.com --no-distill

Run as Background Agent

Ask Claude Code:

"Run reference-curator as a background agent for topic X"

This launches autonomous processing while you continue other work.

---

Individual Skills

Use these when you need granular control over each stage.

1. Reference Discovery

Purpose: Search and discover authoritative sources

/reference-discovery "prompt engineering techniques"

Output: Discovery manifest (JSON) with scored URLs

Tier	Score	Sources
tier1_official	≥0.60	Official docs, vendor blogs
tier2_verified	0.40-0.59	Research papers, cookbooks
tier3_community	<0.40	Tutorials, blog posts

---

2. Web Crawler

Purpose: Crawl URLs and save raw content

# From discovery manifest
/web-crawler /path/to/manifest.json

# Single URL
/web-crawler https://docs.example.com/guide

Output: Markdown files in ~/Documents/05_AI Agent/10_Reference Library/raw/YYYY/MM/

Crawler Selection:

Crawler	Best For
Firecrawl (default)	SPAs, JS-rendered sites
WebFetch	Simple documentation pages
Node.js	Small static sites (≤50 pages)

---

3. Content Repository

Purpose: Store documents in MySQL with versioning

# Store crawled content
/content-repository store --topic "my-topic"

# Query existing documents
/content-repository query --topic "my-topic"

Prerequisites: MySQL database reference_library with schema applied.

---

4. Content Distiller

Purpose: Summarize and extract key concepts

/content-distiller --topic "my-topic"

Output per document:

• Executive summary (2-3 sentences)
• Key concepts (JSON)
• Code snippets (JSON)
• Structured content (Markdown)
• ~15-20% compression ratio

---

5. Quality Reviewer

Purpose: Score and approve content

/quality-reviewer --topic "my-topic"

Scoring Criteria:

Criterion	Weight
Accuracy	25%
Completeness	20%
Clarity	20%
PE Quality	25%
Usability	10%

Decisions:

Score	Decision	Action
≥0.85	APPROVE	Ready for export
0.60-0.84	REFACTOR	Re-distill with feedback
0.40-0.59	DEEP_RESEARCH	Crawl more sources
<0.40	REJECT	Archive

---

6. Markdown Exporter

Purpose: Generate final export files

# Project files (for Claude Projects)
/markdown-exporter --topic "my-topic" --format project_files

# Fine-tuning dataset (JSONL)
/markdown-exporter --topic "my-topic" --format fine_tuning

Output Structure:

~/Documents/05_AI Agent/10_Reference Library/exports/
├── INDEX.md
└── my-topic/
    ├── _index.md
    ├── 01-document-one.md
    ├── 02-document-two.md
    └── ...

---

Raw Mode

What is Raw Mode?

By default, the pipeline distills crawled content down to 25-35% of the original, extracting key concepts, code snippets, and structured summaries. Raw mode preserves the full, unmodified crawled markdown.

--save-raw Flag

Saves intact crawled markdown alongside the distilled content in a raw/ subdirectory.

/reference-curator-pipeline "Anthropic Claude API" --save-raw

Output:

exports/claude-api/
├── INDEX.md
├── 00-prompt-caching.md           # Distilled (25-35% of original)
├── 01-streaming.md
├── raw/
│   ├── 00-prompt-caching.raw.md   # Full crawled content
│   ├── 01-streaming.raw.md
│   └── claude-api-raw-complete.md # Combined raw bundle
└── manifest.json

Use cases:

• Need both concise reference AND full-text source material
• Want to re-distill later with different parameters
• Auditing what content was captured vs. what was extracted

--no-distill Flag

Skips stages 4 (content-distiller) and 5 (quality-reviewer) entirely. The pipeline goes directly from crawl/store to export. Raw files become the primary output. Implies --save-raw automatically.

/reference-curator-pipeline https://docs.anthropic.com --no-distill

Output:

exports/anthropic-docs/
├── README.md
├── 00-getting-started.raw.md
├── 01-prompt-engineering.raw.md
├── anthropic-docs-raw-complete.md
└── manifest.json

Use cases:

• Site archival / preservation
• Migration reference (need exact original content)
• Training data collection (distillation would remove useful context)
• NotebookLM / LLM context sources (full fidelity preferred)
• Quick capture when you will distill manually later

Combining with Other Flags

# Deep archival crawl
/reference-curator-pipeline https://legacy.example.com --depth full --no-distill

# Standard crawl with raw backup and auto-approval
/reference-curator-pipeline "MCP servers" --save-raw --auto-approve

# Fine-tuning dataset from raw content
/reference-curator-pipeline "prompt engineering" --no-distill --export-format fine_tuning

Pipeline Stage Comparison

Stage	Default	--save-raw	--no-distill
1. Discovery	Runs	Runs	Runs
2. Web Crawler	Runs	Runs + writes raw/	Runs + writes raw
3. Repository	Runs	Runs	Runs
4. Distiller	Runs	Runs	Skipped
5. Reviewer	Runs	Runs	Skipped
6. Exporter	Distilled	Distilled + raw	Raw only

---

Common Workflows

Workflow 1: Quick Reference on a Topic

You: "Create a reference library on n8n self-hosting"
Claude: [Runs full pipeline automatically]

Workflow 2: Curate Specific URLs

You: "Add these URLs to my reference library:
      - https://docs.example.com/guide1
      - https://docs.example.com/guide2"
Claude: [Skips discovery, crawls URLs directly]

Workflow 3: Update Existing Topic

You: "Update the MCP developer manual with latest SDK docs"
Claude: [Discovers new sources, crawls, merges with existing]

Workflow 4: Export for Fine-Tuning

You: "Export all approved content as JSONL for fine-tuning"
Claude: /markdown-exporter --format fine_tuning

---

Configuration

Storage Paths

Type	Location
Raw content	~/Documents/05_AI Agent/10_Reference Library/raw/
Raw output (--save-raw)	{output}/{topic-slug}/raw/
Raw output (--no-distill)	{output}/{topic-slug}/ (primary)
Exports	~/Documents/05_AI Agent/10_Reference Library/exports/
Config	~/.config/reference-curator/

MySQL Setup

Database credentials via ~/.my.cnf:

[client]
user = root
password = your_password
socket = /tmp/mysql.sock

Config Files

~/.config/reference-curator/
├── db_config.yaml      # Database connection
├── crawl_config.yaml   # Crawler settings
└── export_config.yaml  # Export preferences

---

Tips

For Best Results

1. Be specific with topics - "Claude Code MCP integration" > "MCP"
2. Use official sources first - tier1 sources get higher quality scores
3. Review before export - Check quality scores, refine if needed
4. Use background agents - For large topics, run as background task

Troubleshooting

Issue	Solution
MySQL auth failed	Check ~/.my.cnf permissions (chmod 600)
Missing table	Run schema.sql from shared/schema.sql
Low quality scores	Add more tier1 sources, re-distill
Crawl timeout	Try different crawler or reduce max_pages
Export empty	Check documents are approved in quality review

Checking Pipeline Status

-- View pending documents
SELECT title, review_status FROM distilled_content
JOIN documents USING (doc_id) WHERE review_status = 'pending';

-- View export history
SELECT export_name, total_documents, status FROM export_jobs;

---

Example Session

You: Create a reference library on "Anthropic Claude API best practices"

Claude: Starting reference-curator pipeline...

[1/6] Reference Discovery
- Searching docs.anthropic.com, github.com/anthropics...
- Found 12 authoritative sources (8 tier1, 3 tier2, 1 tier3)

[2/6] Web Crawling
- Crawling 12 URLs with WebFetch...
- Saved 12 markdown files

[3/6] Content Repository
- Stored 12 documents in MySQL
- Created topic: claude-api-best-practices

[4/6] Content Distillation
- Distilled 12 documents (~18% avg compression)
- Extracted 45 key concepts

[5/6] Quality Review
- Scored all documents
- 11 APPROVED (≥0.85), 1 REFACTOR (0.78)
- Re-distilling 1 document... APPROVED (0.86)

[6/6] Markdown Export
- Generated 14 files (12 docs + 2 indexes)
- Output: ~/Documents/.../exports/claude-api-best-practices/

Pipeline complete! Reference library ready.

Example: Archival Mode (--no-distill)

You: Archive the GA4 developer docs without distilling — just raw content

Claude: Starting reference-curator pipeline (--no-distill mode)...

[1/6] Reference Discovery
- Skipped (URL input mode)

[2/6] Web Crawling
- Crawling developers.google.com/analytics with Firecrawl...
- Saved 28 raw markdown files (.raw.md)

[3/6] Content Repository
- Stored 28 documents in MySQL

[4/6] Content Distillation — SKIPPED (--no-distill)
[5/6] Quality Review — SKIPPED (--no-distill)

[6/6] Markdown Export (raw mode)
- Generated 28 .raw.md files + combined raw bundle
- Output: ~/Documents/.../google-analytics-4/

Pipeline complete! 28 raw pages archived.
Files ready for NotebookLM upload or direct LLM context use.

---

Need Help?

• Skill source code: custom-skills/90-reference-curator/
• MySQL schema: shared/schema.sql
• Config templates: shared/config/