# Reference Curator Skills - Refactoring Log

**Date**: 2025-01-28
**Version**: 2.0
**Author**: Claude Code (Opus 4.5)

---

## Summary

Complete restructuring of the Reference Curator skill suite from a flat structure to dual-platform format, with full installation automation.

---

## Changes Made

### 1. Directory Restructuring

**Before:**
```
90-reference-curator/
├── SKILL.md (Skill 01)
└── mnt/user-data/outputs/reference-curator-skills/
    ├── 02-web-crawler/SKILL.md
    ├── 03-content-repository/SKILL.md
    ├── 04-content-distiller/SKILL.md
    ├── 05-quality-reviewer/SKILL.md
    └── 06-markdown-exporter/SKILL.md
```

**After:**
```
90-reference-curator/
├── README.md
├── install.sh                    # NEW: Installation script
├── 01-reference-discovery/
│   ├── code/CLAUDE.md           # NEW: Claude Code directive
│   └── desktop/SKILL.md
├── 02-web-crawler-orchestrator/
│   ├── code/CLAUDE.md           # NEW
│   └── desktop/SKILL.md
├── 03-content-repository/
│   ├── code/CLAUDE.md           # NEW
│   └── desktop/SKILL.md
├── 04-content-distiller/
│   ├── code/CLAUDE.md           # NEW
│   └── desktop/SKILL.md
├── 05-quality-reviewer/
│   ├── code/CLAUDE.md           # NEW
│   └── desktop/SKILL.md
├── 06-markdown-exporter/
│   ├── code/CLAUDE.md           # NEW
│   └── desktop/SKILL.md
└── shared/
    ├── schema.sql               # NEW: MySQL schema
    └── config/
        ├── db_config.yaml       # NEW
        ├── crawl_config.yaml    # NEW
        └── export_config.yaml   # NEW
```

### 2. New Files Created

| File | Purpose |
|------|---------|
| `install.sh` | Interactive installation script |
| `shared/schema.sql` | MySQL schema (9 tables, 2 views) |
| `shared/config/db_config.yaml` | Database connection config |
| `shared/config/crawl_config.yaml` | Crawler routing config |
| `shared/config/export_config.yaml` | Export format config |
| `*/code/CLAUDE.md` | 6 Claude Code directives |

### 3. Crawler Configuration

Implemented intelligent crawler routing:

| Crawler | Condition | Use Case |
|---------|-----------|----------|
| **Node.js** (default) | ≤50 pages, static | Small documentation sites |
| **Python aiohttp** | ≤200 pages, SEO needed | Technical docs |
| **Scrapy** | >200 pages, multi-domain | Enterprise crawls |
| **Firecrawl MCP** | SPA, JS-rendered | Dynamic sites |

### 4. Installation Script Features

```bash
./install.sh           # Interactive installation
./install.sh --check   # Verify status
./install.sh --reset   # Reset (preserves data)
```

**Handles:**
- Config file deployment to `~/.config/reference-curator/`
- Storage directory creation at `~/reference-library/`
- MySQL credentials setup in `~/.envrc`
- Database creation and schema application
- Skill symlink registration in `~/.claude/skills/`

### 5. MySQL Schema

**Tables (9):**
- `sources` - Authoritative source registry
- `documents` - Crawled document storage
- `distilled_content` - Processed summaries
- `review_logs` - QA decision history
- `topics` - Topic taxonomy
- `document_topics` - Document-topic mapping
- `export_jobs` - Export task tracking
- `crawl_schedule` - Scheduled crawl jobs
- `change_detection` - Content change tracking

**Views (2):**
- `v_pending_reviews` - Documents awaiting review
- `v_export_ready` - Approved documents ready for export

### 6. Environment Setup

**Files installed:**
```
~/.config/reference-curator/
├── db_config.yaml
├── crawl_config.yaml
└── export_config.yaml

~/reference-library/
├── raw/
├── processed/
└── exports/

~/.claude/skills/
├── reference-discovery -> .../01-reference-discovery/desktop
├── web-crawler-orchestrator -> .../02-web-crawler-orchestrator/desktop
├── content-repository -> .../03-content-repository/desktop
├── content-distiller -> .../04-content-distiller/desktop
├── quality-reviewer -> .../05-quality-reviewer/desktop
└── markdown-exporter -> .../06-markdown-exporter/desktop
```

---

## Deleted

- `mnt/user-data/outputs/` directory (moved to proper structure)
- Root-level `SKILL.md` (moved to `01-reference-discovery/desktop/`)

---

## Verification

```bash
$ ./install.sh --check

✓ Configuration files (3/3)
✓ Storage directories (3/3)
✓ MySQL database (11 tables)
✓ Skill registrations (6/6)

All components installed correctly.
```

---

## Next Steps

1. Add Python scripts to `*/code/scripts/` folders for automation
2. Implement `select_crawler.py` for intelligent routing logic
3. Add unit tests for database operations
4. Create example workflows in `*/desktop/examples/`