Andrew Yim
f215c11c32
Build the refcurator shared Python package and 7 CLI scripts that were previously specification-only. Add Gemini CLI as an independent pre-distillation quality evaluator, replacing the circular Claude-self-review pattern. Key changes: - shared/lib/src/refcurator/: 7-module package (config, db, models, utils, manifest, gemini) with PyMySQL + JSON file dual backend - 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer, exporter, pipeline — each with subcommands for data management - Gemini quality gate: evaluates raw content BEFORE distillation using 5 criteria (relevance, authority, completeness, freshness, distill_value) - Pipeline reordered: discovery → crawl → store → evaluate → distill → export - Bug fixes from Codex adversarial review: - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries - Exporter uses MAX(review_id) to prevent shipping stale approvals - Distiller updates existing rows on refactor instead of forking - Updated all 7 CLAUDE.md directives with real script references - install.sh updated with refcurator package install step 51/51 E2E tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
94 lines
2.4 KiB
Markdown
94 lines
2.4 KiB
Markdown
# Markdown Exporter
|
|
|
|
Exports approved reference content as structured markdown files for project knowledge or fine-tuning datasets.
|
|
|
|
## Trigger Keywords
|
|
"export references", "generate project files", "create markdown output", "export for fine-tuning", "build knowledge base"
|
|
|
|
## Export Types
|
|
|
|
| Type | Format | Use Case |
|
|
|------|--------|----------|
|
|
| `project` | Nested markdown | Claude Projects knowledge |
|
|
| `finetuning` | JSONL | Model fine-tuning dataset |
|
|
|
|
## Workflow
|
|
|
|
### Step 1: Export Project Files
|
|
```bash
|
|
uv run python scripts/exporter.py project \
|
|
--output ~/reference-library/exports/ \
|
|
--min-score 0.80 \
|
|
--structure nested_by_topic
|
|
```
|
|
|
|
Output structure:
|
|
```
|
|
exports/
|
|
├── INDEX.md
|
|
├── prompt-engineering/
|
|
│ ├── _index.md
|
|
│ ├── 00-chain-of-thought.md
|
|
│ └── 01-few-shot-prompting.md
|
|
└── claude-models/
|
|
├── _index.md
|
|
└── 00-model-comparison.md
|
|
```
|
|
|
|
### Step 2: Generate INDEX
|
|
```bash
|
|
uv run python scripts/exporter.py index --output ~/reference-library/exports/INDEX.md
|
|
```
|
|
|
|
### Step 3: Add Cross-References
|
|
```bash
|
|
uv run python scripts/exporter.py crossrefs --input ~/reference-library/exports/
|
|
```
|
|
|
|
### Step 4: Verify Export
|
|
```bash
|
|
uv run python scripts/exporter.py verify --path ~/reference-library/exports/
|
|
```
|
|
|
|
### Step 5: Fine-tuning Export (Optional)
|
|
```bash
|
|
uv run python scripts/exporter.py finetuning \
|
|
--output ~/reference-library/exports/training.jsonl \
|
|
--max-tokens 4096
|
|
```
|
|
|
|
JSONL format:
|
|
```json
|
|
{
|
|
"messages": [
|
|
{"role": "system", "content": "You are an expert on AI and prompt engineering."},
|
|
{"role": "user", "content": "Explain {title}"},
|
|
{"role": "assistant", "content": "{structured_content}"}
|
|
],
|
|
"metadata": {"source": "{url}", "quality_score": 0.92}
|
|
}
|
|
```
|
|
|
|
### Step 6: Log Export Job
|
|
```bash
|
|
uv run python scripts/exporter.py log --name "April 2026 Export" --type project_files --docs 45
|
|
```
|
|
|
|
## Scripts
|
|
|
|
| Command | Purpose |
|
|
|---------|---------|
|
|
| `exporter.py project` | Export as nested markdown files |
|
|
| `exporter.py finetuning` | Export as JSONL training dataset |
|
|
| `exporter.py index` | Generate INDEX.md table of contents |
|
|
| `exporter.py crossrefs` | Add cross-reference links |
|
|
| `exporter.py verify` | Verify export integrity |
|
|
| `exporter.py log` | Log export job to DB |
|
|
|
|
## Integration
|
|
|
|
| From | To |
|
|
|------|-----|
|
|
| quality-reviewer (approved) | → |
|
|
| → | Project knowledge / Fine-tuning dataset |
|