Files

Andrew Yim f215c11c32 feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 18:22:28 +09:00

2.4 KiB

Raw Blame History

Markdown Exporter

Exports approved reference content as structured markdown files for project knowledge or fine-tuning datasets.

Trigger Keywords

"export references", "generate project files", "create markdown output", "export for fine-tuning", "build knowledge base"

Export Types

Type	Format	Use Case
`project`	Nested markdown	Claude Projects knowledge
`finetuning`	JSONL	Model fine-tuning dataset

Workflow

Step 1: Export Project Files

uv run python scripts/exporter.py project \
  --output ~/reference-library/exports/ \
  --min-score 0.80 \
  --structure nested_by_topic

Output structure:

exports/
├── INDEX.md
├── prompt-engineering/
│   ├── _index.md
│   ├── 00-chain-of-thought.md
│   └── 01-few-shot-prompting.md
└── claude-models/
    ├── _index.md
    └── 00-model-comparison.md

Step 2: Generate INDEX

uv run python scripts/exporter.py index --output ~/reference-library/exports/INDEX.md

Step 3: Add Cross-References

uv run python scripts/exporter.py crossrefs --input ~/reference-library/exports/

Step 4: Verify Export

uv run python scripts/exporter.py verify --path ~/reference-library/exports/

Step 5: Fine-tuning Export (Optional)

uv run python scripts/exporter.py finetuning \
  --output ~/reference-library/exports/training.jsonl \
  --max-tokens 4096

JSONL format:

{
  "messages": [
    {"role": "system", "content": "You are an expert on AI and prompt engineering."},
    {"role": "user", "content": "Explain {title}"},
    {"role": "assistant", "content": "{structured_content}"}
  ],
  "metadata": {"source": "{url}", "quality_score": 0.92}
}

Step 6: Log Export Job

uv run python scripts/exporter.py log --name "April 2026 Export" --type project_files --docs 45

Scripts

Command	Purpose
`exporter.py project`	Export as nested markdown files
`exporter.py finetuning`	Export as JSONL training dataset
`exporter.py index`	Generate INDEX.md table of contents
`exporter.py crossrefs`	Add cross-reference links
`exporter.py verify`	Verify export integrity
`exporter.py log`	Log export job to DB

Integration

From	To
quality-reviewer (approved)	→
→	Project knowledge / Fine-tuning dataset

2.4 KiB Raw Blame History