Files

Andrew Yim f215c11c32 feat(reference-curator): implement Python scripts + Gemini quality gate

Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 18:22:28 +09:00

3.3 KiB

Raw Blame History

Reference Discovery

Search and identify authoritative sources for reference materials. Validates source credibility, prioritizes by relevance, and outputs curated URL lists with metadata.

Trigger Keywords

"find references", "search documentation", "discover sources", "find authoritative materials", "research topic sources"

Source Priority Hierarchy

Tier	Source Type	Examples
Tier 1	Official documentation	docs.anthropic.com, docs.claude.com, platform.openai.com/docs
Tier 1	Engineering blogs (official)	anthropic.com/news, openai.com/blog
Tier 1	Official GitHub repos	github.com/anthropics/, github.com/openai/
Tier 2	Research papers	arxiv.org, papers with citations
Tier 2	Verified community guides	Cookbook examples, official tutorials
Tier 3	Community content	Blog posts, tutorials, Stack Overflow

Workflow

Step 1: Define Search Scope

Gather topic, target vendors, and freshness requirements from user input.

Step 2: Execute Web Search

Use WebSearch tool with targeted queries:

site:docs.anthropic.com {topic}
site:github.com/anthropics {topic}
site:arxiv.org {topic}

Step 3: Score and Validate Sources

Apply credibility scoring:

Domain credibility (0.10 - 0.40)
Freshness signals (0.10 - 0.20)
Relevance signals (0.15)

Step 4: Output URL Manifest

Save discovered URLs as a manifest JSON, deduplicating against the existing repository:

# Create manifest from discovered URLs
uv run python scripts/discover.py create-manifest --topic "prompt engineering" --output manifest.json < urls.json

# Deduplicate against existing DB
uv run python scripts/discover.py dedup --manifest manifest.json

# Register a new source
uv run python scripts/discover.py register-source \
  --name "Anthropic Docs" --type official_docs \
  --url "https://docs.anthropic.com" --tier tier1_official --vendor anthropic

# List registered sources
uv run python scripts/discover.py list-sources --vendor anthropic

Manifest format:

{
  "discovery_date": "2025-01-28T10:30:00",
  "topic": "prompt engineering",
  "total_urls": 15,
  "urls": [
    {
      "url": "https://docs.anthropic.com/en/docs/prompt-engineering",
      "title": "Prompt Engineering Guide",
      "credibility_tier": "tier1_official",
      "credibility_score": 0.85,
      "source_type": "official_docs",
      "vendor": "anthropic"
    }
  ]
}

Output

manifest.json → Handoff to 02-web-crawler-orchestrator
New sources registered in sources table via register-source

Deduplication

Before outputting:

Normalize URLs (remove trailing slashes, query params)
Check against existing documents table
Merge duplicates, keeping highest credibility score

Scripts

All scripts require the refcurator package. Run with uv run python from the skill directory.

Command	Purpose
`discover.py create-manifest`	Create manifest from URL entries JSON
`discover.py dedup`	Deduplicate manifest against DB
`discover.py register-source`	Register a new source
`discover.py list-sources`	List registered sources

Integration

From	To
WebSearch results	→ manifest.json
→ manifest.json	web-crawler-orchestrator
→ register-source	content-repository (sources table)

3.3 KiB Raw Blame History