Andrew Yim
f215c11c32
Build the refcurator shared Python package and 7 CLI scripts that were previously specification-only. Add Gemini CLI as an independent pre-distillation quality evaluator, replacing the circular Claude-self-review pattern. Key changes: - shared/lib/src/refcurator/: 7-module package (config, db, models, utils, manifest, gemini) with PyMySQL + JSON file dual backend - 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer, exporter, pipeline — each with subcommands for data management - Gemini quality gate: evaluates raw content BEFORE distillation using 5 criteria (relevance, authority, completeness, freshness, distill_value) - Pipeline reordered: discovery → crawl → store → evaluate → distill → export - Bug fixes from Codex adversarial review: - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries - Exporter uses MAX(review_id) to prevent shipping stale approvals - Distiller updates existing rows on refactor instead of forking - Updated all 7 CLAUDE.md directives with real script references - install.sh updated with refcurator package install step 51/51 E2E tests passing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
104 lines
3.3 KiB
Markdown
104 lines
3.3 KiB
Markdown
# Reference Discovery
|
|
|
|
Search and identify authoritative sources for reference materials. Validates source credibility, prioritizes by relevance, and outputs curated URL lists with metadata.
|
|
|
|
## Trigger Keywords
|
|
"find references", "search documentation", "discover sources", "find authoritative materials", "research topic sources"
|
|
|
|
## Source Priority Hierarchy
|
|
|
|
| Tier | Source Type | Examples |
|
|
|------|-------------|----------|
|
|
| **Tier 1** | Official documentation | docs.anthropic.com, docs.claude.com, platform.openai.com/docs |
|
|
| **Tier 1** | Engineering blogs (official) | anthropic.com/news, openai.com/blog |
|
|
| **Tier 1** | Official GitHub repos | github.com/anthropics/*, github.com/openai/* |
|
|
| **Tier 2** | Research papers | arxiv.org, papers with citations |
|
|
| **Tier 2** | Verified community guides | Cookbook examples, official tutorials |
|
|
| **Tier 3** | Community content | Blog posts, tutorials, Stack Overflow |
|
|
|
|
## Workflow
|
|
|
|
### Step 1: Define Search Scope
|
|
Gather topic, target vendors, and freshness requirements from user input.
|
|
|
|
### Step 2: Execute Web Search
|
|
Use WebSearch tool with targeted queries:
|
|
```
|
|
site:docs.anthropic.com {topic}
|
|
site:github.com/anthropics {topic}
|
|
site:arxiv.org {topic}
|
|
```
|
|
|
|
### Step 3: Score and Validate Sources
|
|
Apply credibility scoring:
|
|
- Domain credibility (0.10 - 0.40)
|
|
- Freshness signals (0.10 - 0.20)
|
|
- Relevance signals (0.15)
|
|
|
|
### Step 4: Output URL Manifest
|
|
Save discovered URLs as a manifest JSON, deduplicating against the existing repository:
|
|
|
|
```bash
|
|
# Create manifest from discovered URLs
|
|
uv run python scripts/discover.py create-manifest --topic "prompt engineering" --output manifest.json < urls.json
|
|
|
|
# Deduplicate against existing DB
|
|
uv run python scripts/discover.py dedup --manifest manifest.json
|
|
|
|
# Register a new source
|
|
uv run python scripts/discover.py register-source \
|
|
--name "Anthropic Docs" --type official_docs \
|
|
--url "https://docs.anthropic.com" --tier tier1_official --vendor anthropic
|
|
|
|
# List registered sources
|
|
uv run python scripts/discover.py list-sources --vendor anthropic
|
|
```
|
|
|
|
Manifest format:
|
|
```json
|
|
{
|
|
"discovery_date": "2025-01-28T10:30:00",
|
|
"topic": "prompt engineering",
|
|
"total_urls": 15,
|
|
"urls": [
|
|
{
|
|
"url": "https://docs.anthropic.com/en/docs/prompt-engineering",
|
|
"title": "Prompt Engineering Guide",
|
|
"credibility_tier": "tier1_official",
|
|
"credibility_score": 0.85,
|
|
"source_type": "official_docs",
|
|
"vendor": "anthropic"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## Output
|
|
- `manifest.json` → Handoff to `02-web-crawler-orchestrator`
|
|
- New sources registered in `sources` table via `register-source`
|
|
|
|
## Deduplication
|
|
Before outputting:
|
|
- Normalize URLs (remove trailing slashes, query params)
|
|
- Check against existing `documents` table
|
|
- Merge duplicates, keeping highest credibility score
|
|
|
|
## Scripts
|
|
|
|
All scripts require the `refcurator` package. Run with `uv run python` from the skill directory.
|
|
|
|
| Command | Purpose |
|
|
|---------|---------|
|
|
| `discover.py create-manifest` | Create manifest from URL entries JSON |
|
|
| `discover.py dedup` | Deduplicate manifest against DB |
|
|
| `discover.py register-source` | Register a new source |
|
|
| `discover.py list-sources` | List registered sources |
|
|
|
|
## Integration
|
|
|
|
| From | To |
|
|
|------|-----|
|
|
| WebSearch results | → manifest.json |
|
|
| → manifest.json | web-crawler-orchestrator |
|
|
| → register-source | content-repository (sources table) |
|