ourdigital/our-claude-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
129 Commits 1 Branch 0 Tags
e1bd7996721c348399f012822e151cb340c81792
Commit Graph

3 Commits

Author SHA1 Message Date
Andrew Yim
382b55e9c8 feat(reference-curator): add --wayback-fallback flag + CAPTCHA/IP block detection 
- Add _is_captcha_response() to detect Google sorry page, 429, and
  "unusual traffic" markers in crawler responses
- Add --wayback-fallback CLI flag: auto-switches remaining URLs to
  web.archive.org/web/2024/ after 3 consecutive CAPTCHA blocks
- Add --delay CLI flag for polite crawling with configurable interval
- Add retry logic with exponential backoff on 429/5xx responses
- Document GOTCHA section in crawler CLAUDE.md with detection signals,
  Wayback detour solution, and prevention tips
- Add GOTCHA callout in pipeline orchestrator Stage 2 docs
- Record source: wayback in frontmatter for traceability

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-13 01:55:11 +09:00
Andrew Yim
b1c2dca080 feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration 
- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory
  alongside distilled content, with YAML frontmatter for traceability
- Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer),
  pure archival mode for raw documentation capture
- Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend:
  token-bucket rate limiting, semaphore concurrency, tenacity retries,
  full page metadata extraction (headings, links, schema, OG)
- New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests
  with progress tracking and resume support
- Update crawler selection: docs sites now default to seo-aiohttp
- Update crawl_config.yaml with seo-aiohttp routing rules
- Update pipeline orchestrator with flag resolution and skip paths
- Update README.md and USER-GUIDE.md with raw mode documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-12 23:26:50 +09:00
Andrew Yim
f215c11c32 feat(reference-curator): implement Python scripts + Gemini quality gate 
Build the refcurator shared Python package and 7 CLI scripts that were
previously specification-only. Add Gemini CLI as an independent pre-distillation
quality evaluator, replacing the circular Claude-self-review pattern.

Key changes:
- shared/lib/src/refcurator/: 7-module package (config, db, models, utils,
  manifest, gemini) with PyMySQL + JSON file dual backend
- 7 Click CLI scripts: discover, crawl_mgr, repo, distiller, reviewer,
  exporter, pipeline — each with subcommands for data management
- Gemini quality gate: evaluates raw content BEFORE distillation using
  5 criteria (relevance, authority, completeness, freshness, distill_value)
- Pipeline reordered: discovery → crawl → store → evaluate → distill → export
- Bug fixes from Codex adversarial review:
  - FileBackend now hard-fails on JOIN/aggregate/GROUP BY queries
  - Exporter uses MAX(review_id) to prevent shipping stale approvals
  - Distiller updates existing rows on refactor instead of forking
- Updated all 7 CLAUDE.md directives with real script references
- install.sh updated with refcurator package install step

51/51 E2E tests passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-12 18:22:28 +09:00