Files

Andrew Yim 1706a820fe feat(seo-schema-generator): merge site-extraction + source-to-schema into one skill

Unify the two schema-generation scenarios into a single slot-17 skill, both
feeding one claims register -> build -> validate(16) pipeline:

- Mode 1 (existing site): NEW scripts/extract_site_claims.py turns URLs / local
  HTML / a directory into a claims register. Existing JSON-LD -> CONFIRMED;
  title/OpenGraph -> PENDING (never auto-shipped). + site-extraction-methodology.md
  and bundled fixtures/site/ demo pages.
- Mode 2 (not-yet-published site): land the source-to-schema engine
  (build_schema_drafts.py, type_templates.json, claims/source registers, 3 refs,
  sample_claims.csv) from the Desktop builder.
- Rewrite SKILL.md (v2.0) around the two-mode framing; the claims register is the
  shared pivot. Only CONFIRMED, non-conflicting claims become schema; unfilled
  template slots are pruned, never emitted as placeholders.
- Retire the old template-fill generator (code/ + desktop/); update root CLAUDE.md.

Self-tested both chains end-to-end: Mode 2 sample -> build -> validate PASS (P0=0);
Mode 1 fixtures -> extract -> build -> validate PASS (P0=0), JSON-LD round-trips with
nested address intact. Fixed two adapter bugs (nested node promotion; relative-path URI).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-28 00:38:40 +09:00

3.8 KiB

Raw Permalink Blame History

Site-Extraction Methodology (Mode 1 — from an existing website)

How to turn an existing site into a claims register, and why this mode is easier than Mode 2 (collected sources) but still must not blindly trust what it scrapes.

Pair skill: extraction is scripts/extract_site_claims.py; the build engine and the QA gate are shared with Mode 2. See source-to-schema-methodology.md for Mode 2.

Why a website is the easy case (and where it still bites)

A published site has a single source of truth — the pages themselves — so there is little to reconcile. The risks are different from Mode 2:

Risk on an existing site	What it causes	Countermeasure
Trusting inferred meta as fact	wrong/old values shipped as schema	meta/OG seeded PENDING, never auto-shipped
Existing JSON-LD is partial or stale	gaps, outdated facts	extracted as CONFIRMED but spot-checked at review
Many near-identical pages	duplicate descriptions, bloated register	one entity per real thing; let Layer 4 catch dupes
JS-rendered schema not in raw HTML	"nothing extracted"	use a rendered snapshot / live fetch, or fall to Mode 2

The 5 steps

1. Choose the pages

Pick the canonical page per entity (home, about/company, each property/location, key product/FAQ pages). One representative page per entity is enough to seed it; you don't need the whole crawl.

2. Extract

Run the adapter on URLs, local .html files, or a directory (offline):

python scripts/extract_site_claims.py https://site/ https://site/about --out site_claims
python scripts/extract_site_claims.py ./snapshot/ --out site_claims   # offline

It produces two tiers of claims:

Existing JSON-LD → CONFIRMED (authority 1). The site already published these facts about itself; flattened to dotted-path claims.
<title> / meta description / OpenGraph / <html lang> / canonical → PENDING (authority 2). Inferred, not authoritative. These will not ship until confirmed.

3. Review the register (the critical human step)

Open site_claims/claims_register.csv:

Spot-check CONFIRMED rows — extraction is faithful, but the site's own JSON-LD can be wrong/stale. Correct values; clear nothing silently.
Confirm or drop PENDING rows — set status=CONFIRMED only for facts you've verified; delete the rest. PENDING rows are excluded by the builder by design.
Add what the page didn't expose — telephone, full address, geo, sameAs, priceRange. The richest schema usually needs facts no single page renders.
Set conflict=Y on any value you're unsure about to keep it out until resolved.

4. Build

python scripts/build_schema_drafts.py site_claims/claims_register.csv --out drafts_out

Unfilled slots are pruned; only CONFIRMED, non-conflicting claims become schema. Read drafts_out/build_report.md for everything excluded and why.

5. Validate (the gate)

python ../16-seo-schema-validator/scripts/validate_schema.py \
    drafts_out/schema_drafts_dataset.csv --out qa_out

Gate = zero P0. Fix P0, re-build, re-validate, then open client review against the report (not raw JSON).

When NOT to use Mode 1

If the existing site already has good, complete JSON-LD, you don't need to regenerate it — audit it in place with 16-seo-schema-validator Mode B (validate_schema.py --live <URL>). Mode 1 is for sites whose pages carry the facts but not yet the structured data, or whose schema needs a rebuild.

entity_id convention

The adapter assigns prefix:slug ids (org:, site:, hotel:, dining:, page:, …) derived from each node's @id fragment or page URL. Rename them to stable, human ids during review (e.g. hotel:theshilla-seoul) so re-runs and Mode 2 additions line up.

3.8 KiB Raw Permalink Blame History