feat(seo-schema-generator): merge site-extraction + source-to-schema into one skill
Unify the two schema-generation scenarios into a single slot-17 skill, both feeding one claims register -> build -> validate(16) pipeline: - Mode 1 (existing site): NEW scripts/extract_site_claims.py turns URLs / local HTML / a directory into a claims register. Existing JSON-LD -> CONFIRMED; title/OpenGraph -> PENDING (never auto-shipped). + site-extraction-methodology.md and bundled fixtures/site/ demo pages. - Mode 2 (not-yet-published site): land the source-to-schema engine (build_schema_drafts.py, type_templates.json, claims/source registers, 3 refs, sample_claims.csv) from the Desktop builder. - Rewrite SKILL.md (v2.0) around the two-mode framing; the claims register is the shared pivot. Only CONFIRMED, non-conflicting claims become schema; unfilled template slots are pruned, never emitted as placeholders. - Retire the old template-fill generator (code/ + desktop/); update root CLAUDE.md. Self-tested both chains end-to-end: Mode 2 sample -> build -> validate PASS (P0=0); Mode 1 fixtures -> extract -> build -> validate PASS (P0=0), JSON-LD round-trips with nested address intact. Fixed two adapter bugs (nested node promotion; relative-path URI). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# Site-Extraction Methodology (Mode 1 — from an existing website)
|
||||
|
||||
How to turn an existing site into a claims register, and why this mode is *easier*
|
||||
than Mode 2 (collected sources) but still must not blindly trust what it scrapes.
|
||||
|
||||
Pair skill: extraction is `scripts/extract_site_claims.py`; the build engine and the
|
||||
QA gate are shared with Mode 2. See `source-to-schema-methodology.md` for Mode 2.
|
||||
|
||||
## Why a website is the easy case (and where it still bites)
|
||||
|
||||
A published site has a **single source of truth** — the pages themselves — so there is
|
||||
little to reconcile. The risks are different from Mode 2:
|
||||
|
||||
| Risk on an existing site | What it causes | Countermeasure |
|
||||
|---|---|---|
|
||||
| Trusting inferred meta as fact | wrong/old values shipped as schema | meta/OG seeded **PENDING**, never auto-shipped |
|
||||
| Existing JSON-LD is partial or stale | gaps, outdated facts | extracted as CONFIRMED but **spot-checked** at review |
|
||||
| Many near-identical pages | duplicate descriptions, bloated register | one entity per real thing; let Layer 4 catch dupes |
|
||||
| JS-rendered schema not in raw HTML | "nothing extracted" | use a rendered snapshot / live fetch, or fall to Mode 2 |
|
||||
|
||||
## The 5 steps
|
||||
|
||||
### 1. Choose the pages
|
||||
Pick the canonical page per entity (home, about/company, each property/location, key
|
||||
product/FAQ pages). One representative page per entity is enough to seed it; you don't
|
||||
need the whole crawl.
|
||||
|
||||
### 2. Extract
|
||||
Run the adapter on URLs, local `.html` files, or a directory (offline):
|
||||
```bash
|
||||
python scripts/extract_site_claims.py https://site/ https://site/about --out site_claims
|
||||
python scripts/extract_site_claims.py ./snapshot/ --out site_claims # offline
|
||||
```
|
||||
It produces two tiers of claims:
|
||||
- **Existing JSON-LD → `CONFIRMED` (authority 1).** The site already published these
|
||||
facts about itself; flattened to dotted-path claims.
|
||||
- **`<title>` / meta description / OpenGraph / `<html lang>` / canonical → `PENDING`
|
||||
(authority 2).** Inferred, not authoritative. These will **not** ship until confirmed.
|
||||
|
||||
### 3. Review the register (the critical human step)
|
||||
Open `site_claims/claims_register.csv`:
|
||||
- **Spot-check CONFIRMED rows** — extraction is faithful, but the site's own JSON-LD can
|
||||
be wrong/stale. Correct values; clear nothing silently.
|
||||
- **Confirm or drop PENDING rows** — set `status=CONFIRMED` only for facts you've verified;
|
||||
delete the rest. PENDING rows are excluded by the builder by design.
|
||||
- **Add what the page didn't expose** — telephone, full address, `geo`, `sameAs`,
|
||||
`priceRange`. The richest schema usually needs facts no single page renders.
|
||||
- Set `conflict=Y` on any value you're unsure about to keep it out until resolved.
|
||||
|
||||
### 4. Build
|
||||
```bash
|
||||
python scripts/build_schema_drafts.py site_claims/claims_register.csv --out drafts_out
|
||||
```
|
||||
Unfilled slots are pruned; only CONFIRMED, non-conflicting claims become schema. Read
|
||||
`drafts_out/build_report.md` for everything excluded and why.
|
||||
|
||||
### 5. Validate (the gate)
|
||||
```bash
|
||||
python ../16-seo-schema-validator/scripts/validate_schema.py \
|
||||
drafts_out/schema_drafts_dataset.csv --out qa_out
|
||||
```
|
||||
Gate = **zero P0**. Fix P0, re-build, re-validate, then open client review against the
|
||||
report (not raw JSON).
|
||||
|
||||
## When NOT to use Mode 1
|
||||
|
||||
If the existing site **already has good, complete JSON-LD**, you don't need to regenerate
|
||||
it — **audit it in place** with `16-seo-schema-validator` Mode B
|
||||
(`validate_schema.py --live <URL>`). Mode 1 is for sites whose pages carry the *facts* but
|
||||
not yet the *structured data*, or whose schema needs a rebuild.
|
||||
|
||||
## entity_id convention
|
||||
|
||||
The adapter assigns `prefix:slug` ids (`org:`, `site:`, `hotel:`, `dining:`, `page:`, …)
|
||||
derived from each node's `@id` fragment or page URL. Rename them to stable, human ids
|
||||
during review (e.g. `hotel:theshilla-seoul`) so re-runs and Mode 2 additions line up.
|
||||
Reference in New Issue
Block a user