Andrew Yim
1706a820fe
Unify the two schema-generation scenarios into a single slot-17 skill, both feeding one claims register -> build -> validate(16) pipeline: - Mode 1 (existing site): NEW scripts/extract_site_claims.py turns URLs / local HTML / a directory into a claims register. Existing JSON-LD -> CONFIRMED; title/OpenGraph -> PENDING (never auto-shipped). + site-extraction-methodology.md and bundled fixtures/site/ demo pages. - Mode 2 (not-yet-published site): land the source-to-schema engine (build_schema_drafts.py, type_templates.json, claims/source registers, 3 refs, sample_claims.csv) from the Desktop builder. - Rewrite SKILL.md (v2.0) around the two-mode framing; the claims register is the shared pivot. Only CONFIRMED, non-conflicting claims become schema; unfilled template slots are pruned, never emitted as placeholders. - Retire the old template-fill generator (code/ + desktop/); update root CLAUDE.md. Self-tested both chains end-to-end: Mode 2 sample -> build -> validate PASS (P0=0); Mode 1 fixtures -> extract -> build -> validate PASS (P0=0), JSON-LD round-trips with nested address intact. Fixed two adapter bugs (nested node promotion; relative-path URI). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
77 lines
3.8 KiB
Markdown
77 lines
3.8 KiB
Markdown
# Site-Extraction Methodology (Mode 1 — from an existing website)
|
|
|
|
How to turn an existing site into a claims register, and why this mode is *easier*
|
|
than Mode 2 (collected sources) but still must not blindly trust what it scrapes.
|
|
|
|
Pair skill: extraction is `scripts/extract_site_claims.py`; the build engine and the
|
|
QA gate are shared with Mode 2. See `source-to-schema-methodology.md` for Mode 2.
|
|
|
|
## Why a website is the easy case (and where it still bites)
|
|
|
|
A published site has a **single source of truth** — the pages themselves — so there is
|
|
little to reconcile. The risks are different from Mode 2:
|
|
|
|
| Risk on an existing site | What it causes | Countermeasure |
|
|
|---|---|---|
|
|
| Trusting inferred meta as fact | wrong/old values shipped as schema | meta/OG seeded **PENDING**, never auto-shipped |
|
|
| Existing JSON-LD is partial or stale | gaps, outdated facts | extracted as CONFIRMED but **spot-checked** at review |
|
|
| Many near-identical pages | duplicate descriptions, bloated register | one entity per real thing; let Layer 4 catch dupes |
|
|
| JS-rendered schema not in raw HTML | "nothing extracted" | use a rendered snapshot / live fetch, or fall to Mode 2 |
|
|
|
|
## The 5 steps
|
|
|
|
### 1. Choose the pages
|
|
Pick the canonical page per entity (home, about/company, each property/location, key
|
|
product/FAQ pages). One representative page per entity is enough to seed it; you don't
|
|
need the whole crawl.
|
|
|
|
### 2. Extract
|
|
Run the adapter on URLs, local `.html` files, or a directory (offline):
|
|
```bash
|
|
python scripts/extract_site_claims.py https://site/ https://site/about --out site_claims
|
|
python scripts/extract_site_claims.py ./snapshot/ --out site_claims # offline
|
|
```
|
|
It produces two tiers of claims:
|
|
- **Existing JSON-LD → `CONFIRMED` (authority 1).** The site already published these
|
|
facts about itself; flattened to dotted-path claims.
|
|
- **`<title>` / meta description / OpenGraph / `<html lang>` / canonical → `PENDING`
|
|
(authority 2).** Inferred, not authoritative. These will **not** ship until confirmed.
|
|
|
|
### 3. Review the register (the critical human step)
|
|
Open `site_claims/claims_register.csv`:
|
|
- **Spot-check CONFIRMED rows** — extraction is faithful, but the site's own JSON-LD can
|
|
be wrong/stale. Correct values; clear nothing silently.
|
|
- **Confirm or drop PENDING rows** — set `status=CONFIRMED` only for facts you've verified;
|
|
delete the rest. PENDING rows are excluded by the builder by design.
|
|
- **Add what the page didn't expose** — telephone, full address, `geo`, `sameAs`,
|
|
`priceRange`. The richest schema usually needs facts no single page renders.
|
|
- Set `conflict=Y` on any value you're unsure about to keep it out until resolved.
|
|
|
|
### 4. Build
|
|
```bash
|
|
python scripts/build_schema_drafts.py site_claims/claims_register.csv --out drafts_out
|
|
```
|
|
Unfilled slots are pruned; only CONFIRMED, non-conflicting claims become schema. Read
|
|
`drafts_out/build_report.md` for everything excluded and why.
|
|
|
|
### 5. Validate (the gate)
|
|
```bash
|
|
python ../16-seo-schema-validator/scripts/validate_schema.py \
|
|
drafts_out/schema_drafts_dataset.csv --out qa_out
|
|
```
|
|
Gate = **zero P0**. Fix P0, re-build, re-validate, then open client review against the
|
|
report (not raw JSON).
|
|
|
|
## When NOT to use Mode 1
|
|
|
|
If the existing site **already has good, complete JSON-LD**, you don't need to regenerate
|
|
it — **audit it in place** with `16-seo-schema-validator` Mode B
|
|
(`validate_schema.py --live <URL>`). Mode 1 is for sites whose pages carry the *facts* but
|
|
not yet the *structured data*, or whose schema needs a rebuild.
|
|
|
|
## entity_id convention
|
|
|
|
The adapter assigns `prefix:slug` ids (`org:`, `site:`, `hotel:`, `dining:`, `page:`, …)
|
|
derived from each node's `@id` fragment or page URL. Rename them to stable, human ids
|
|
during review (e.g. `hotel:theshilla-seoul`) so re-runs and Mode 2 additions line up.
|