# Site-Extraction Methodology (Mode 1 — from an existing website) How to turn an existing site into a claims register, and why this mode is *easier* than Mode 2 (collected sources) but still must not blindly trust what it scrapes. Pair skill: extraction is `scripts/extract_site_claims.py`; the build engine and the QA gate are shared with Mode 2. See `source-to-schema-methodology.md` for Mode 2. ## Why a website is the easy case (and where it still bites) A published site has a **single source of truth** — the pages themselves — so there is little to reconcile. The risks are different from Mode 2: | Risk on an existing site | What it causes | Countermeasure | |---|---|---| | Trusting inferred meta as fact | wrong/old values shipped as schema | meta/OG seeded **PENDING**, never auto-shipped | | Existing JSON-LD is partial or stale | gaps, outdated facts | extracted as CONFIRMED but **spot-checked** at review | | Many near-identical pages | duplicate descriptions, bloated register | one entity per real thing; let Layer 4 catch dupes | | JS-rendered schema not in raw HTML | "nothing extracted" | use a rendered snapshot / live fetch, or fall to Mode 2 | ## The 5 steps ### 1. Choose the pages Pick the canonical page per entity (home, about/company, each property/location, key product/FAQ pages). One representative page per entity is enough to seed it; you don't need the whole crawl. ### 2. Extract Run the adapter on URLs, local `.html` files, or a directory (offline): ```bash python scripts/extract_site_claims.py https://site/ https://site/about --out site_claims python scripts/extract_site_claims.py ./snapshot/ --out site_claims # offline ``` It produces two tiers of claims: - **Existing JSON-LD → `CONFIRMED` (authority 1).** The site already published these facts about itself; flattened to dotted-path claims. - **`` / meta description / OpenGraph / `<html lang>` / canonical → `PENDING` (authority 2).** Inferred, not authoritative. These will **not** ship until confirmed. ### 3. Review the register (the critical human step) Open `site_claims/claims_register.csv`: - **Spot-check CONFIRMED rows** — extraction is faithful, but the site's own JSON-LD can be wrong/stale. Correct values; clear nothing silently. - **Confirm or drop PENDING rows** — set `status=CONFIRMED` only for facts you've verified; delete the rest. PENDING rows are excluded by the builder by design. - **Add what the page didn't expose** — telephone, full address, `geo`, `sameAs`, `priceRange`. The richest schema usually needs facts no single page renders. - Set `conflict=Y` on any value you're unsure about to keep it out until resolved. ### 4. Build ```bash python scripts/build_schema_drafts.py site_claims/claims_register.csv --out drafts_out ``` Unfilled slots are pruned; only CONFIRMED, non-conflicting claims become schema. Read `drafts_out/build_report.md` for everything excluded and why. ### 5. Validate (the gate) ```bash python ../16-seo-schema-validator/scripts/validate_schema.py \ drafts_out/schema_drafts_dataset.csv --out qa_out ``` Gate = **zero P0**. Fix P0, re-build, re-validate, then open client review against the report (not raw JSON). ## When NOT to use Mode 1 If the existing site **already has good, complete JSON-LD**, you don't need to regenerate it — **audit it in place** with `16-seo-schema-validator` Mode B (`validate_schema.py --live <URL>`). Mode 1 is for sites whose pages carry the *facts* but not yet the *structured data*, or whose schema needs a rebuild. ## entity_id convention The adapter assigns `prefix:slug` ids (`org:`, `site:`, `hotel:`, `dining:`, `page:`, …) derived from each node's `@id` fragment or page URL. Rename them to stable, human ids during review (e.g. `hotel:theshilla-seoul`) so re-runs and Mode 2 additions line up.