# Site-Extraction Methodology (Mode 1 — from an existing website)

How to turn an existing site into a claims register, and why this mode is *easier*
than Mode 2 (collected sources) but still must not blindly trust what it scrapes.

Pair skill: extraction is `scripts/extract_site_claims.py`; the build engine and the
QA gate are shared with Mode 2. See `source-to-schema-methodology.md` for Mode 2.

## Why a website is the easy case (and where it still bites)

A published site has a **single source of truth** — the pages themselves — so there is
little to reconcile. The risks are different from Mode 2:

| Risk on an existing site | What it causes | Countermeasure |
|---|---|---|
| Trusting inferred meta as fact | wrong/old values shipped as schema | meta/OG seeded **PENDING**, never auto-shipped |
| Existing JSON-LD is partial or stale | gaps, outdated facts | extracted as CONFIRMED but **spot-checked** at review |
| Many near-identical pages | duplicate descriptions, bloated register | one entity per real thing; let Layer 4 catch dupes |
| JS-rendered schema not in raw HTML | "nothing extracted" | use a rendered snapshot / live fetch, or fall to Mode 2 |

## The 5 steps

### 1. Choose the pages
Pick the canonical page per entity (home, about/company, each property/location, key
product/FAQ pages). One representative page per entity is enough to seed it; you don't
need the whole crawl.

### 2. Extract
Run the adapter on URLs, local `.html` files, or a directory (offline):
```bash
python scripts/extract_site_claims.py https://site/ https://site/about --out site_claims
python scripts/extract_site_claims.py ./snapshot/ --out site_claims   # offline
```
It produces two tiers of claims:
- **Existing JSON-LD → `CONFIRMED` (authority 1).** The site already published these
  facts about itself; flattened to dotted-path claims.
- **`<title>` / meta description / OpenGraph / `<html lang>` / canonical → `PENDING`
  (authority 2).** Inferred, not authoritative. These will **not** ship until confirmed.

### 3. Review the register (the critical human step)
Open `site_claims/claims_register.csv`:
- **Spot-check CONFIRMED rows** — extraction is faithful, but the site's own JSON-LD can
  be wrong/stale. Correct values; clear nothing silently.
- **Confirm or drop PENDING rows** — set `status=CONFIRMED` only for facts you've verified;
  delete the rest. PENDING rows are excluded by the builder by design.
- **Add what the page didn't expose** — telephone, full address, `geo`, `sameAs`,
  `priceRange`. The richest schema usually needs facts no single page renders.
- Set `conflict=Y` on any value you're unsure about to keep it out until resolved.

### 4. Build
```bash
python scripts/build_schema_drafts.py site_claims/claims_register.csv --out drafts_out
```
Unfilled slots are pruned; only CONFIRMED, non-conflicting claims become schema. Read
`drafts_out/build_report.md` for everything excluded and why.

### 5. Validate (the gate)
```bash
python ../16-seo-schema-validator/scripts/validate_schema.py \
    drafts_out/schema_drafts_dataset.csv --out qa_out
```
Gate = **zero P0**. Fix P0, re-build, re-validate, then open client review against the
report (not raw JSON).

## When NOT to use Mode 1

If the existing site **already has good, complete JSON-LD**, you don't need to regenerate
it — **audit it in place** with `16-seo-schema-validator` Mode B
(`validate_schema.py --live <URL>`). Mode 1 is for sites whose pages carry the *facts* but
not yet the *structured data*, or whose schema needs a rebuild.

## entity_id convention

The adapter assigns `prefix:slug` ids (`org:`, `site:`, `hotel:`, `dining:`, `page:`, …)
derived from each node's `@id` fragment or page URL. Rename them to stable, human ids
during review (e.g. `hotel:theshilla-seoul`) so re-runs and Mode 2 additions line up.
