our-claude-skills/custom-skills/35-seo-signal-validation/DESIGN.md

# Design Spec — `35-seo-signal-validation`

- **Status:** Draft for review
- **Date:** 2026-06-26
- **Author:** Andrew Yim (andrew.yim@ourdigital.org) + Claude Code
- **Genesis:** JHR josunhotel.com — SEMrush reported an organic "surge" attributed to "호텔" 16→3. Cross-checking GSC/GA4/live-SERP proved it a modeling artifact (real position ~12, ~5 clicks/mo; growth was all brand/seasonal). See workspace memory `feedback-semrush-serp-signal-validation`.
- **Related skills:** delegates to `20-seo-serp-analysis`, `21-seo-position-tracking`, `28-seo-knowledge-graph`.

---

## 1. Purpose

Given a `(term/intent, entity)` pair — and optionally a *claim* (a third-party tool's reported movement) or a *baseline* (a prior state to compare against) — return an **evidence-backed verdict** on whether SERP and Knowledge-Graph impact is **real**, **misattributed**, an **artifact**, or **unprovable with available data**.

The skill exists because OurDigital/clients repeatedly face *modeled* third-party signals (SEMrush/Ahrefs estimated organic traffic, position snapshots) that are easy to over-trust. This skill makes the validation cascade — measured → live → entity → attribution — a single repeatable procedure that ends in a defensible verdict and a client-safe narrative.

It generalizes the genesis case to **any term/intent and any entity** (a brand, a company, or a person), and to two additional jobs beyond refuting external claims: proving our own work's impact, and standalone "where do we really stand" checks.

## 2. Boundary — how this differs from neighbors

| Skill | Owns | This skill instead |
|---|---|---|
| `20-seo-serp-analysis` | What the SERP *looks like* (features, competitor positions, intent) | …calls it for the live-SERP layer |
| `21-seo-position-tracking` | Rank *over time*, change detection, visibility | …calls it for GSC-as-ground-truth |
| `28-seo-knowledge-graph` | Entity presence audit (KG panel, Wikidata, Naver) | …calls it for the entity layer |

None of the three **adjudicates the truth of a claimed cross-layer movement**. This skill is the *conductor*: signal/claim in → verdict + evidence ledger out. It duplicates none of their measurement logic; it sequences and synthesizes them.

## 3. Engine — the validation loop

A **cost-ordered evidence cascade** that short-circuits when a cheap layer is already decisive (this is exactly how the JHR "호텔" claim was refuted before any expensive step). The "loop" is the cascade, not a scheduler.

### 3.0 Pre-step — classify the entity (gates which layers are available)

- **First-party entity** — a site/property the user owns or has GSC/GA4 access to (e.g., JHR `sc-domain:josunhotel.com`, GA4 `258308769`). → **L1 measured ground truth available.**
- **Third-party entity** — a competitor brand or a person the user does NOT control. → **L1 unavailable**; rely on L2 + L3 + clearly-tiered third-party estimates; cap confidence lower and prefer INCONCLUSIVE over guessing.

The skill detects this from whether a verified GSC property / GA4 property is supplied or resolvable; if ambiguous, ask once.

### 3.1 L1 — Measured (first-party, native history) → delegates to `21-seo-position-tracking`

- **GSC** via `mcp__dda__gsc_fetch_performance`:
  - term query-level (exact match) AND site-wide, **recent vs prior** window.
  - report real avg position, clicks, impressions, CTR; **day-normalize** (compare periods often differ in calendar-day count).
  - note **~43% query-level anonymization** — query-sum ≠ aggregate; never treat the disclosed subset as the whole.
  - **query-clicks delta** (recent − prior) to name which terms actually moved (brand/seasonal vs the claimed term).
- **GA4** via `mcp__dda__ga4_run_report`: `Organic Search` sessions monthly trend (dimensions `yearMonth` + `sessionDefaultChannelGroup`, metric `sessions`); GA4 captures **all engines incl. Naver**, so use it to test whether a "surge" exceeds normal month-to-month variance.
- **Short-circuit:** if the claimed keyword has trivial clicks and a real position nowhere near the claim → **ARTIFACT**, stop (skip L2/L3 unless caller wants the full picture).

### 3.2 L2 — Live SERP (third-party measured, point-in-time) → delegates to `20-seo-serp-analysis`

- **Live geo-correct Google render** via `claude-in-chrome` (`navigate` → `read_page`/`computer`): force `gl`/`hl` + correct geo, `pws=0`; **decline precise-location prompts** (privacy). Confirm whether the domain actually holds the claimed position; capture the feature landscape (ads, local map-pack, PAA, knowledge panel, image/video) that explains *why* a brand site can't hold a head term.
- **Cheap rank spot-check** via `mcp__ourseo__check_serp(keyword, domain)` when a full render is unnecessary.
- **[KR market]** Naver SERP composition via `our research naver serp` (blog/cafe/지식iN/Smart Store/brand zone) — required for Korean entities since Semrush/Ahrefs don't model Naver.

### 3.3 L3 — Entity / Knowledge Graph (the differentiator) → delegates to `28-seo-knowledge-graph`

A real impact event should leave corroborating traces in the **entity layer**, not just a rank number. Five checks:

1. **Google KG API** entity match + `resultScore` — `mcp__ourseo__search_knowledge_graph(query)` (uses `GOOGLE_KG_API_KEY`).
2. **Wikidata** QID presence + key claims — **verify the QID against `Special:EntityData/{Q}.json` labels** before trusting it (bakes in the JHR false-match guard: Q109455878 = office tower ≠ hotel; Q490787 = Shinsegae Inc. ≠ Group).
3. **Knowledge Panel** presence/attributes on the live entity-name SERP (Chrome).
4. **sameAs** consistency on the entity's `Organization`/`Person` JSON-LD.
5. **[KR]** Naver 백과사전 / 지식iN presence.

`mcp__ourseo__monitor_brand` supplements with brand-mention/brand-SERP ownership signal.

### 3.4 L4 — Attribution synthesis → verdict

Cross-check: does the **measured delta (L1)** corroborate the **live reality (L2)**, and does the **entity layer (L3)** show consistent movement? The query-clicks delta names the true drivers. Output a verdict (§5) with an evidence ledger.

## 4. Entry modes (thin wrappers over the engine)

| Mode | Input contract | Engine use |
|---|---|---|
| **`adjudicate(claim)`** | `{term, entity, claim:{source, metric, from→to}}` e.g. `SEMrush: 호텔 pos 16→3, organic surge` | Full cascade; verdict confirms/refutes the claim |
| **`prove(baseline)`** | `{term, entity, change:{what, when}}` | Measured before/after from GSC/GA4 history; entity baseline = most recent existing Notion KG-audit archive — **if none exists, report current entity state only and mark the entity-layer delta INCONCLUSIVE** (the change pre-dates any captured baseline); live captured now |
| **`snapshot()`** | `{term, entity}` | Cascade with no claim; "where do we really stand" across all four layers |

All three call the *same* engine; they differ only in what they compare against.

## 5. Verdict logic

| Verdict | Condition |
|---|---|
| **CONFIRMED** | Measured + live + (where relevant) entity all corroborate movement attributable to the term/intent |
| **PARTIAL** | Real movement, but misattributed (e.g., growth is brand/seasonal, not the claimed head term) or only some layers agree |
| **ARTIFACT** | Modeling/snapshot artifact — measured + live reality don't support it (the JHR 호텔 case) |
| **INCONCLUSIVE** | Insufficient data (query anonymized, GSC lag, no entity baseline, third-party entity with no measured access) — names exactly what's missing + how to resolve |

**Confidence cap:** third-party entities (no L1) cannot reach CONFIRMED on traffic claims — at most PARTIAL, and ARTIFACT only when live+entity reality clearly contradicts the claim.

**Standing skepticism rules** (baked in from `feedback-semrush-serp-signal-validation`):
- Estimated organic traffic = **smoke-detector, not scale** (Σ est-volume × position-CTR curve).
- **Head-term over-fire**: one high-volume keyword caught at an estimated high rank inflates the whole modeled number.
- **KR Naver blind spot**: Semrush models Google only; misses a large share of Korean organic.
- **Single-geo/device snapshot** diverges from GSC's national average.
- **Data-trust hierarchy**: 1st-party measured (GA4/GSC) > 3rd-party measured (backlinks, crawled rank) > 3rd-party modeled (estimated traffic).

### Output of the verdict
- **Evidence ledger** — per layer: finding + its data-trust tier + whether it corroborates or contradicts the claim.
- **Client-safe narrative** — the defensible story (e.g., "summer brand/long-tail demand lifted impressions +18%, clicks modest" — NOT "ranked #3 for 호텔").

## 6. Output

- **Always:** inline structured report (verdict + ledger + narrative + "what would raise confidence").
- **Optional:** archive to Notion **Working with AI DB** (`data_source_id f8f19ede-32bd-43ac-9f60-0651f6f40afe`) via the **notion-writer script** (per global policy — never Notion MCP write tools). Properties follow the DB schema (Type=Memo/Research, Account Code, Topic=SEO, etc.).
- **Optional:** if the run surfaces a new *generalizable* gotcha, append a memory entry to the active workspace's memory dir.

## 7. Repo layout & conventions

```
35-seo-signal-validation/
  SKILL.md                 self-contained: classification, 4-layer cascade,
                           5 KG checks, 4-way verdict, skepticism rules, output
  DESIGN.md  PLAN.md       spec + plan (live with the skill; no new top-level dir)
  code/
    CLAUDE.md              code-environment notes (env, export→script flow)
    scripts/
      gsc_signal_delta.py  deterministic L1/L4 GSC delta + mover ranking
      test_gsc_signal_delta.py
      requirements.txt     (stdlib only)
```

Target environment: Claude Code only (no desktop/ variant — matches precedent 95/96). Registered in .claude-plugin/marketplace.json under ourdigital-seo.

**Triggers:** `validate serp signal`, `is this ranking real`, `prove SEO impact`, `SEMrush surge real?`, `signal validation`, `신호 검증`, `순위 변화 진짜?`, `오가닉 급증 검증`.

**Conventions honored:** no new output directories beyond this approved folder; Notion writes via notion-writer script only; never crawl/audit Marriott for JHR (sameAs reference only); KR deliverables in Korean, English internal notes OK.

## 8. Non-goals (YAGNI)

- No cron/scheduler and no snapshot DB (stateless, on-demand). A snapshot store + watchlist monitor is a documented **future option**, built only if proven needed.
- Does **not** replace the three instrument skills — it sequences them.
- Does **not** fabricate a verdict when data is thin — returns INCONCLUSIVE with a remediation list.
- Not a general SEO audit; scoped to validating a specific `(term, entity)` impact question.

## 9. Future options (explicitly out of v1)

- Lightweight snapshot store in the existing `dda` SQLite workspace to enable true over-time entity-layer deltas.
- Optional scheduled monitor over a `(term, entity)` watchlist that flags anomalies for `adjudicate`.
- Multi-engine claim intake (parse a pasted SEMrush/Ahrefs export directly).