our-claude-skills/custom-skills/35-seo-signal-validation/DESIGN.md

Design Spec — 35-seo-signal-validation

• Status: Draft for review
• Date: 2026-06-26
• Author: Andrew Yim (andrew.yim@ourdigital.org) + Claude Code
• Genesis: JHR josunhotel.com — SEMrush reported an organic "surge" attributed to "호텔" 16→3. Cross-checking GSC/GA4/live-SERP proved it a modeling artifact (real position ~12, ~5 clicks/mo; growth was all brand/seasonal). See workspace memory feedback-semrush-serp-signal-validation.
• Related skills: delegates to 20-seo-serp-analysis, 21-seo-position-tracking, 28-seo-knowledge-graph.

---

1. Purpose

Given a (term/intent, entity) pair — and optionally a claim (a third-party tool's reported movement) or a baseline (a prior state to compare against) — return an evidence-backed verdict on whether SERP and Knowledge-Graph impact is real, misattributed, an artifact, or unprovable with available data.

The skill exists because OurDigital/clients repeatedly face modeled third-party signals (SEMrush/Ahrefs estimated organic traffic, position snapshots) that are easy to over-trust. This skill makes the validation cascade — measured → live → entity → attribution — a single repeatable procedure that ends in a defensible verdict and a client-safe narrative.

It generalizes the genesis case to any term/intent and any entity (a brand, a company, or a person), and to two additional jobs beyond refuting external claims: proving our own work's impact, and standalone "where do we really stand" checks.

2. Boundary — how this differs from neighbors

Skill	Owns	This skill instead
20-seo-serp-analysis	What the SERP looks like (features, competitor positions, intent)	…calls it for the live-SERP layer
21-seo-position-tracking	Rank over time, change detection, visibility	…calls it for GSC-as-ground-truth
28-seo-knowledge-graph	Entity presence audit (KG panel, Wikidata, Naver)	…calls it for the entity layer

None of the three adjudicates the truth of a claimed cross-layer movement. This skill is the conductor: signal/claim in → verdict + evidence ledger out. It duplicates none of their measurement logic; it sequences and synthesizes them.

3. Engine — the validation loop

A cost-ordered evidence cascade that short-circuits when a cheap layer is already decisive (this is exactly how the JHR "호텔" claim was refuted before any expensive step). The "loop" is the cascade, not a scheduler.

3.0 Pre-step — classify the entity (gates which layers are available)

• First-party entity — a site/property the user owns or has GSC/GA4 access to (e.g., JHR sc-domain:josunhotel.com, GA4 258308769). → L1 measured ground truth available.
• Third-party entity — a competitor brand or a person the user does NOT control. → L1 unavailable; rely on L2 + L3 + clearly-tiered third-party estimates; cap confidence lower and prefer INCONCLUSIVE over guessing.

The skill detects this from whether a verified GSC property / GA4 property is supplied or resolvable; if ambiguous, ask once.

3.1 L1 — Measured (first-party, native history) → delegates to 21-seo-position-tracking

• GSC via mcp__dda__gsc_fetch_performance:
  • term query-level (exact match) AND site-wide, recent vs prior window.
  • report real avg position, clicks, impressions, CTR; day-normalize (compare periods often differ in calendar-day count).
  • note ~43% query-level anonymization — query-sum ≠ aggregate; never treat the disclosed subset as the whole.
  • query-clicks delta (recent − prior) to name which terms actually moved (brand/seasonal vs the claimed term).
• GA4 via mcp__dda__ga4_run_report: Organic Search sessions monthly trend (dimensions yearMonth + sessionDefaultChannelGroup, metric sessions); GA4 captures all engines incl. Naver, so use it to test whether a "surge" exceeds normal month-to-month variance.
• Short-circuit: if the claimed keyword has trivial clicks and a real position nowhere near the claim → ARTIFACT, stop (skip L2/L3 unless caller wants the full picture).

3.2 L2 — Live SERP (third-party measured, point-in-time) → delegates to 20-seo-serp-analysis

• Live geo-correct Google render via claude-in-chrome (navigate → read_page/computer): force gl/hl + correct geo, pws=0; decline precise-location prompts (privacy). Confirm whether the domain actually holds the claimed position; capture the feature landscape (ads, local map-pack, PAA, knowledge panel, image/video) that explains why a brand site can't hold a head term.
• Cheap rank spot-check via mcp__ourseo__check_serp(keyword, domain) when a full render is unnecessary.
• [KR market] Naver SERP composition via our research naver serp (blog/cafe/지식iN/Smart Store/brand zone) — required for Korean entities since Semrush/Ahrefs don't model Naver.

3.3 L3 — Entity / Knowledge Graph (the differentiator) → delegates to 28-seo-knowledge-graph

A real impact event should leave corroborating traces in the entity layer, not just a rank number. Five checks:

1. Google KG API entity match + resultScore — mcp__ourseo__search_knowledge_graph(query) (uses GOOGLE_KG_API_KEY).
2. Wikidata QID presence + key claims — verify the QID against Special:EntityData/{Q}.json labels before trusting it (bakes in the JHR false-match guard: Q109455878 = office tower ≠ hotel; Q490787 = Shinsegae Inc. ≠ Group).
3. Knowledge Panel presence/attributes on the live entity-name SERP (Chrome).
4. sameAs consistency on the entity's Organization/Person JSON-LD.
5. [KR] Naver 백과사전 / 지식iN presence.

mcp__ourseo__monitor_brand supplements with brand-mention/brand-SERP ownership signal.

3.4 L4 — Attribution synthesis → verdict

Cross-check: does the measured delta (L1) corroborate the live reality (L2), and does the entity layer (L3) show consistent movement? The query-clicks delta names the true drivers. Output a verdict (§5) with an evidence ledger.

4. Entry modes (thin wrappers over the engine)

Mode	Input contract	Engine use
adjudicate(claim)	{term, entity, claim:{source, metric, from→to}} e.g. SEMrush: 호텔 pos 16→3, organic surge	Full cascade; verdict confirms/refutes the claim
prove(baseline)	{term, entity, change:{what, when}}	Measured before/after from GSC/GA4 history; entity baseline = most recent existing Notion KG-audit archive — if none exists, report current entity state only and mark the entity-layer delta INCONCLUSIVE (the change pre-dates any captured baseline); live captured now
snapshot()	{term, entity}	Cascade with no claim; "where do we really stand" across all four layers

All three call the same engine; they differ only in what they compare against.

5. Verdict logic

Verdict	Condition
CONFIRMED	Measured + live + (where relevant) entity all corroborate movement attributable to the term/intent
PARTIAL	Real movement, but misattributed (e.g., growth is brand/seasonal, not the claimed head term) or only some layers agree
ARTIFACT	Modeling/snapshot artifact — measured + live reality don't support it (the JHR 호텔 case)
INCONCLUSIVE	Insufficient data (query anonymized, GSC lag, no entity baseline, third-party entity with no measured access) — names exactly what's missing + how to resolve

Confidence cap: third-party entities (no L1) cannot reach CONFIRMED on traffic claims — at most PARTIAL, and ARTIFACT only when live+entity reality clearly contradicts the claim.

Standing skepticism rules (baked in from feedback-semrush-serp-signal-validation):

• Estimated organic traffic = smoke-detector, not scale (Σ est-volume × position-CTR curve).
• Head-term over-fire: one high-volume keyword caught at an estimated high rank inflates the whole modeled number.
• KR Naver blind spot: Semrush models Google only; misses a large share of Korean organic.
• Single-geo/device snapshot diverges from GSC's national average.
• Data-trust hierarchy: 1st-party measured (GA4/GSC) > 3rd-party measured (backlinks, crawled rank) > 3rd-party modeled (estimated traffic).

Output of the verdict

• Evidence ledger — per layer: finding + its data-trust tier + whether it corroborates or contradicts the claim.
• Client-safe narrative — the defensible story (e.g., "summer brand/long-tail demand lifted impressions +18%, clicks modest" — NOT "ranked #3 for 호텔").

6. Output

• Always: inline structured report (verdict + ledger + narrative + "what would raise confidence").
• Optional: archive to Notion Working with AI DB (data_source_id f8f19ede-32bd-43ac-9f60-0651f6f40afe) via the notion-writer script (per global policy — never Notion MCP write tools). Properties follow the DB schema (Type=Memo/Research, Account Code, Topic=SEO, etc.).
• Optional: if the run surfaces a new generalizable gotcha, append a memory entry to the active workspace's memory dir.

7. Repo layout & conventions

35-seo-signal-validation/
  SKILL.md                 self-contained: classification, 4-layer cascade,
                           5 KG checks, 4-way verdict, skepticism rules, output
  DESIGN.md  PLAN.md       spec + plan (live with the skill; no new top-level dir)
  code/
    CLAUDE.md              code-environment notes (env, export→script flow)
    scripts/
      gsc_signal_delta.py  deterministic L1/L4 GSC delta + mover ranking
      test_gsc_signal_delta.py
      requirements.txt     (stdlib only)

Target environment: Claude Code only (no desktop/ variant — matches precedent 95/96). Registered in .claude-plugin/marketplace.json under ourdigital-seo.

Triggers: validate serp signal, is this ranking real, prove SEO impact, SEMrush surge real?, signal validation, 신호 검증, 순위 변화 진짜?, 오가닉 급증 검증.

Conventions honored: no new output directories beyond this approved folder; Notion writes via notion-writer script only; never crawl/audit Marriott for JHR (sameAs reference only); KR deliverables in Korean, English internal notes OK.

8. Non-goals (YAGNI)

• No cron/scheduler and no snapshot DB (stateless, on-demand). A snapshot store + watchlist monitor is a documented future option, built only if proven needed.
• Does not replace the three instrument skills — it sequences them.
• Does not fabricate a verdict when data is thin — returns INCONCLUSIVE with a remediation list.
• Not a general SEO audit; scoped to validating a specific (term, entity) impact question.

9. Future options (explicitly out of v1)

• Lightweight snapshot store in the existing dda SQLite workspace to enable true over-time entity-layer deltas.
• Optional scheduled monitor over a (term, entity) watchlist that flags anomalies for adjudicate.
• Multi-engine claim intake (parse a pasted SEMrush/Ahrefs export directly).