From 18b39ef6eaea536045c19abace791b03398bf985 Mon Sep 17 00:00:00 2001 From: Andrew Yim Date: Fri, 26 Jun 2026 09:58:50 +0900 Subject: [PATCH] docs(skill): add 35-seo-signal-validation design spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New conductor skill that adjudicates whether a claimed SERP/Knowledge-Graph movement for a (term, entity) pair is real, misattributed, an artifact, or unprovable. Stateless, on-demand; delegates measurement to seo-serp-analysis, seo-position-tracking, and seo-knowledge-graph. Genesis: JHR "호텔 16→3" SEMrush claim refuted via GSC/GA4/live-SERP cross-check. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01KuT3W81t88QQFaxY2ruWv2 --- .../35-seo-signal-validation/DESIGN.md | 139 ++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 custom-skills/35-seo-signal-validation/DESIGN.md diff --git a/custom-skills/35-seo-signal-validation/DESIGN.md b/custom-skills/35-seo-signal-validation/DESIGN.md new file mode 100644 index 0000000..407f18a --- /dev/null +++ b/custom-skills/35-seo-signal-validation/DESIGN.md @@ -0,0 +1,139 @@ +# Design Spec — `35-seo-signal-validation` + +- **Status:** Draft for review +- **Date:** 2026-06-26 +- **Author:** Andrew Yim (andrew.yim@ourdigital.org) + Claude Code +- **Genesis:** JHR josunhotel.com — SEMrush reported an organic "surge" attributed to "호텔" 16→3. Cross-checking GSC/GA4/live-SERP proved it a modeling artifact (real position ~12, ~5 clicks/mo; growth was all brand/seasonal). See workspace memory `feedback-semrush-serp-signal-validation`. +- **Related skills:** delegates to `20-seo-serp-analysis`, `21-seo-position-tracking`, `28-seo-knowledge-graph`. + +--- + +## 1. Purpose + +Given a `(term/intent, entity)` pair — and optionally a *claim* (a third-party tool's reported movement) or a *baseline* (a prior state to compare against) — return an **evidence-backed verdict** on whether SERP and Knowledge-Graph impact is **real**, **misattributed**, an **artifact**, or **unprovable with available data**. + +The skill exists because OurDigital/clients repeatedly face *modeled* third-party signals (SEMrush/Ahrefs estimated organic traffic, position snapshots) that are easy to over-trust. This skill makes the validation cascade — measured → live → entity → attribution — a single repeatable procedure that ends in a defensible verdict and a client-safe narrative. + +It generalizes the genesis case to **any term/intent and any entity** (a brand, a company, or a person), and to two additional jobs beyond refuting external claims: proving our own work's impact, and standalone "where do we really stand" checks. + +## 2. Boundary — how this differs from neighbors + +| Skill | Owns | This skill instead | +|---|---|---| +| `20-seo-serp-analysis` | What the SERP *looks like* (features, competitor positions, intent) | …calls it for the live-SERP layer | +| `21-seo-position-tracking` | Rank *over time*, change detection, visibility | …calls it for GSC-as-ground-truth | +| `28-seo-knowledge-graph` | Entity presence audit (KG panel, Wikidata, Naver) | …calls it for the entity layer | + +None of the three **adjudicates the truth of a claimed cross-layer movement**. This skill is the *conductor*: signal/claim in → verdict + evidence ledger out. It duplicates none of their measurement logic; it sequences and synthesizes them. + +## 3. Engine — the validation loop + +A **cost-ordered evidence cascade** that short-circuits when a cheap layer is already decisive (this is exactly how the JHR "호텔" claim was refuted before any expensive step). The "loop" is the cascade, not a scheduler. + +### 3.0 Pre-step — classify the entity (gates which layers are available) + +- **First-party entity** — a site/property the user owns or has GSC/GA4 access to (e.g., JHR `sc-domain:josunhotel.com`, GA4 `258308769`). → **L1 measured ground truth available.** +- **Third-party entity** — a competitor brand or a person the user does NOT control. → **L1 unavailable**; rely on L2 + L3 + clearly-tiered third-party estimates; cap confidence lower and prefer INCONCLUSIVE over guessing. + +The skill detects this from whether a verified GSC property / GA4 property is supplied or resolvable; if ambiguous, ask once. + +### 3.1 L1 — Measured (first-party, native history) → delegates to `21-seo-position-tracking` + +- **GSC** via `mcp__dda__gsc_fetch_performance`: + - term query-level (exact match) AND site-wide, **recent vs prior** window. + - report real avg position, clicks, impressions, CTR; **day-normalize** (compare periods often differ in calendar-day count). + - note **~43% query-level anonymization** — query-sum ≠ aggregate; never treat the disclosed subset as the whole. + - **query-clicks delta** (recent − prior) to name which terms actually moved (brand/seasonal vs the claimed term). +- **GA4** via `mcp__dda__ga4_run_report`: `Organic Search` sessions monthly trend (dimensions `yearMonth` + `sessionDefaultChannelGroup`, metric `sessions`); GA4 captures **all engines incl. Naver**, so use it to test whether a "surge" exceeds normal month-to-month variance. +- **Short-circuit:** if the claimed keyword has trivial clicks and a real position nowhere near the claim → **ARTIFACT**, stop (skip L2/L3 unless caller wants the full picture). + +### 3.2 L2 — Live SERP (third-party measured, point-in-time) → delegates to `20-seo-serp-analysis` + +- **Live geo-correct Google render** via `claude-in-chrome` (`navigate` → `read_page`/`computer`): force `gl`/`hl` + correct geo, `pws=0`; **decline precise-location prompts** (privacy). Confirm whether the domain actually holds the claimed position; capture the feature landscape (ads, local map-pack, PAA, knowledge panel, image/video) that explains *why* a brand site can't hold a head term. +- **Cheap rank spot-check** via `mcp__ourseo__check_serp(keyword, domain)` when a full render is unnecessary. +- **[KR market]** Naver SERP composition via `our research naver serp` (blog/cafe/지식iN/Smart Store/brand zone) — required for Korean entities since Semrush/Ahrefs don't model Naver. + +### 3.3 L3 — Entity / Knowledge Graph (the differentiator) → delegates to `28-seo-knowledge-graph` + +A real impact event should leave corroborating traces in the **entity layer**, not just a rank number. Five checks: + +1. **Google KG API** entity match + `resultScore` — `mcp__ourseo__search_knowledge_graph(query)` (uses `GOOGLE_KG_API_KEY`). +2. **Wikidata** QID presence + key claims — **verify the QID against `Special:EntityData/{Q}.json` labels** before trusting it (bakes in the JHR false-match guard: Q109455878 = office tower ≠ hotel; Q490787 = Shinsegae Inc. ≠ Group). +3. **Knowledge Panel** presence/attributes on the live entity-name SERP (Chrome). +4. **sameAs** consistency on the entity's `Organization`/`Person` JSON-LD. +5. **[KR]** Naver 백과사전 / 지식iN presence. + +`mcp__ourseo__monitor_brand` supplements with brand-mention/brand-SERP ownership signal. + +### 3.4 L4 — Attribution synthesis → verdict + +Cross-check: does the **measured delta (L1)** corroborate the **live reality (L2)**, and does the **entity layer (L3)** show consistent movement? The query-clicks delta names the true drivers. Output a verdict (§5) with an evidence ledger. + +## 4. Entry modes (thin wrappers over the engine) + +| Mode | Input contract | Engine use | +|---|---|---| +| **`adjudicate(claim)`** | `{term, entity, claim:{source, metric, from→to}}` e.g. `SEMrush: 호텔 pos 16→3, organic surge` | Full cascade; verdict confirms/refutes the claim | +| **`prove(baseline)`** | `{term, entity, change:{what, when}}` | Measured before/after from GSC/GA4 history; entity baseline = most recent existing Notion KG-audit archive — **if none exists, report current entity state only and mark the entity-layer delta INCONCLUSIVE** (the change pre-dates any captured baseline); live captured now | +| **`snapshot()`** | `{term, entity}` | Cascade with no claim; "where do we really stand" across all four layers | + +All three call the *same* engine; they differ only in what they compare against. + +## 5. Verdict logic + +| Verdict | Condition | +|---|---| +| **CONFIRMED** | Measured + live + (where relevant) entity all corroborate movement attributable to the term/intent | +| **PARTIAL** | Real movement, but misattributed (e.g., growth is brand/seasonal, not the claimed head term) or only some layers agree | +| **ARTIFACT** | Modeling/snapshot artifact — measured + live reality don't support it (the JHR 호텔 case) | +| **INCONCLUSIVE** | Insufficient data (query anonymized, GSC lag, no entity baseline, third-party entity with no measured access) — names exactly what's missing + how to resolve | + +**Confidence cap:** third-party entities (no L1) cannot reach CONFIRMED on traffic claims — at most PARTIAL, and ARTIFACT only when live+entity reality clearly contradicts the claim. + +**Standing skepticism rules** (baked in from `feedback-semrush-serp-signal-validation`): +- Estimated organic traffic = **smoke-detector, not scale** (Σ est-volume × position-CTR curve). +- **Head-term over-fire**: one high-volume keyword caught at an estimated high rank inflates the whole modeled number. +- **KR Naver blind spot**: Semrush models Google only; misses a large share of Korean organic. +- **Single-geo/device snapshot** diverges from GSC's national average. +- **Data-trust hierarchy**: 1st-party measured (GA4/GSC) > 3rd-party measured (backlinks, crawled rank) > 3rd-party modeled (estimated traffic). + +### Output of the verdict +- **Evidence ledger** — per layer: finding + its data-trust tier + whether it corroborates or contradicts the claim. +- **Client-safe narrative** — the defensible story (e.g., "summer brand/long-tail demand lifted impressions +18%, clicks modest" — NOT "ranked #3 for 호텔"). + +## 6. Output + +- **Always:** inline structured report (verdict + ledger + narrative + "what would raise confidence"). +- **Optional:** archive to Notion **Working with AI DB** (`data_source_id f8f19ede-32bd-43ac-9f60-0651f6f40afe`) via the **notion-writer script** (per global policy — never Notion MCP write tools). Properties follow the DB schema (Type=Memo/Research, Account Code, Topic=SEO, etc.). +- **Optional:** if the run surfaces a new *generalizable* gotcha, append a memory entry to the active workspace's memory dir. + +## 7. Repo layout & conventions + +``` +35-seo-signal-validation/ + SKILL.md frontmatter (name/description/triggers), mode dispatch, + the cascade, short-circuit + verdict synthesis + DESIGN.md this spec (lives with the skill; no new top-level dir) + references/ + evidence-cascade.md L1–L4 concrete tool calls + short-circuit rules + kg-impact-checks.md the 5 entity checks incl. Wikidata QID verification + verdict-taxonomy.md conditions + ledger format + narrative template + backend-selection.md pointer to 20/21's existing backend tables (no dup) +``` + +**Triggers:** `validate serp signal`, `is this ranking real`, `prove SEO impact`, `SEMrush surge real?`, `signal validation`, `신호 검증`, `순위 변화 진짜?`, `오가닉 급증 검증`. + +**Conventions honored:** no new output directories beyond this approved folder; Notion writes via notion-writer script only; never crawl/audit Marriott for JHR (sameAs reference only); KR deliverables in Korean, English internal notes OK. + +## 8. Non-goals (YAGNI) + +- No cron/scheduler and no snapshot DB (stateless, on-demand). A snapshot store + watchlist monitor is a documented **future option**, built only if proven needed. +- Does **not** replace the three instrument skills — it sequences them. +- Does **not** fabricate a verdict when data is thin — returns INCONCLUSIVE with a remediation list. +- Not a general SEO audit; scoped to validating a specific `(term, entity)` impact question. + +## 9. Future options (explicitly out of v1) + +- Lightweight snapshot store in the existing `dda` SQLite workspace to enable true over-time entity-layer deltas. +- Optional scheduled monitor over a `(term, entity)` watchlist that flags anomalies for `adjudicate`. +- Multi-engine claim intake (parse a pasted SEMrush/Ahrefs export directly).