docs(skill): add 35-seo-signal-validation design spec
New conductor skill that adjudicates whether a claimed SERP/Knowledge-Graph movement for a (term, entity) pair is real, misattributed, an artifact, or unprovable. Stateless, on-demand; delegates measurement to seo-serp-analysis, seo-position-tracking, and seo-knowledge-graph. Genesis: JHR "호텔 16→3" SEMrush claim refuted via GSC/GA4/live-SERP cross-check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuT3W81t88QQFaxY2ruWv2
This commit is contained in:
139
custom-skills/35-seo-signal-validation/DESIGN.md
Normal file
139
custom-skills/35-seo-signal-validation/DESIGN.md
Normal file
@@ -0,0 +1,139 @@
|
||||
# Design Spec — `35-seo-signal-validation`
|
||||
|
||||
- **Status:** Draft for review
|
||||
- **Date:** 2026-06-26
|
||||
- **Author:** Andrew Yim (andrew.yim@ourdigital.org) + Claude Code
|
||||
- **Genesis:** JHR josunhotel.com — SEMrush reported an organic "surge" attributed to "호텔" 16→3. Cross-checking GSC/GA4/live-SERP proved it a modeling artifact (real position ~12, ~5 clicks/mo; growth was all brand/seasonal). See workspace memory `feedback-semrush-serp-signal-validation`.
|
||||
- **Related skills:** delegates to `20-seo-serp-analysis`, `21-seo-position-tracking`, `28-seo-knowledge-graph`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Given a `(term/intent, entity)` pair — and optionally a *claim* (a third-party tool's reported movement) or a *baseline* (a prior state to compare against) — return an **evidence-backed verdict** on whether SERP and Knowledge-Graph impact is **real**, **misattributed**, an **artifact**, or **unprovable with available data**.
|
||||
|
||||
The skill exists because OurDigital/clients repeatedly face *modeled* third-party signals (SEMrush/Ahrefs estimated organic traffic, position snapshots) that are easy to over-trust. This skill makes the validation cascade — measured → live → entity → attribution — a single repeatable procedure that ends in a defensible verdict and a client-safe narrative.
|
||||
|
||||
It generalizes the genesis case to **any term/intent and any entity** (a brand, a company, or a person), and to two additional jobs beyond refuting external claims: proving our own work's impact, and standalone "where do we really stand" checks.
|
||||
|
||||
## 2. Boundary — how this differs from neighbors
|
||||
|
||||
| Skill | Owns | This skill instead |
|
||||
|---|---|---|
|
||||
| `20-seo-serp-analysis` | What the SERP *looks like* (features, competitor positions, intent) | …calls it for the live-SERP layer |
|
||||
| `21-seo-position-tracking` | Rank *over time*, change detection, visibility | …calls it for GSC-as-ground-truth |
|
||||
| `28-seo-knowledge-graph` | Entity presence audit (KG panel, Wikidata, Naver) | …calls it for the entity layer |
|
||||
|
||||
None of the three **adjudicates the truth of a claimed cross-layer movement**. This skill is the *conductor*: signal/claim in → verdict + evidence ledger out. It duplicates none of their measurement logic; it sequences and synthesizes them.
|
||||
|
||||
## 3. Engine — the validation loop
|
||||
|
||||
A **cost-ordered evidence cascade** that short-circuits when a cheap layer is already decisive (this is exactly how the JHR "호텔" claim was refuted before any expensive step). The "loop" is the cascade, not a scheduler.
|
||||
|
||||
### 3.0 Pre-step — classify the entity (gates which layers are available)
|
||||
|
||||
- **First-party entity** — a site/property the user owns or has GSC/GA4 access to (e.g., JHR `sc-domain:josunhotel.com`, GA4 `258308769`). → **L1 measured ground truth available.**
|
||||
- **Third-party entity** — a competitor brand or a person the user does NOT control. → **L1 unavailable**; rely on L2 + L3 + clearly-tiered third-party estimates; cap confidence lower and prefer INCONCLUSIVE over guessing.
|
||||
|
||||
The skill detects this from whether a verified GSC property / GA4 property is supplied or resolvable; if ambiguous, ask once.
|
||||
|
||||
### 3.1 L1 — Measured (first-party, native history) → delegates to `21-seo-position-tracking`
|
||||
|
||||
- **GSC** via `mcp__dda__gsc_fetch_performance`:
|
||||
- term query-level (exact match) AND site-wide, **recent vs prior** window.
|
||||
- report real avg position, clicks, impressions, CTR; **day-normalize** (compare periods often differ in calendar-day count).
|
||||
- note **~43% query-level anonymization** — query-sum ≠ aggregate; never treat the disclosed subset as the whole.
|
||||
- **query-clicks delta** (recent − prior) to name which terms actually moved (brand/seasonal vs the claimed term).
|
||||
- **GA4** via `mcp__dda__ga4_run_report`: `Organic Search` sessions monthly trend (dimensions `yearMonth` + `sessionDefaultChannelGroup`, metric `sessions`); GA4 captures **all engines incl. Naver**, so use it to test whether a "surge" exceeds normal month-to-month variance.
|
||||
- **Short-circuit:** if the claimed keyword has trivial clicks and a real position nowhere near the claim → **ARTIFACT**, stop (skip L2/L3 unless caller wants the full picture).
|
||||
|
||||
### 3.2 L2 — Live SERP (third-party measured, point-in-time) → delegates to `20-seo-serp-analysis`
|
||||
|
||||
- **Live geo-correct Google render** via `claude-in-chrome` (`navigate` → `read_page`/`computer`): force `gl`/`hl` + correct geo, `pws=0`; **decline precise-location prompts** (privacy). Confirm whether the domain actually holds the claimed position; capture the feature landscape (ads, local map-pack, PAA, knowledge panel, image/video) that explains *why* a brand site can't hold a head term.
|
||||
- **Cheap rank spot-check** via `mcp__ourseo__check_serp(keyword, domain)` when a full render is unnecessary.
|
||||
- **[KR market]** Naver SERP composition via `our research naver serp` (blog/cafe/지식iN/Smart Store/brand zone) — required for Korean entities since Semrush/Ahrefs don't model Naver.
|
||||
|
||||
### 3.3 L3 — Entity / Knowledge Graph (the differentiator) → delegates to `28-seo-knowledge-graph`
|
||||
|
||||
A real impact event should leave corroborating traces in the **entity layer**, not just a rank number. Five checks:
|
||||
|
||||
1. **Google KG API** entity match + `resultScore` — `mcp__ourseo__search_knowledge_graph(query)` (uses `GOOGLE_KG_API_KEY`).
|
||||
2. **Wikidata** QID presence + key claims — **verify the QID against `Special:EntityData/{Q}.json` labels** before trusting it (bakes in the JHR false-match guard: Q109455878 = office tower ≠ hotel; Q490787 = Shinsegae Inc. ≠ Group).
|
||||
3. **Knowledge Panel** presence/attributes on the live entity-name SERP (Chrome).
|
||||
4. **sameAs** consistency on the entity's `Organization`/`Person` JSON-LD.
|
||||
5. **[KR]** Naver 백과사전 / 지식iN presence.
|
||||
|
||||
`mcp__ourseo__monitor_brand` supplements with brand-mention/brand-SERP ownership signal.
|
||||
|
||||
### 3.4 L4 — Attribution synthesis → verdict
|
||||
|
||||
Cross-check: does the **measured delta (L1)** corroborate the **live reality (L2)**, and does the **entity layer (L3)** show consistent movement? The query-clicks delta names the true drivers. Output a verdict (§5) with an evidence ledger.
|
||||
|
||||
## 4. Entry modes (thin wrappers over the engine)
|
||||
|
||||
| Mode | Input contract | Engine use |
|
||||
|---|---|---|
|
||||
| **`adjudicate(claim)`** | `{term, entity, claim:{source, metric, from→to}}` e.g. `SEMrush: 호텔 pos 16→3, organic surge` | Full cascade; verdict confirms/refutes the claim |
|
||||
| **`prove(baseline)`** | `{term, entity, change:{what, when}}` | Measured before/after from GSC/GA4 history; entity baseline = most recent existing Notion KG-audit archive — **if none exists, report current entity state only and mark the entity-layer delta INCONCLUSIVE** (the change pre-dates any captured baseline); live captured now |
|
||||
| **`snapshot()`** | `{term, entity}` | Cascade with no claim; "where do we really stand" across all four layers |
|
||||
|
||||
All three call the *same* engine; they differ only in what they compare against.
|
||||
|
||||
## 5. Verdict logic
|
||||
|
||||
| Verdict | Condition |
|
||||
|---|---|
|
||||
| **CONFIRMED** | Measured + live + (where relevant) entity all corroborate movement attributable to the term/intent |
|
||||
| **PARTIAL** | Real movement, but misattributed (e.g., growth is brand/seasonal, not the claimed head term) or only some layers agree |
|
||||
| **ARTIFACT** | Modeling/snapshot artifact — measured + live reality don't support it (the JHR 호텔 case) |
|
||||
| **INCONCLUSIVE** | Insufficient data (query anonymized, GSC lag, no entity baseline, third-party entity with no measured access) — names exactly what's missing + how to resolve |
|
||||
|
||||
**Confidence cap:** third-party entities (no L1) cannot reach CONFIRMED on traffic claims — at most PARTIAL, and ARTIFACT only when live+entity reality clearly contradicts the claim.
|
||||
|
||||
**Standing skepticism rules** (baked in from `feedback-semrush-serp-signal-validation`):
|
||||
- Estimated organic traffic = **smoke-detector, not scale** (Σ est-volume × position-CTR curve).
|
||||
- **Head-term over-fire**: one high-volume keyword caught at an estimated high rank inflates the whole modeled number.
|
||||
- **KR Naver blind spot**: Semrush models Google only; misses a large share of Korean organic.
|
||||
- **Single-geo/device snapshot** diverges from GSC's national average.
|
||||
- **Data-trust hierarchy**: 1st-party measured (GA4/GSC) > 3rd-party measured (backlinks, crawled rank) > 3rd-party modeled (estimated traffic).
|
||||
|
||||
### Output of the verdict
|
||||
- **Evidence ledger** — per layer: finding + its data-trust tier + whether it corroborates or contradicts the claim.
|
||||
- **Client-safe narrative** — the defensible story (e.g., "summer brand/long-tail demand lifted impressions +18%, clicks modest" — NOT "ranked #3 for 호텔").
|
||||
|
||||
## 6. Output
|
||||
|
||||
- **Always:** inline structured report (verdict + ledger + narrative + "what would raise confidence").
|
||||
- **Optional:** archive to Notion **Working with AI DB** (`data_source_id f8f19ede-32bd-43ac-9f60-0651f6f40afe`) via the **notion-writer script** (per global policy — never Notion MCP write tools). Properties follow the DB schema (Type=Memo/Research, Account Code, Topic=SEO, etc.).
|
||||
- **Optional:** if the run surfaces a new *generalizable* gotcha, append a memory entry to the active workspace's memory dir.
|
||||
|
||||
## 7. Repo layout & conventions
|
||||
|
||||
```
|
||||
35-seo-signal-validation/
|
||||
SKILL.md frontmatter (name/description/triggers), mode dispatch,
|
||||
the cascade, short-circuit + verdict synthesis
|
||||
DESIGN.md this spec (lives with the skill; no new top-level dir)
|
||||
references/
|
||||
evidence-cascade.md L1–L4 concrete tool calls + short-circuit rules
|
||||
kg-impact-checks.md the 5 entity checks incl. Wikidata QID verification
|
||||
verdict-taxonomy.md conditions + ledger format + narrative template
|
||||
backend-selection.md pointer to 20/21's existing backend tables (no dup)
|
||||
```
|
||||
|
||||
**Triggers:** `validate serp signal`, `is this ranking real`, `prove SEO impact`, `SEMrush surge real?`, `signal validation`, `신호 검증`, `순위 변화 진짜?`, `오가닉 급증 검증`.
|
||||
|
||||
**Conventions honored:** no new output directories beyond this approved folder; Notion writes via notion-writer script only; never crawl/audit Marriott for JHR (sameAs reference only); KR deliverables in Korean, English internal notes OK.
|
||||
|
||||
## 8. Non-goals (YAGNI)
|
||||
|
||||
- No cron/scheduler and no snapshot DB (stateless, on-demand). A snapshot store + watchlist monitor is a documented **future option**, built only if proven needed.
|
||||
- Does **not** replace the three instrument skills — it sequences them.
|
||||
- Does **not** fabricate a verdict when data is thin — returns INCONCLUSIVE with a remediation list.
|
||||
- Not a general SEO audit; scoped to validating a specific `(term, entity)` impact question.
|
||||
|
||||
## 9. Future options (explicitly out of v1)
|
||||
|
||||
- Lightweight snapshot store in the existing `dda` SQLite workspace to enable true over-time entity-layer deltas.
|
||||
- Optional scheduled monitor over a `(term, entity)` watchlist that flags anomalies for `adjudicate`.
|
||||
- Multi-engine claim intake (parse a pasted SEMrush/Ahrefs export directly).
|
||||
Reference in New Issue
Block a user