feat(seo-schema-validator): back the upgraded SKILL.md with a working 5-layer pipeline
The "Upgrade Schema Validator" commit added SKILL.md referencing files that did not exist. Implement them so the skill actually runs: - scripts/validate_schema.py — 5-layer offline validator (L0 coverage, L1 syntax, L2 vocabulary/value-format, L3 rich-result, L4 consistency) with xlsx/csv/jsonl/ json/dir/live-URL adapters. Gate = zero P0; exits 1 on failure. - scripts/schema_rules.json — curated hotel-focused, offline rule set (edit-only extension point). - scripts/make_sample.py + fixtures/sample_schema.csv — deliberately flawed fixture seeding ≥1 defect per layer; used to self-test. - references/ — validation-methodology, defect-taxonomy (25 codes), hotel-type-map. - templates/ — client-qa-report, decision-log. - code/CLAUDE.md — redirect legacy single-URL tool to the new pipeline. Noise control: MISSING_RECOMMENDED aggregated one-line-per-node; unexpected-property checks opt-in via --strict. Generalized client-specific shilla-type-map → hotel-type-map. Self-tested: default P0=5/P1=4/P2=14 FAIL, --strict --no-recommended P2=0, adapters verified. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -42,12 +42,12 @@ This skill's primary job is **Mode A**: catch errors before the client sees them
|
|||||||
|
|
||||||
Full rationale and the type→requirement matrix: `references/validation-methodology.md`.
|
Full rationale and the type→requirement matrix: `references/validation-methodology.md`.
|
||||||
Severity + category codes: `references/defect-taxonomy.md`.
|
Severity + category codes: `references/defect-taxonomy.md`.
|
||||||
Shilla page-type → schema-type map: `references/shilla-type-map.md`.
|
Hotel page-type → schema-type map: `references/hotel-type-map.md`.
|
||||||
Client-facing report + P1 decision log: `templates/client-qa-report-template.md`, `templates/decision-log.md`.
|
Client-facing report + P1 decision log: `templates/client-qa-report-template.md`, `templates/decision-log.md`.
|
||||||
|
|
||||||
## Stage gates (aligned to the project lifecycle 설계→개발→테스트→안정화→런칭 후)
|
## Stage gates (aligned to the project lifecycle 설계→개발→테스트→안정화→런칭 후)
|
||||||
|
|
||||||
- **G1 설계** — Lock the schema spec and the page-type→type map (`shilla-type-map.md`). Approve the entry template. *DoD:* every page template has an assigned schema type and a required-property list.
|
- **G1 설계** — Lock the schema spec and the page-type→type map (`hotel-type-map.md`). Approve the entry template. *DoD:* every page template has an assigned schema type and a required-property list.
|
||||||
- **G2 개발** — Authors produce entries. Run the validator with `--strict`. *DoD (gate):* **zero P0**, JSON parses for 100% of entries. Entries failing this NEVER reach client review.
|
- **G2 개발** — Authors produce entries. Run the validator with `--strict`. *DoD (gate):* **zero P0**, JSON parses for 100% of entries. Entries failing this NEVER reach client review.
|
||||||
- **G3 테스트** — Re-run; triage P1 in `defect_log.csv` (assign owner/status). Client reviews ONLY the clean, P0-free entries, against a defect report — not raw JSON. *DoD:* P1 triaged, decisions logged in `templates/decision-log.md`.
|
- **G3 테스트** — Re-run; triage P1 in `defect_log.csv` (assign owner/status). Client reviews ONLY the clean, P0-free entries, against a defect report — not raw JSON. *DoD:* P1 triaged, decisions logged in `templates/decision-log.md`.
|
||||||
- **G4 안정화** — Fix → re-run → confirm no regressions. Spot-check a sample in Google Rich Results Test (online, outside this runtime). *DoD:* P0=0, P1 accepted/closed, online validator green on sample.
|
- **G4 안정화** — Fix → re-run → confirm no regressions. Spot-check a sample in Google Rich Results Test (online, outside this runtime). *DoD:* P0=0, P1 accepted/closed, online validator green on sample.
|
||||||
|
|||||||
@@ -1,148 +1,57 @@
|
|||||||
# CLAUDE.md
|
# CLAUDE.md — seo-schema-validator (Claude Code)
|
||||||
|
|
||||||
## Overview
|
## Canonical entry point
|
||||||
|
|
||||||
Structured data validator: extract, parse, and validate JSON-LD, Microdata, and RDFa markup against schema.org vocabulary.
|
This skill was upgraded to a **5-layer dataset-QA pipeline**. The authoritative
|
||||||
|
directive and run instructions live in the skill root:
|
||||||
|
|
||||||
## Quick Start
|
- **`../SKILL.md`** — modes, the 5 layers, stage gates, how to run.
|
||||||
|
- **`../scripts/validate_schema.py`** — the validator (run this, not the legacy script below).
|
||||||
|
- **`../scripts/schema_rules.json`** — the offline rule set (edit this to add a type/rule).
|
||||||
|
- **`../references/`** — `validation-methodology.md`, `defect-taxonomy.md`, `hotel-type-map.md`.
|
||||||
|
- **`../templates/`** — `client-qa-report-template.md`, `decision-log.md`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install -r scripts/requirements.txt
|
# Primary use — QA an AUTHORED dataset before the client sees it (Mode A)
|
||||||
python scripts/schema_validator.py --url https://example.com
|
python ../scripts/validate_schema.py DATASET.xlsx --url-list URLLIST.xlsx --out schema_qa_out
|
||||||
|
|
||||||
|
# Highest-signal pre-review gate
|
||||||
|
python ../scripts/validate_schema.py DATASET.csv --strict --no-recommended --out qa_strict
|
||||||
|
|
||||||
|
# Try the bundled flawed fixture first
|
||||||
|
python ../scripts/make_sample.py
|
||||||
|
python ../scripts/validate_schema.py ../fixtures/sample_schema.csv --out demo_out
|
||||||
|
|
||||||
|
# Post-deploy live audit (Mode B) — feeds seo-comprehensive-audit stage 4
|
||||||
|
python ../scripts/validate_schema.py --live https://example.com --out live_out
|
||||||
```
|
```
|
||||||
|
|
||||||
## Scripts
|
Gate rule: **PASS = zero P0.** The process exits 1 when the gate fails, so it stops
|
||||||
|
`&&` chains and CI. Only P0-free entries advance to client review.
|
||||||
|
|
||||||
| Script | Purpose |
|
## Legacy single-URL tool (kept for quick one-offs)
|
||||||
|--------|---------|
|
|
||||||
| `schema_validator.py` | Extract and validate structured data |
|
|
||||||
| `base_client.py` | Shared utilities |
|
|
||||||
|
|
||||||
## Usage
|
`scripts/schema_validator.py --url <URL>` extracts and validates structured data from
|
||||||
|
one live page (JSON-LD / Microdata / RDFa via extruct). It predates the pipeline and is
|
||||||
|
**not** the gate. For any dataset or client-facing QA, use `validate_schema.py` above.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Validate page schema
|
pip install -r scripts/requirements.txt # extruct, jsonschema, rdflib, lxml, requests
|
||||||
python scripts/schema_validator.py --url https://example.com
|
|
||||||
|
|
||||||
# JSON output
|
|
||||||
python scripts/schema_validator.py --url https://example.com --json
|
python scripts/schema_validator.py --url https://example.com --json
|
||||||
|
|
||||||
# Validate local file
|
|
||||||
python scripts/schema_validator.py --file schema.json
|
|
||||||
|
|
||||||
# Check Rich Results eligibility
|
|
||||||
python scripts/schema_validator.py --url https://example.com --rich-results
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Supported Formats
|
## Notion output (OurDigital SEO Audit Log)
|
||||||
|
|
||||||
| Format | Detection |
|
When a run is part of an OurDigital/D.intelligence audit, log a summary to the SEO Audit
|
||||||
|--------|-----------|
|
Log database. Per the user-level Notion rule, push **page content** with the
|
||||||
| JSON-LD | `<script type="application/ld+json">` |
|
`notion-writer` skill; use Notion MCP only for **properties** (Status, Category, etc.).
|
||||||
| Microdata | `itemscope`, `itemtype`, `itemprop` |
|
|
||||||
| RDFa | `vocab`, `typeof`, `property` |
|
|
||||||
|
|
||||||
## Validation Levels
|
|
||||||
|
|
||||||
### 1. Syntax Validation
|
|
||||||
- Valid JSON structure
|
|
||||||
- Proper nesting
|
|
||||||
- No syntax errors
|
|
||||||
|
|
||||||
### 2. Schema.org Vocabulary
|
|
||||||
- Valid @type values
|
|
||||||
- Known properties
|
|
||||||
- Correct property types
|
|
||||||
|
|
||||||
### 3. Google Rich Results
|
|
||||||
- Required properties present
|
|
||||||
- Recommended properties
|
|
||||||
- Feature-specific requirements
|
|
||||||
|
|
||||||
## Schema Types Validated
|
|
||||||
|
|
||||||
| Type | Required Properties | Rich Result |
|
|
||||||
|------|---------------------|-------------|
|
|
||||||
| Article | headline, author, datePublished | Yes |
|
|
||||||
| Product | name, offers | Yes |
|
|
||||||
| LocalBusiness | name, address | Yes |
|
|
||||||
| FAQPage | mainEntity | Yes |
|
|
||||||
| Organization | name, url | Yes |
|
|
||||||
| BreadcrumbList | itemListElement | Yes |
|
|
||||||
| WebSite | name, url | Sitelinks |
|
|
||||||
|
|
||||||
## Output
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"url": "https://example.com",
|
|
||||||
"schemas_found": 3,
|
|
||||||
"schemas": [
|
|
||||||
{
|
|
||||||
"@type": "Organization",
|
|
||||||
"valid": true,
|
|
||||||
"rich_results_eligible": true,
|
|
||||||
"issues": [],
|
|
||||||
"warnings": []
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"summary": {
|
|
||||||
"valid": 3,
|
|
||||||
"invalid": 0,
|
|
||||||
"rich_results_eligible": 2
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Issue Severity
|
|
||||||
|
|
||||||
| Level | Description |
|
|
||||||
|-------|-------------|
|
|
||||||
| Error | Invalid schema, blocks rich results |
|
|
||||||
| Warning | Missing recommended property |
|
|
||||||
| Info | Optimization suggestion |
|
|
||||||
|
|
||||||
## Dependencies
|
|
||||||
|
|
||||||
```
|
|
||||||
extruct>=0.16.0
|
|
||||||
jsonschema>=4.21.0
|
|
||||||
rdflib>=7.0.0
|
|
||||||
lxml>=5.1.0
|
|
||||||
requests>=2.31.0
|
|
||||||
```
|
|
||||||
|
|
||||||
## Notion Output (Required)
|
|
||||||
|
|
||||||
**IMPORTANT**: All audit reports MUST be saved to the OurDigital SEO Audit Log database.
|
|
||||||
|
|
||||||
### Database Configuration
|
|
||||||
|
|
||||||
| Field | Value |
|
| Field | Value |
|
||||||
|-------|-------|
|
|-------|-------|
|
||||||
| Database ID | `2c8581e5-8a1e-8035-880b-e38cefc2f3ef` |
|
| Database ID | `2c8581e5-8a1e-8035-880b-e38cefc2f3ef` |
|
||||||
| URL | https://www.notion.so/dintelligence/2c8581e58a1e8035880be38cefc2f3ef |
|
| Category | `Schema/Structured Data` |
|
||||||
|
| Priority | map gate: FAIL→Critical/High, PASS-with-P1→Medium, PASS-clean→Low |
|
||||||
### Required Properties
|
| Audit ID | `SCHEMA-YYYYMMDD-NNN` |
|
||||||
|
|
||||||
| Property | Type | Description |
|
|
||||||
|----------|------|-------------|
|
|
||||||
| Issue | Title | Report title (Korean + date) |
|
|
||||||
| Site | URL | Audited website URL |
|
|
||||||
| Category | Select | Technical SEO, On-page SEO, Performance, Schema/Structured Data, Sitemap, Robots.txt, Content, Local SEO |
|
|
||||||
| Priority | Select | Critical, High, Medium, Low |
|
|
||||||
| Found Date | Date | Audit date (YYYY-MM-DD) |
|
|
||||||
| Audit ID | Rich Text | Format: [TYPE]-YYYYMMDD-NNN |
|
|
||||||
|
|
||||||
### Language Guidelines
|
|
||||||
|
|
||||||
- Report content in Korean (한국어)
|
|
||||||
- Keep technical English terms as-is (e.g., SEO Audit, Core Web Vitals, Schema Markup)
|
|
||||||
- URLs and code remain unchanged
|
|
||||||
|
|
||||||
### Example MCP Call
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mcp-cli call notion/API-post-page '{"parent": {"database_id": "2c8581e5-8a1e-8035-880b-e38cefc2f3ef"}, "properties": {...}}'
|
|
||||||
```
|
|
||||||
|
|
||||||
|
Report content in Korean; keep technical terms (Schema, JSON-LD, rich result) and
|
||||||
|
URLs/code unchanged.
|
||||||
|
|||||||
@@ -0,0 +1,16 @@
|
|||||||
|
url,언어코드,PC/MOBILE,page_type,스키마
|
||||||
|
https://www.josunhotel.com/en/brand/grand,en,PC,brand-hub,"{""@context"": ""https://schema.org"", ""@type"": ""Organization"", ""@id"": ""https://www.josunhotel.com/#org"", ""name"": ""Josun Hotels & Resorts"", ""url"": ""https://www.josunhotel.com/"", ""logo"": ""https://www.josunhotel.com/logo.png"", ""sameAs"": [""https://www.instagram.com/josunhotelsandresorts/""]}"
|
||||||
|
https://www.josunhotel.com/ko/grand,ko,PC,hotel,"{""@context"": ""https://schema.org"", ""@type"": ""Hotel"", name: ""그랜드조선"",}"
|
||||||
|
https://www.josunhotel.com/ko/grand/rooms,ko,MOBILE,rooms,"{""@context"": ""https://schema.org"", ""name"": ""디럭스룸"", ""url"": ""https://www.josunhotel.com/ko/grand/rooms""}"
|
||||||
|
https://www.josunhotel.com/ko/palace,ko,PC,hotel,"{""@context"": ""https://example.org"", ""@type"": ""Hotel"", ""name"": ""조선팰리스"", ""address"": {""@type"": ""PostalAddress"", ""streetAddress"": ""테헤란로 231"", ""addressLocality"": ""서울"", ""addressCountry"": ""KR""}}"
|
||||||
|
https://www.josunhotel.com/ko/lescape,ko,PC,hotel,"{""@context"": ""https://schema.org"", ""@type"": ""Hotel"", ""name"": ""레스케이프 호텔"", ""telephone"": ""+82-2-317-4000"", ""description"": ""조선호텔앤리조트가 운영하는 럭셔리 호텔로, 도심 속에서 품격 있는 휴식을 제공합니다. 최상의 서비스와 시설을 경험하실 수 있습니다.""}"
|
||||||
|
https://www.josunhotel.com/ko/grand/dining,ko,PC,restaurant,"{""@context"": ""https://schema.org"", ""@type"": ""Restaurant"", ""name"": ""예시 레스토랑"", ""address"": {""@type"": ""PostalAddress"", ""streetAddress"": ""수정필요"", ""addressCountry"": ""KR""}, ""servesCuisine"": ""Korean""}"
|
||||||
|
https://www.josunhotel.com/ko/westin,ko,PC,hotel,"{""@context"": ""https://schema.org"", ""@type"": ""Hotel"", ""name"": ""웨스틴 조선 서울"", ""telephone"": ""+82-2-771-0500"", ""address"": {""@type"": ""PostalAddress"", ""streetAddress"": ""소공로 106"", ""addressLocality"": ""서울"", ""addressCountry"": ""KR""}, ""description"": ""조선호텔앤리조트가 운영하는 럭셔리 호텔로, 도심 속에서 품격 있는 휴식을 제공합니다. 최상의 서비스와 시설을 경험하실 수 있습니다.""}"
|
||||||
|
https://www.josunhotel.com/en/westin,en,PC,hotel,"{""@context"": ""https://schema.org"", ""@type"": ""Hotel"", ""name"": ""웨스틴 조선 서울"", ""telephone"": ""+82-2-771-9999"", ""address"": {""@type"": ""PostalAddress"", ""streetAddress"": ""소공로 106"", ""addressLocality"": ""Seoul"", ""addressCountry"": ""KR""}, ""description"": ""조선호텔앤리조트가 운영하는 럭셔리 호텔로, 도심 속에서 품격 있는 휴식을 제공합니다. 최상의 서비스와 시설을 경험하실 수 있습니다.""}"
|
||||||
|
https://www.josunhotel.com/ko,ko,PC,home,"{""@context"": ""https://schema.org"", ""@type"": ""WebSite"", ""name"": ""조선호텔앤리조트"", ""url"": ""https://www.josunhotel.com/"", ""publisher"": {""@id"": ""https://www.josunhotel.com/#missing-org""}}"
|
||||||
|
https://www.josunhotel.com/ko/grand/location,ko,PC,location,"{""@context"": ""https://schema.org"", ""@type"": ""Hotel"", ""name"": ""그랜드 조선 부산"", ""address"": {""@type"": ""PostalAddress"", ""streetAddress"": ""동백로 60"", ""addressLocality"": ""부산"", ""addressCountry"": ""KR""}, ""geo"": {""@type"": ""GeoCoordinates"", ""latitude"": 129.1603, ""longitude"": 35.1586}}"
|
||||||
|
https://www.josunhotel.com/ko/offers/spring,ko,PC,offer,"{""@context"": ""https://schema.org"", ""@type"": ""Offer"", ""price"": ""350000"", ""priceCurrency"": ""KRW"", ""validFrom"": ""2026년 3월 1일"", ""url"": ""https://www.josunhotel.com/ko/offers/spring""}"
|
||||||
|
https://www.josunhotel.com/ko/offers/dining,ko,PC,offer,"{""@context"": ""https://schema.org"", ""@type"": ""Offer"", ""price"": ""120000"", ""priceCurrency"": ""₩"", ""availability"": ""https://schema.org/InStock""}"
|
||||||
|
https://www.josunhotel.com/ko/spa,ko,PC,facility,"{""@context"": ""https://schema.org"", ""@type"": ""SpaResort"", ""name"": ""조선 스파""}"
|
||||||
|
https://www.josunhotel.com/ko/grand/intro,ko,PC,hotel,"{""@context"": ""https://schema.org"", ""@type"": ""Hotel"", ""name"": ""그랜드 조선 제주"", ""address"": {""@type"": ""PostalAddress"", ""streetAddress"": ""중문관광로 75"", ""addressLocality"": ""제주"", ""addressCountry"": ""KR""}, ""description"": ""조선호텔앤리조트가 운영하는 럭셔리 호텔로, 도심 속에서 품격 있는 휴식을 제공합니다. 최상의 서비스와 시설을 경험하실 수 있습니다.""}"
|
||||||
|
https://www.josunhotel.com/ko/faq?stale=1,ko,MOBILE,faq,"{""@context"": ""https://schema.org"", ""@type"": ""FAQPage"", ""mainEntity"": [{""@type"": ""Question"", ""name"": ""체크인 시간은 언제인가요?"", ""acceptedAnswer"": {""@type"": ""Answer"", ""text"": ""오후 3시부터 체크인 가능합니다.""}}]}"
|
||||||
|
@@ -0,0 +1,74 @@
|
|||||||
|
# Defect Taxonomy
|
||||||
|
|
||||||
|
Every code `validate_schema.py` can emit, its default severity, and what to do.
|
||||||
|
The validator writes these to `defect_log.csv` (columns: `entry_id, url, node_type,
|
||||||
|
layer, code, severity, message, status, owner, note`) and `results.json`.
|
||||||
|
|
||||||
|
## Severity model
|
||||||
|
|
||||||
|
| Severity | Definition | Owner action |
|
||||||
|
|---|---|---|
|
||||||
|
| **P0** | Blocker — breaks parsing, blocks the rich result, or ships wrong/placeholder data. **Fails the gate.** | Must fix before the entry reaches client review. |
|
||||||
|
| **P1** | Real defect, doesn't block the rich result. | Fix before launch; track in the triage log. |
|
||||||
|
| **P2** | Optimization — recommended properties, formatting, orphan URLs. | Backlog; fix opportunistically. |
|
||||||
|
|
||||||
|
`--strict` promotes vocabulary/format warnings (and unknown types) from P2 to P1 and
|
||||||
|
turns on `UNEXPECTED_PROPERTY`. `--no-recommended` drops `MISSING_RECOMMENDED` entirely.
|
||||||
|
**Neither changes the gate — the gate is always "zero P0."**
|
||||||
|
|
||||||
|
## Code reference
|
||||||
|
|
||||||
|
### Layer 0 — Coverage
|
||||||
|
| Code | Sev | Trigger | Fix |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `COVERAGE_MISSING` | P1 | Inventory URL has no authored entry. | Author the entry, or remove the URL from the inventory. |
|
||||||
|
| `COVERAGE_ORPHAN` | P2 | Entry URL isn't in the inventory. | Fix the URL typo, or update the canonical list. |
|
||||||
|
|
||||||
|
### Layer 1 — Syntax
|
||||||
|
| Code | Sev | Trigger | Fix |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `INVALID_JSON` | P0 | JSON does not parse. | Fix the JSON (trailing comma, unquoted key, smart quotes). |
|
||||||
|
| `NO_SCHEMA_IN_HTML` | P0 | Live page has no `ld+json` block (Mode B). | Confirm the tag deployed and renders. |
|
||||||
|
| `MISSING_CONTEXT` | P1 | No top-level `@context`. | Add `"@context": "https://schema.org"`. |
|
||||||
|
| `WRONG_CONTEXT` | P1 | `@context` isn't schema.org. | Correct the context URL. |
|
||||||
|
| `NO_TYPE` | P1 | No `@type` anywhere in the entry. | Add the intended `@type`. |
|
||||||
|
| `ENCODING_CORRUPTION` | P1 | Replacement char `<60>` present. | Re-export as UTF-8; check the source pipeline. |
|
||||||
|
| `FETCH_ERROR` | P1 | Live URL could not be fetched (Mode B). | Check the URL/network; retry. |
|
||||||
|
|
||||||
|
### Layer 2 — Vocabulary & value formats
|
||||||
|
| Code | Sev (strict) | Trigger | Fix |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `UNKNOWN_TYPE` | P2 (P1) | `@type` not in the curated rule set. | If intended, add it to `schema_rules.json`; else correct the type. |
|
||||||
|
| `UNEXPECTED_PROPERTY` | — (P1) | Property unknown for a known type (**strict only**). | Remove the typo'd property, or add it to the type's `allowed`. |
|
||||||
|
| `BAD_URL` | P2 (P1) | A URL property isn't an `http(s)` URL. | Use an absolute URL. |
|
||||||
|
| `BAD_DATE` | P2 (P1) | A date property isn't ISO-8601. | Use `YYYY-MM-DD` (or full datetime). |
|
||||||
|
| `BAD_LANG` | P2 (P1) | `inLanguage`/`availableLanguage` isn't a BCP-47 code. | Use `ko`, `en`, `ja`, `zh`, … |
|
||||||
|
| `BAD_CURRENCY` | P2 (P1) | `priceCurrency` isn't a 3-letter ISO-4217 code. | Use `KRW`/`USD`, not `₩`/`$`. |
|
||||||
|
| `BAD_NUMBER` | P2 (P1) | A numeric property isn't numeric. | Remove units/commas; keep digits. |
|
||||||
|
|
||||||
|
### Layer 3 — Rich-result requirements
|
||||||
|
| Code | Sev | Trigger | Fix |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `MISSING_REQUIRED` | P0 | A Google-required property is absent. | Add the property — the rich result is blocked without it. |
|
||||||
|
| `MISSING_RECOMMENDED` | P2 | Recommended properties absent (one line per node, lists all). | Add what applies to improve eligibility/appearance. |
|
||||||
|
|
||||||
|
### Layer 4 — Consistency
|
||||||
|
| Code | Sev | Trigger | Fix |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `PLACEHOLDER_TEXT` | P0 | Boilerplate token in a string (`예시`, `수정필요`, `lorem`, `{{`, …). | Replace with real content. |
|
||||||
|
| `NAP_PHONE_MISMATCH` | P0 | Same business, different `telephone` across entries. | Reconcile to the canonical phone. |
|
||||||
|
| `NAP_ADDRESS_MISMATCH` | P0 | Same business, different `streetAddress`. | Reconcile to the canonical address. |
|
||||||
|
| `DUPLICATE_ID` | P1 | One `@id` defined ≥2× with differing content. | Make definitions identical, or split the `@id`. |
|
||||||
|
| `DANGLING_ID` | P1 | `{"@id": …}` reference to a node never defined. | Define the node, or fix the reference. |
|
||||||
|
| `GEO_SWAPPED` | P1 | latitude/longitude transposed (swapping fixes it). | Swap the values. |
|
||||||
|
| `GEO_OUT_OF_RANGE` | P1 | Coordinates impossible (lat∉[-90,90] or lon∉[-180,180]). | Correct the coordinates. |
|
||||||
|
| `DUPLICATE_DESCRIPTION` | P1 | Same description reused across ≥3 entries. | Write distinct descriptions per page. |
|
||||||
|
|
||||||
|
## Triage workflow
|
||||||
|
|
||||||
|
1. Sort `defect_log.csv` by severity (already sorted P0→P1→P2 on write).
|
||||||
|
2. **P0:** assign an owner, fix, re-run. No P0 may survive into client review.
|
||||||
|
3. **P1:** set `owner` + `status`, decide fix-now vs accept; log accepted ones in
|
||||||
|
`templates/decision-log.md`.
|
||||||
|
4. **P2:** schedule into the optimization backlog.
|
||||||
|
5. Re-run after fixes and confirm no regressions before advancing the stage gate.
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
# Hotel Page-Type → Schema-Type Map
|
||||||
|
|
||||||
|
The G1 (설계) deliverable: every page template gets an assigned schema `@type` and a
|
||||||
|
required-property list *before* anyone authors entries. Locking this map first is what
|
||||||
|
prevents the most expensive error — authoring hundreds of entries against the wrong type.
|
||||||
|
|
||||||
|
This is the reusable, hotel-domain map. The worked example is JHR (Josun Hotels &
|
||||||
|
Resorts, `josunhotel.com`) — a multi-brand, multi-language, multi-property group — but
|
||||||
|
the mapping applies to any lodging-group site (it replaces the earlier client-specific
|
||||||
|
draft). Adapt the brand/property layer to the client; the page-type → type rules are stable.
|
||||||
|
|
||||||
|
## Site shape this map assumes
|
||||||
|
|
||||||
|
```
|
||||||
|
대표 허브 (group) → Organization + WebSite (one canonical node set, @id-anchored)
|
||||||
|
브랜드 허브 (brand) → Brand / Organization + Hotel families
|
||||||
|
개별 호텔 (property) → Hotel / LodgingBusiness / Resort
|
||||||
|
├─ 객실 (rooms) → HotelRoom / Suite (nested or itemList)
|
||||||
|
├─ 다이닝 (dining) → Restaurant / BarOrPub
|
||||||
|
├─ 시설·웨딩·연회 → LocalBusiness / MeetingRoom (nested)
|
||||||
|
├─ 프로모션 (offers) → Offer / AggregateOffer
|
||||||
|
└─ FAQ / 안내 → FAQPage
|
||||||
|
```
|
||||||
|
|
||||||
|
Each rendered URL also multiplies by **language × device** (ko/en/ja/zh × PC/MOBILE),
|
||||||
|
which is why the entry count reaches the thousands. The schema `@type` does **not**
|
||||||
|
change across language/device — only the localized string values do. (Use that fact:
|
||||||
|
NAP, geo, and `@id` must stay identical across the language variants of one property;
|
||||||
|
Layer 4 will catch it when they drift.)
|
||||||
|
|
||||||
|
## The map
|
||||||
|
|
||||||
|
| Page template (Korean / English) | Primary `@type` | Required (P0) | Add these (recommended) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 대표 홈 / group home | `WebSite` (+ `Organization`) | name, url / name, url | publisher, potentialAction(SearchAction), inLanguage / logo, sameAs |
|
||||||
|
| 브랜드 허브 / brand hub | `Organization` (+ `Brand`) | name, url | logo, sameAs, brand |
|
||||||
|
| 호텔 메인 / property home | `Hotel` (or `LodgingBusiness`, `Resort`) | name, address | telephone, image, priceRange, geo, url, starRating, aggregateRating |
|
||||||
|
| 객실 목록·상세 / rooms | `Hotel` w/ `containsPlace`→`HotelRoom`/`Suite`, or `ItemList` | name, address (host) | image, occupancy, bed, amenityFeature |
|
||||||
|
| 다이닝 / restaurant | `Restaurant` (or `BarOrPub`) | name, address | servesCuisine, priceRange, telephone, menu, openingHoursSpecification, acceptsReservations |
|
||||||
|
| 부대시설·웨딩·연회 / facilities | `LocalBusiness` w/ `MeetingRoom` nested | name, address | telephone, openingHoursSpecification, image, url |
|
||||||
|
| 프로모션·패키지 / offers | `Offer` (or `AggregateOffer`) | price, priceCurrency / lowPrice, priceCurrency | availability, url, validFrom, priceValidUntil |
|
||||||
|
| 멤버십 / membership | `MemberProgram` | name | hasTiers, hostingOrganization, url |
|
||||||
|
| 위치·오시는 길 / location | `Hotel` w/ `geo`→`GeoCoordinates` | name, address | geo (lat/long), hasMap |
|
||||||
|
| FAQ / 자주 묻는 질문 | `FAQPage` → `Question`/`Answer` | mainEntity / name, acceptedAnswer / text | — |
|
||||||
|
| 공지·매거진·기사 / article | `Article` / `NewsArticle` / `BlogPosting` | headline | author, datePublished, image, dateModified, publisher |
|
||||||
|
| 모든 하위 페이지 / breadcrumbs | `BreadcrumbList` → `ListItem` | itemListElement / position | item, name |
|
||||||
|
|
||||||
|
## Conventions that keep Layer 4 green
|
||||||
|
|
||||||
|
- **Anchor shared nodes by `@id`.** Define `Organization` and `WebSite` once
|
||||||
|
(`https://…/#organization`, `https://…/#website`) and reference them everywhere with
|
||||||
|
`{"@id": "…"}`. Avoids `DANGLING_ID` (define before you reference) and `DUPLICATE_ID`
|
||||||
|
(don't redefine with different content).
|
||||||
|
- **One canonical NAP per property.** The same `telephone` and `streetAddress` must
|
||||||
|
appear in every language/device variant of a property, or `NAP_*_MISMATCH` (P0) fires.
|
||||||
|
- **Distinct descriptions.** A reused boilerplate description across ≥3 pages →
|
||||||
|
`DUPLICATE_DESCRIPTION` (P1). Write per-page copy.
|
||||||
|
- **geo as `{latitude, longitude}`** in decimal degrees; Korea is lat ≈ 33–39, lon ≈
|
||||||
|
124–132. Transposing them trips `GEO_SWAPPED`.
|
||||||
|
- **No placeholders.** `예시 / 수정필요 / 임시 / lorem / {{…}}` anywhere → `PLACEHOLDER_TEXT`
|
||||||
|
(P0). The gate exists precisely to stop these from reaching the client.
|
||||||
|
|
||||||
|
## Using the map in the lifecycle
|
||||||
|
|
||||||
|
- **G1 설계:** fill this table for the client's actual templates; that *is* the schema
|
||||||
|
spec. DoD: every template has a type + required list.
|
||||||
|
- **G2 개발:** authors produce entries against it; `--strict` run; zero P0 to advance.
|
||||||
|
- **G3+:** the map is the reference reviewers and the validator agree on.
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
# Validation Methodology
|
||||||
|
|
||||||
|
The reasoning behind the 5 layers, and the type → requirement matrix the validator
|
||||||
|
enforces. The matrix is the human-readable mirror of `scripts/schema_rules.json` —
|
||||||
|
if you change one, change the other.
|
||||||
|
|
||||||
|
## Why a machine gate before human review
|
||||||
|
|
||||||
|
At a few dozen entries, a person can eyeball JSON-LD. At hundreds (a multi-language,
|
||||||
|
multi-device, multi-property hotel site easily reaches 2,000+ URLs), eyeballing
|
||||||
|
fails in a predictable way: the reviewer drowns in *mechanical* errors (a missing
|
||||||
|
required field, a bad date format, a typo'd URL) and never reaches the *judgement*
|
||||||
|
errors that actually need a human (is this the right schema type for this page? is
|
||||||
|
this description accurate?).
|
||||||
|
|
||||||
|
The fix is not "review harder." It is to split the work by who is best at it:
|
||||||
|
|
||||||
|
| Error class | Best checker | This skill |
|
||||||
|
|---|---|---|
|
||||||
|
| Mechanical (parse, required-present, value format, duplicate, consistency) | A script, every time | Layers 0–4, automated |
|
||||||
|
| Judgement (type choice, copy accuracy, intent) | A human, once | Client reviews only P0-free entries |
|
||||||
|
|
||||||
|
So the gate runs first. **An entry reaches client review only when it has zero P0.**
|
||||||
|
The client then reviews a clean set against a defect report — never raw JSON in a meeting.
|
||||||
|
|
||||||
|
## The layers, in order
|
||||||
|
|
||||||
|
Each layer assumes the previous one passed for that entry. A fatal L1 failure
|
||||||
|
(unparseable JSON, no `@type`) stops deeper layers for that entry — there is nothing
|
||||||
|
to inspect.
|
||||||
|
|
||||||
|
### L0 — Coverage (needs `--url-list`)
|
||||||
|
Compares the canonical URL inventory against the URLs that actually have an entry.
|
||||||
|
- `COVERAGE_MISSING` (P1): inventory URL with no authored entry — a gap to fill.
|
||||||
|
- `COVERAGE_ORPHAN` (P2): entry whose URL isn't in the inventory — a typo, a stale
|
||||||
|
path, or a list that's out of date. (Expect many orphans if your inventory is a
|
||||||
|
subset; expect ~zero when it's the real canonical list.)
|
||||||
|
|
||||||
|
### L1 — Syntax
|
||||||
|
The cheapest, hardest blockers. If these fail, nothing downstream is trustworthy.
|
||||||
|
- `INVALID_JSON` (P0), `NO_SCHEMA_IN_HTML` (P0, live mode).
|
||||||
|
- `MISSING_CONTEXT` / `WRONG_CONTEXT` / `NO_TYPE` / `ENCODING_CORRUPTION` (P1).
|
||||||
|
|
||||||
|
### L2 — Vocabulary & value formats
|
||||||
|
Is the type known, and are values well-formed?
|
||||||
|
- `UNKNOWN_TYPE` (P2; P1 in `--strict`): type isn't in the curated rule set. A
|
||||||
|
*warning*, not an error — add it to `schema_rules.json` if it's intended.
|
||||||
|
- `BAD_URL` / `BAD_DATE` / `BAD_LANG` / `BAD_CURRENCY` / `BAD_NUMBER` (P2; P1 strict).
|
||||||
|
- `UNEXPECTED_PROPERTY` (P1, `--strict` only): a property not known for a known type.
|
||||||
|
**Off by default** — flagging every unexpected property offline produces exactly the
|
||||||
|
false-positive flood that makes reviewers distrust the tool.
|
||||||
|
|
||||||
|
### L3 — Rich-result requirements
|
||||||
|
The contract Google enforces for eligibility.
|
||||||
|
- `MISSING_REQUIRED` (P0): a required property is absent → the rich result is blocked.
|
||||||
|
- `MISSING_RECOMMENDED` (P2): recommended properties absent. **Aggregated to one line
|
||||||
|
per node** (never one defect per property) — this is the single most important
|
||||||
|
noise-control decision in the tool.
|
||||||
|
|
||||||
|
### L4 — Consistency (cross-node / cross-entry)
|
||||||
|
The errors a per-entry check can't see.
|
||||||
|
- `PLACEHOLDER_TEXT` (P0): boilerplate that escaped authoring (`예시`, `수정필요`,
|
||||||
|
`lorem`, `{{`, …). Almost always a real, embarrassing leak.
|
||||||
|
- `NAP_PHONE_MISMATCH` / `NAP_ADDRESS_MISMATCH` (P0): the same business shows
|
||||||
|
different Name/Address/Phone across entries — a local-SEO and trust problem.
|
||||||
|
- `DUPLICATE_ID` (P1): one `@id` defined twice with different content.
|
||||||
|
- `DANGLING_ID` (P1): a `{"@id": …}` reference points at a node never defined.
|
||||||
|
- `GEO_SWAPPED` / `GEO_OUT_OF_RANGE` (P1): latitude/longitude transposed or impossible.
|
||||||
|
- `DUPLICATE_DESCRIPTION` (P1): the same description reused across ≥3 entries.
|
||||||
|
|
||||||
|
## Severity → gate
|
||||||
|
|
||||||
|
| Severity | Meaning | Gate effect |
|
||||||
|
|---|---|---|
|
||||||
|
| **P0** | Blocker. Breaks parsing, blocks the rich result, or publishes wrong data. | **Fails the gate.** Process exits 1. Entry must not reach client review. |
|
||||||
|
| **P1** | Fix before launch. Real defect, doesn't block the rich result. | Triage backlog. |
|
||||||
|
| **P2** | Optimization. Recommended props, style, orphan URLs. | Optimization backlog. |
|
||||||
|
|
||||||
|
Full code list: `defect-taxonomy.md`.
|
||||||
|
|
||||||
|
## Type → requirement matrix (mirror of `schema_rules.json`)
|
||||||
|
|
||||||
|
`required` missing → **P0**. `recommended` missing → **P2** (aggregated). Anything in
|
||||||
|
`allowed` is accepted silently. Properties outside all three are flagged only in `--strict`.
|
||||||
|
|
||||||
|
| Type | Required (P0 if missing) | Recommended (P2 if missing) |
|
||||||
|
|---|---|---|
|
||||||
|
| Organization | name, url | logo, sameAs, contactPoint, address |
|
||||||
|
| WebSite | name, url | publisher, potentialAction, inLanguage |
|
||||||
|
| WebPage | name | url, isPartOf, primaryImageOfPage, breadcrumb, datePublished, dateModified |
|
||||||
|
| Hotel / LodgingBusiness / Resort | name, address | telephone, image, priceRange, geo, url, starRating, aggregateRating |
|
||||||
|
| LocalBusiness | name, address | telephone, openingHoursSpecification, geo, image, url, priceRange, aggregateRating |
|
||||||
|
| Restaurant / FoodEstablishment | name, address | servesCuisine, priceRange, telephone, menu, openingHoursSpecification |
|
||||||
|
| FAQPage | mainEntity | — |
|
||||||
|
| Question | name, acceptedAnswer | — |
|
||||||
|
| Answer | text | — |
|
||||||
|
| BreadcrumbList / ItemList | itemListElement | — |
|
||||||
|
| ListItem | position | item, name |
|
||||||
|
| Product | name | image, offers, brand, aggregateRating, review, description, sku |
|
||||||
|
| Offer | price, priceCurrency | availability, url, validFrom, priceValidUntil |
|
||||||
|
| Article / NewsArticle / BlogPosting | headline | author, datePublished, image, dateModified, publisher |
|
||||||
|
| Event | name, startDate, location | endDate, offers, performer, image, eventStatus, eventAttendanceMode, organizer |
|
||||||
|
| Review | reviewRating, author | datePublished, reviewBody, itemReviewed |
|
||||||
|
| AggregateRating | ratingValue | reviewCount, ratingCount, bestRating |
|
||||||
|
| MemberProgram | name | hasTiers, hostingOrganization, url |
|
||||||
|
|
||||||
|
**Container types** (validated for value formats, but *not* for required/recommended,
|
||||||
|
because they only ever appear nested): PostalAddress, GeoCoordinates, ImageObject,
|
||||||
|
ContactPoint, OpeningHoursSpecification, Rating, Brand, EntryPoint, Place, OfferCatalog,
|
||||||
|
ReserveAction, MeetingRoom, Room/HotelRoom/Suite, MemberProgramTier, Menu/MenuItem, … (full
|
||||||
|
list in `schema_rules.json` → `container_types`).
|
||||||
|
|
||||||
|
## Extending the rules
|
||||||
|
|
||||||
|
Add a type, tighten a requirement, or recognize a new container by editing
|
||||||
|
`scripts/schema_rules.json` **only** — no Python change needed:
|
||||||
|
- New rich-result type → add to `known_types` with `required` / `recommended` / `allowed`.
|
||||||
|
- New nested type to stop "unknown type" warnings → add to `container_types`.
|
||||||
|
- New value-format property → add to the relevant `value_formats` group.
|
||||||
|
- New placeholder token to catch → add to `placeholder_tokens`.
|
||||||
|
|
||||||
|
After any edit, re-run `make_sample.py` + `validate_schema.py` against the fixture to
|
||||||
|
confirm you didn't regress.
|
||||||
160
custom-skills/16-seo-schema-validator/scripts/make_sample.py
Normal file
160
custom-skills/16-seo-schema-validator/scripts/make_sample.py
Normal file
@@ -0,0 +1,160 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
make_sample.py — generate fixtures/sample_schema.csv.
|
||||||
|
|
||||||
|
A small, deliberately FLAWED hotel dataset (Josun-style, fictional values) that
|
||||||
|
seeds at least one defect per validation layer. Use it to learn the tool and to
|
||||||
|
regression-test changes to validate_schema.py or schema_rules.json:
|
||||||
|
|
||||||
|
python make_sample.py
|
||||||
|
python validate_schema.py ../fixtures/sample_schema.csv --out /tmp/demo_out
|
||||||
|
|
||||||
|
Each row's comment names the defect(s) it is designed to trigger.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import csv
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
OUT = Path(__file__).resolve().parent.parent / "fixtures" / "sample_schema.csv"
|
||||||
|
|
||||||
|
CTX = "https://schema.org"
|
||||||
|
SHARED_DESC = ("조선호텔앤리조트가 운영하는 럭셔리 호텔로, 도심 속에서 품격 있는 휴식을 "
|
||||||
|
"제공합니다. 최상의 서비스와 시설을 경험하실 수 있습니다.") # >30 chars, reused 3x
|
||||||
|
|
||||||
|
|
||||||
|
def jd(obj):
|
||||||
|
return json.dumps(obj, ensure_ascii=False)
|
||||||
|
|
||||||
|
|
||||||
|
# Each tuple: (url, lang, device, page_type, jsonld_string)
|
||||||
|
ROWS = []
|
||||||
|
|
||||||
|
# 1) CLEAN Organization — only a recommended gap (P2 MISSING_RECOMMENDED, aggregated)
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/en/brand/grand", "en", "PC", "brand-hub",
|
||||||
|
jd({"@context": CTX, "@type": "Organization", "@id": "https://www.josunhotel.com/#org",
|
||||||
|
"name": "Josun Hotels & Resorts", "url": "https://www.josunhotel.com/",
|
||||||
|
"logo": "https://www.josunhotel.com/logo.png",
|
||||||
|
"sameAs": ["https://www.instagram.com/josunhotelsandresorts/"]}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 2) INVALID JSON (P0 INVALID_JSON) — trailing comma, unquoted key
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/grand", "ko", "PC", "hotel",
|
||||||
|
'{"@context": "https://schema.org", "@type": "Hotel", name: "그랜드조선",}',
|
||||||
|
))
|
||||||
|
|
||||||
|
# 3) MISSING @type (P1 NO_TYPE)
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/grand/rooms", "ko", "MOBILE", "rooms",
|
||||||
|
jd({"@context": CTX, "name": "디럭스룸", "url": "https://www.josunhotel.com/ko/grand/rooms"}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 4) WRONG @context (P1 WRONG_CONTEXT)
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/palace", "ko", "PC", "hotel",
|
||||||
|
jd({"@context": "https://example.org", "@type": "Hotel", "name": "조선팰리스",
|
||||||
|
"address": {"@type": "PostalAddress", "streetAddress": "테헤란로 231",
|
||||||
|
"addressLocality": "서울", "addressCountry": "KR"}}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 5) Hotel MISSING REQUIRED address (P0 MISSING_REQUIRED)
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/lescape", "ko", "PC", "hotel",
|
||||||
|
jd({"@context": CTX, "@type": "Hotel", "name": "레스케이프 호텔",
|
||||||
|
"telephone": "+82-2-317-4000", "description": SHARED_DESC}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 6) PLACEHOLDER text (P0 PLACEHOLDER_TEXT)
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/grand/dining", "ko", "PC", "restaurant",
|
||||||
|
jd({"@context": CTX, "@type": "Restaurant", "name": "예시 레스토랑",
|
||||||
|
"address": {"@type": "PostalAddress", "streetAddress": "수정필요",
|
||||||
|
"addressCountry": "KR"}, "servesCuisine": "Korean"}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 7a + 7b) NAP PHONE MISMATCH (P0 NAP_PHONE_MISMATCH) — same business, two phones
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/westin", "ko", "PC", "hotel",
|
||||||
|
jd({"@context": CTX, "@type": "Hotel", "name": "웨스틴 조선 서울",
|
||||||
|
"telephone": "+82-2-771-0500",
|
||||||
|
"address": {"@type": "PostalAddress", "streetAddress": "소공로 106",
|
||||||
|
"addressLocality": "서울", "addressCountry": "KR"},
|
||||||
|
"description": SHARED_DESC}),
|
||||||
|
))
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/en/westin", "en", "PC", "hotel",
|
||||||
|
jd({"@context": CTX, "@type": "Hotel", "name": "웨스틴 조선 서울",
|
||||||
|
"telephone": "+82-2-771-9999",
|
||||||
|
"address": {"@type": "PostalAddress", "streetAddress": "소공로 106",
|
||||||
|
"addressLocality": "Seoul", "addressCountry": "KR"},
|
||||||
|
"description": SHARED_DESC}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 8) DANGLING @id reference (P1 DANGLING_ID) — publisher points at undefined node
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko", "ko", "PC", "home",
|
||||||
|
jd({"@context": CTX, "@type": "WebSite", "name": "조선호텔앤리조트",
|
||||||
|
"url": "https://www.josunhotel.com/",
|
||||||
|
"publisher": {"@id": "https://www.josunhotel.com/#missing-org"}}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 9) SWAPPED geo (P1 GEO_SWAPPED) — lat/long transposed for Seoul
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/grand/location", "ko", "PC", "location",
|
||||||
|
jd({"@context": CTX, "@type": "Hotel", "name": "그랜드 조선 부산",
|
||||||
|
"address": {"@type": "PostalAddress", "streetAddress": "동백로 60",
|
||||||
|
"addressLocality": "부산", "addressCountry": "KR"},
|
||||||
|
"geo": {"@type": "GeoCoordinates", "latitude": 129.1603, "longitude": 35.1586}}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 10) BAD date (P2 BAD_DATE) in an Offer-bearing page
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/offers/spring", "ko", "PC", "offer",
|
||||||
|
jd({"@context": CTX, "@type": "Offer", "price": "350000", "priceCurrency": "KRW",
|
||||||
|
"validFrom": "2026년 3월 1일", "url": "https://www.josunhotel.com/ko/offers/spring"}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 11) BAD currency symbol (P2 BAD_CURRENCY)
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/offers/dining", "ko", "PC", "offer",
|
||||||
|
jd({"@context": CTX, "@type": "Offer", "price": "120000", "priceCurrency": "₩",
|
||||||
|
"availability": "https://schema.org/InStock"}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 12) UNKNOWN type (P2 UNKNOWN_TYPE)
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/spa", "ko", "PC", "facility",
|
||||||
|
jd({"@context": CTX, "@type": "SpaResort", "name": "조선 스파"}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 13) Third reuse of SHARED_DESC → triggers DUPLICATE_DESCRIPTION (P1) across rows 5,7a,7b,13
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/grand/intro", "ko", "PC", "hotel",
|
||||||
|
jd({"@context": CTX, "@type": "Hotel", "name": "그랜드 조선 제주",
|
||||||
|
"address": {"@type": "PostalAddress", "streetAddress": "중문관광로 75",
|
||||||
|
"addressLocality": "제주", "addressCountry": "KR"},
|
||||||
|
"description": SHARED_DESC}),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 14) CLEAN FAQPage — exercises a passing entry (and an inventory-orphan URL for L0 demo)
|
||||||
|
ROWS.append((
|
||||||
|
"https://www.josunhotel.com/ko/faq?stale=1", "ko", "MOBILE", "faq",
|
||||||
|
jd({"@context": CTX, "@type": "FAQPage", "mainEntity": [
|
||||||
|
{"@type": "Question", "name": "체크인 시간은 언제인가요?",
|
||||||
|
"acceptedAnswer": {"@type": "Answer", "text": "오후 3시부터 체크인 가능합니다."}}]}),
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
OUT.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with open(OUT, "w", newline="", encoding="utf-8-sig") as f:
|
||||||
|
w = csv.writer(f)
|
||||||
|
w.writerow(["url", "언어코드", "PC/MOBILE", "page_type", "스키마"]) # Korean aliases on purpose
|
||||||
|
w.writerows(ROWS)
|
||||||
|
print(f"Wrote {len(ROWS)} entries → {OUT}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
# validate_schema.py runs on the Python standard library alone for
|
||||||
|
# CSV / JSON / JSONL / directory inputs (the offline default).
|
||||||
|
#
|
||||||
|
# Optional extras, installed only when you need them:
|
||||||
|
openpyxl>=3.1 # required to read .xlsx datasets and .xlsx URL inventories
|
||||||
|
requests>=2.31 # required only for --live (Mode B) URL fetching
|
||||||
200
custom-skills/16-seo-schema-validator/scripts/schema_rules.json
Normal file
200
custom-skills/16-seo-schema-validator/scripts/schema_rules.json
Normal file
@@ -0,0 +1,200 @@
|
|||||||
|
{
|
||||||
|
"_meta": {
|
||||||
|
"version": "1.0",
|
||||||
|
"scope": "Curated, hotel-focused subset of schema.org + Google rich-result requirements.",
|
||||||
|
"intent": "Self-contained offline rules (the runtime cannot reach schema.org or Google). Unknown types/properties degrade to warnings, never hard errors, to avoid false positives. To support a new type or tighten a rule, edit THIS file only.",
|
||||||
|
"sources": "schema.org/Hotel, schema.org/LocalBusiness, Google Search Central 'Structured data' rich-result docs (as of 2025)."
|
||||||
|
},
|
||||||
|
|
||||||
|
"valid_contexts": [
|
||||||
|
"https://schema.org",
|
||||||
|
"http://schema.org",
|
||||||
|
"https://schema.org/",
|
||||||
|
"http://schema.org/",
|
||||||
|
"https://www.schema.org",
|
||||||
|
"http://www.schema.org"
|
||||||
|
],
|
||||||
|
|
||||||
|
"global_properties": [
|
||||||
|
"@context", "@type", "@id", "@graph", "@reverse",
|
||||||
|
"name", "alternateName", "legalName", "description", "disambiguatingDescription",
|
||||||
|
"url", "image", "logo", "sameAs", "identifier", "mainEntityOfPage",
|
||||||
|
"additionalType", "subjectOf", "potentialAction", "inLanguage"
|
||||||
|
],
|
||||||
|
|
||||||
|
"known_types": {
|
||||||
|
"Organization": {
|
||||||
|
"required": ["name", "url"],
|
||||||
|
"recommended": ["logo", "sameAs", "contactPoint", "address"],
|
||||||
|
"allowed": ["legalName", "foundingDate", "parentOrganization", "subOrganization", "brand", "telephone", "email", "founder", "numberOfEmployees", "memberOf", "hasMerchantReturnPolicy", "member"]
|
||||||
|
},
|
||||||
|
"Corporation": {
|
||||||
|
"required": ["name", "url"],
|
||||||
|
"recommended": ["logo", "sameAs", "address"],
|
||||||
|
"allowed": ["legalName", "foundingDate", "parentOrganization", "tickerSymbol", "telephone", "email", "brand"]
|
||||||
|
},
|
||||||
|
"WebSite": {
|
||||||
|
"required": ["name", "url"],
|
||||||
|
"recommended": ["publisher", "potentialAction", "inLanguage"],
|
||||||
|
"allowed": ["alternateName", "about", "copyrightHolder", "copyrightYear"]
|
||||||
|
},
|
||||||
|
"WebPage": {
|
||||||
|
"required": ["name"],
|
||||||
|
"recommended": ["url", "isPartOf", "primaryImageOfPage", "breadcrumb", "datePublished", "dateModified"],
|
||||||
|
"allowed": ["about", "mentions", "speakable", "lastReviewed", "reviewedBy", "significantLink"]
|
||||||
|
},
|
||||||
|
"LocalBusiness": {
|
||||||
|
"required": ["name", "address"],
|
||||||
|
"recommended": ["telephone", "openingHoursSpecification", "geo", "image", "url", "priceRange", "aggregateRating"],
|
||||||
|
"allowed": ["email", "openingHours", "paymentAccepted", "currenciesAccepted", "areaServed", "hasMap", "department", "menu", "review", "containedInPlace", "containsPlace", "amenityFeature"]
|
||||||
|
},
|
||||||
|
"Hotel": {
|
||||||
|
"required": ["name", "address"],
|
||||||
|
"recommended": ["telephone", "image", "priceRange", "geo", "url", "starRating", "aggregateRating", "checkinTime", "checkoutTime"],
|
||||||
|
"allowed": ["email", "amenityFeature", "petsAllowed", "numberOfRooms", "availableLanguage", "containedInPlace", "containsPlace", "makesOffer", "brand", "currenciesAccepted", "smokingAllowed", "openingHoursSpecification", "audience", "review"]
|
||||||
|
},
|
||||||
|
"LodgingBusiness": {
|
||||||
|
"required": ["name", "address"],
|
||||||
|
"recommended": ["telephone", "image", "priceRange", "geo", "url", "starRating", "aggregateRating", "checkinTime", "checkoutTime"],
|
||||||
|
"allowed": ["email", "amenityFeature", "petsAllowed", "numberOfRooms", "availableLanguage", "containedInPlace", "containsPlace", "makesOffer", "currenciesAccepted", "smokingAllowed"]
|
||||||
|
},
|
||||||
|
"Resort": {
|
||||||
|
"required": ["name", "address"],
|
||||||
|
"recommended": ["telephone", "image", "priceRange", "geo", "url", "starRating", "aggregateRating"],
|
||||||
|
"allowed": ["email", "amenityFeature", "numberOfRooms", "containedInPlace", "containsPlace", "checkinTime", "checkoutTime"]
|
||||||
|
},
|
||||||
|
"Restaurant": {
|
||||||
|
"required": ["name", "address"],
|
||||||
|
"recommended": ["servesCuisine", "priceRange", "telephone", "menu", "openingHoursSpecification", "image", "url", "geo", "acceptsReservations"],
|
||||||
|
"allowed": ["email", "hasMenu", "starRating", "aggregateRating", "review", "containedInPlace", "smokingAllowed"]
|
||||||
|
},
|
||||||
|
"FoodEstablishment": {
|
||||||
|
"required": ["name", "address"],
|
||||||
|
"recommended": ["servesCuisine", "priceRange", "telephone", "menu", "openingHoursSpecification"],
|
||||||
|
"allowed": ["email", "hasMenu", "acceptsReservations", "containedInPlace"]
|
||||||
|
},
|
||||||
|
"BarOrPub": {
|
||||||
|
"required": ["name", "address"],
|
||||||
|
"recommended": ["telephone", "openingHoursSpecification", "priceRange", "servesCuisine"],
|
||||||
|
"allowed": ["menu", "hasMenu", "image", "url"]
|
||||||
|
},
|
||||||
|
"FAQPage": {
|
||||||
|
"required": ["mainEntity"],
|
||||||
|
"recommended": [],
|
||||||
|
"allowed": ["about", "headline", "datePublished", "dateModified"]
|
||||||
|
},
|
||||||
|
"Question": {
|
||||||
|
"required": ["name", "acceptedAnswer"],
|
||||||
|
"recommended": [],
|
||||||
|
"allowed": ["text", "answerCount", "suggestedAnswer", "upvoteCount", "author"]
|
||||||
|
},
|
||||||
|
"Answer": {
|
||||||
|
"required": ["text"],
|
||||||
|
"recommended": [],
|
||||||
|
"allowed": ["url", "upvoteCount", "author", "dateCreated"]
|
||||||
|
},
|
||||||
|
"BreadcrumbList": {
|
||||||
|
"required": ["itemListElement"],
|
||||||
|
"recommended": [],
|
||||||
|
"allowed": ["numberOfItems", "itemListOrder"]
|
||||||
|
},
|
||||||
|
"ItemList": {
|
||||||
|
"required": ["itemListElement"],
|
||||||
|
"recommended": [],
|
||||||
|
"allowed": ["numberOfItems", "itemListOrder"]
|
||||||
|
},
|
||||||
|
"ListItem": {
|
||||||
|
"required": ["position"],
|
||||||
|
"recommended": ["item", "name"],
|
||||||
|
"allowed": ["url", "image", "nextItem", "previousItem"]
|
||||||
|
},
|
||||||
|
"Product": {
|
||||||
|
"required": ["name"],
|
||||||
|
"recommended": ["image", "offers", "brand", "aggregateRating", "review", "description", "sku"],
|
||||||
|
"allowed": ["gtin", "gtin13", "gtin8", "gtin12", "mpn", "color", "material", "category", "audience", "isVariantOf", "additionalProperty", "hasMerchantReturnPolicy"]
|
||||||
|
},
|
||||||
|
"Offer": {
|
||||||
|
"required": ["price", "priceCurrency"],
|
||||||
|
"recommended": ["availability", "url", "validFrom", "priceValidUntil"],
|
||||||
|
"allowed": ["itemCondition", "seller", "eligibleRegion", "priceSpecification", "shippingDetails", "availabilityStarts"]
|
||||||
|
},
|
||||||
|
"AggregateOffer": {
|
||||||
|
"required": ["lowPrice", "priceCurrency"],
|
||||||
|
"recommended": ["highPrice", "offerCount"],
|
||||||
|
"allowed": ["offers", "availability"]
|
||||||
|
},
|
||||||
|
"Article": {
|
||||||
|
"required": ["headline"],
|
||||||
|
"recommended": ["author", "datePublished", "image", "dateModified", "publisher"],
|
||||||
|
"allowed": ["articleBody", "articleSection", "wordCount", "keywords", "speakable"]
|
||||||
|
},
|
||||||
|
"NewsArticle": {
|
||||||
|
"required": ["headline"],
|
||||||
|
"recommended": ["author", "datePublished", "image", "dateModified", "publisher"],
|
||||||
|
"allowed": ["articleBody", "dateline", "printSection"]
|
||||||
|
},
|
||||||
|
"BlogPosting": {
|
||||||
|
"required": ["headline"],
|
||||||
|
"recommended": ["author", "datePublished", "image", "dateModified", "publisher"],
|
||||||
|
"allowed": ["articleBody", "keywords", "wordCount"]
|
||||||
|
},
|
||||||
|
"Event": {
|
||||||
|
"required": ["name", "startDate", "location"],
|
||||||
|
"recommended": ["endDate", "offers", "performer", "image", "eventStatus", "eventAttendanceMode", "organizer"],
|
||||||
|
"allowed": ["doorTime", "previousStartDate", "typicalAgeRange", "maximumAttendeeCapacity"]
|
||||||
|
},
|
||||||
|
"Review": {
|
||||||
|
"required": ["reviewRating", "author"],
|
||||||
|
"recommended": ["datePublished", "reviewBody", "itemReviewed"],
|
||||||
|
"allowed": ["publisher", "name"]
|
||||||
|
},
|
||||||
|
"AggregateRating": {
|
||||||
|
"required": ["ratingValue"],
|
||||||
|
"recommended": ["reviewCount", "ratingCount", "bestRating"],
|
||||||
|
"allowed": ["worstRating", "itemReviewed"]
|
||||||
|
},
|
||||||
|
"MemberProgram": {
|
||||||
|
"required": ["name"],
|
||||||
|
"recommended": ["hasTiers", "hostingOrganization", "url"],
|
||||||
|
"allowed": ["description", "membershipPointsEarned"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
"container_types": [
|
||||||
|
"PostalAddress", "GeoCoordinates", "GeoShape", "ImageObject", "VideoObject",
|
||||||
|
"ContactPoint", "OpeningHoursSpecification", "Rating", "QuantitativeValue",
|
||||||
|
"MonetaryAmount", "PriceSpecification", "Brand", "EntryPoint", "Place",
|
||||||
|
"OfferCatalog", "ReserveAction", "OrderAction", "SearchAction", "ViewAction",
|
||||||
|
"MeetingRoom", "Room", "HotelRoom", "Suite", "LocationFeatureSpecification",
|
||||||
|
"MemberProgramTier", "MobileApplication", "WebApplication", "SoftwareApplication",
|
||||||
|
"Menu", "MenuItem", "MenuSection", "Country", "AdministrativeArea", "Duration",
|
||||||
|
"PropertyValue", "Person", "Audience", "Language"
|
||||||
|
],
|
||||||
|
|
||||||
|
"value_formats": {
|
||||||
|
"url_props": ["url", "logo", "sameAs", "image", "contentUrl", "thumbnailUrl", "target", "urlTemplate", "installUrl", "menu", "hasMap", "downloadUrl", "embedUrl"],
|
||||||
|
"date_props": ["datePublished", "dateModified", "dateCreated", "startDate", "endDate", "validFrom", "validThrough", "priceValidUntil", "foundingDate", "uploadDate", "availabilityStarts", "availabilityEnds", "lastReviewed", "previousStartDate"],
|
||||||
|
"lang_props": ["inLanguage", "availableLanguage"],
|
||||||
|
"currency_props": ["priceCurrency", "currenciesAccepted"],
|
||||||
|
"number_props": ["price", "lowPrice", "highPrice", "ratingValue", "reviewCount", "ratingCount", "bestRating", "worstRating", "position", "numberOfRooms", "maxValue", "minValue", "offerCount"]
|
||||||
|
},
|
||||||
|
|
||||||
|
"valid_currencies": ["KRW", "USD", "EUR", "JPY", "CNY", "GBP", "HKD", "SGD", "THB", "AUD", "CAD", "CHF", "TWD", "MYR", "PHP", "VND", "IDR", "INR"],
|
||||||
|
|
||||||
|
"valid_language_codes": ["ko", "en", "ja", "zh", "zh-CN", "zh-TW", "zh-Hans", "zh-Hant", "ko-KR", "en-US", "en-GB", "ja-JP", "fr", "de", "es", "ru", "th", "vi", "id", "ms"],
|
||||||
|
|
||||||
|
"placeholder_tokens": [
|
||||||
|
"lorem ipsum", "lorem", "ipsum", "dolor sit", "todo", "tbd", "fixme",
|
||||||
|
"xxx", "yyy", "zzz", "placeholder", "insert here", "insert text",
|
||||||
|
"example.com", "your-domain", "yourdomain", "changeme", "sample text",
|
||||||
|
"{{", "}}", "<insert", "[insert", "n/a", "샘플", "예시", "여기에",
|
||||||
|
"변경필요", "수정필요", "입력필요", "내용입력", "테스트", "임시"
|
||||||
|
],
|
||||||
|
|
||||||
|
"geo": {
|
||||||
|
"lat_min": -90.0, "lat_max": 90.0,
|
||||||
|
"lon_min": -180.0, "lon_max": 180.0,
|
||||||
|
"kr_lat_range": [33.0, 39.0],
|
||||||
|
"kr_lon_range": [124.0, 132.0]
|
||||||
|
}
|
||||||
|
}
|
||||||
854
custom-skills/16-seo-schema-validator/scripts/validate_schema.py
Normal file
854
custom-skills/16-seo-schema-validator/scripts/validate_schema.py
Normal file
@@ -0,0 +1,854 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
validate_schema.py — 5-layer offline JSON-LD schema validator.
|
||||||
|
|
||||||
|
WHY THIS EXISTS
|
||||||
|
---------------
|
||||||
|
When a client reviews hundreds of authored schema entries and says "there are too
|
||||||
|
many errors," the root cause is almost always that nobody ran a machine lint first.
|
||||||
|
Humans end up eyeballing raw JSON in a meeting. This tool moves every cheap,
|
||||||
|
machine-checkable error OUT of human review and INTO an automated gate that runs
|
||||||
|
first — so the client only ever sees clean, P0-free entries plus a defect report.
|
||||||
|
|
||||||
|
It is OFFLINE by design (the runtime cannot reach schema.org or Google). All rules
|
||||||
|
live in schema_rules.json; unknown types/properties degrade to warnings, never hard
|
||||||
|
errors, so the gate does not invent false positives.
|
||||||
|
|
||||||
|
THE 5 LAYERS
|
||||||
|
------------
|
||||||
|
L0 Coverage — URLs with no entry; entries whose URL isn't in the inventory.
|
||||||
|
L1 Syntax — invalid JSON, bad/missing @context, missing @type, encoding corruption.
|
||||||
|
L2 Vocabulary — unknown type, value-format errors (URL/date/lang/currency/number),
|
||||||
|
(strict only) unexpected properties on a known type.
|
||||||
|
L3 Rich-result — Google REQUIRED property missing (blocks rich result); recommended absent.
|
||||||
|
L4 Consistency — NAP mismatch, @id duplicates/dangling refs, swapped geo,
|
||||||
|
placeholder text, duplicate descriptions across entries.
|
||||||
|
|
||||||
|
GATE: PASS iff zero P0. Process exits 1 when the gate fails (so CI/`&&` chains stop).
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python validate_schema.py DATASET [--url-list URLLIST] [--out DIR]
|
||||||
|
[--strict] [--no-recommended]
|
||||||
|
[--live URL ...] [--rules schema_rules.json]
|
||||||
|
DATASET may be .xlsx / .csv (one row per entry, a JSON-LD column) / .jsonl / .json
|
||||||
|
/ a directory of .json|.jsonld files. With --live, validate live URLs instead.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from collections import Counter, defaultdict
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
RULES_DEFAULT = Path(__file__).resolve().parent / "schema_rules.json"
|
||||||
|
|
||||||
|
SEVERITY_ORDER = {"P0": 0, "P1": 1, "P2": 2}
|
||||||
|
|
||||||
|
# Header aliases for tabular input. Keys are normalized (lowercased, spaces removed).
|
||||||
|
COLUMN_ALIASES = {
|
||||||
|
"jsonld": ["jsonld", "jsonld", "json-ld", "json_ld", "schema", "schemamarkup",
|
||||||
|
"structureddata", "structured_data", "markup", "스키마", "구조화데이터",
|
||||||
|
"구조화된데이터", "jsonldcode", "스키마코드"],
|
||||||
|
"url": ["url", "메뉴url", "pageurl", "주소", "링크", "loc", "uri", "캐노니컬", "canonical"],
|
||||||
|
"lang": ["lang", "language", "언어", "언어코드", "locale", "lng"],
|
||||||
|
"device": ["device", "pc/mobile", "pcmobile", "pc_mobile", "platform", "디바이스", "기기"],
|
||||||
|
"page_type": ["page_type", "pagetype", "type", "페이지유형", "페이지타입", "menulevel",
|
||||||
|
"menu_level", "메뉴레벨", "template", "템플릿", "유형"],
|
||||||
|
}
|
||||||
|
|
||||||
|
URL_RE = re.compile(r"^https?://[^\s]+$", re.IGNORECASE)
|
||||||
|
# ISO-8601 date or datetime (date, date+time, optional tz). Loose but rejects free text.
|
||||||
|
DATE_RE = re.compile(
|
||||||
|
r"^\d{4}-\d{2}-\d{2}"
|
||||||
|
r"(?:[T ]\d{2}:\d{2}(?::\d{2})?(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?)?$"
|
||||||
|
)
|
||||||
|
LANG_RE = re.compile(r"^[a-zA-Z]{2,3}(?:-[A-Za-z0-9]{2,4})?$")
|
||||||
|
JSONLD_SCRIPT_RE = re.compile(
|
||||||
|
r'<script[^>]+type=["\']application/ld\+json["\'][^>]*>(.*?)</script>',
|
||||||
|
re.IGNORECASE | re.DOTALL,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Defect collection
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
class DefectLog:
|
||||||
|
"""Accumulates findings. One row per finding, ready for triage."""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.rows = []
|
||||||
|
|
||||||
|
def add(self, severity, layer, code, message, entry_id="", url="", node_type=""):
|
||||||
|
self.rows.append({
|
||||||
|
"entry_id": str(entry_id),
|
||||||
|
"url": url or "",
|
||||||
|
"node_type": node_type or "",
|
||||||
|
"layer": layer,
|
||||||
|
"code": code,
|
||||||
|
"severity": severity,
|
||||||
|
"message": message,
|
||||||
|
"status": "open",
|
||||||
|
"owner": "",
|
||||||
|
"note": "",
|
||||||
|
})
|
||||||
|
|
||||||
|
def counts(self):
|
||||||
|
c = Counter(r["severity"] for r in self.rows)
|
||||||
|
return {"P0": c.get("P0", 0), "P1": c.get("P1", 0), "P2": c.get("P2", 0)}
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Input adapters (Mode A: authored dataset / Mode B: live URLs)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def _norm_header(h):
|
||||||
|
return re.sub(r"\s+", "", str(h or "").strip().lower())
|
||||||
|
|
||||||
|
|
||||||
|
def _detect_columns(headers):
|
||||||
|
"""Map normalized headers to canonical column roles. Returns {role: index}."""
|
||||||
|
found = {}
|
||||||
|
for idx, h in enumerate(headers):
|
||||||
|
nh = _norm_header(h)
|
||||||
|
for role, aliases in COLUMN_ALIASES.items():
|
||||||
|
if role in found:
|
||||||
|
continue
|
||||||
|
if nh in aliases:
|
||||||
|
found[role] = idx
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def _row_to_entry(row, cols, entry_id, source_ref):
|
||||||
|
def cell(role):
|
||||||
|
i = cols.get(role)
|
||||||
|
if i is None or i >= len(row):
|
||||||
|
return None
|
||||||
|
v = row[i]
|
||||||
|
return None if v is None else str(v).strip()
|
||||||
|
raw = cell("jsonld")
|
||||||
|
if not raw:
|
||||||
|
return None # blank JSON-LD cell → no entry to validate
|
||||||
|
return {
|
||||||
|
"entry_id": entry_id,
|
||||||
|
"url": cell("url") or "",
|
||||||
|
"lang": cell("lang") or "",
|
||||||
|
"device": cell("device") or "",
|
||||||
|
"page_type": cell("page_type") or "",
|
||||||
|
"raw": raw,
|
||||||
|
"source_ref": source_ref,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _load_csv(path):
|
||||||
|
entries = []
|
||||||
|
with open(path, newline="", encoding="utf-8-sig") as f:
|
||||||
|
reader = csv.reader(f)
|
||||||
|
rows = list(reader)
|
||||||
|
if not rows:
|
||||||
|
return entries
|
||||||
|
cols = _detect_columns(rows[0])
|
||||||
|
if "jsonld" not in cols:
|
||||||
|
raise ValueError(
|
||||||
|
f"No JSON-LD column found in {path}. Looked for: "
|
||||||
|
f"{', '.join(COLUMN_ALIASES['jsonld'][:6])} … Headers were: {rows[0]}"
|
||||||
|
)
|
||||||
|
for n, row in enumerate(rows[1:], start=2):
|
||||||
|
e = _row_to_entry(row, cols, f"{Path(path).stem}#r{n}", f"{path}:row{n}")
|
||||||
|
if e:
|
||||||
|
entries.append(e)
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
def _load_xlsx(path):
|
||||||
|
try:
|
||||||
|
from openpyxl import load_workbook
|
||||||
|
except ImportError:
|
||||||
|
raise SystemExit(
|
||||||
|
"Reading .xlsx needs openpyxl: pip install openpyxl\n"
|
||||||
|
"(or export the sheet to .csv and pass that instead)."
|
||||||
|
)
|
||||||
|
entries = []
|
||||||
|
wb = load_workbook(path, read_only=True, data_only=True)
|
||||||
|
for sheet in wb.worksheets:
|
||||||
|
rows = list(sheet.iter_rows(values_only=True))
|
||||||
|
if not rows:
|
||||||
|
continue
|
||||||
|
cols = _detect_columns(rows[0])
|
||||||
|
if "jsonld" not in cols:
|
||||||
|
continue # a tab without a JSON-LD column (e.g. summary) — skip silently
|
||||||
|
for n, row in enumerate(rows[1:], start=2):
|
||||||
|
e = _row_to_entry(list(row), cols, f"{sheet.title}#r{n}",
|
||||||
|
f"{path}:{sheet.title}:row{n}")
|
||||||
|
if e:
|
||||||
|
entries.append(e)
|
||||||
|
if not entries:
|
||||||
|
raise ValueError(
|
||||||
|
f"No sheet in {path} had a recognizable JSON-LD column. "
|
||||||
|
f"Looked for: {', '.join(COLUMN_ALIASES['jsonld'][:6])} …"
|
||||||
|
)
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
def _looks_like_schema(obj):
|
||||||
|
"""True if a parsed object is itself JSON-LD (vs a wrapper row)."""
|
||||||
|
if isinstance(obj, list):
|
||||||
|
return True
|
||||||
|
if isinstance(obj, dict):
|
||||||
|
return any(k in obj for k in ("@context", "@type", "@graph"))
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _wrapper_to_entry(obj, entry_id, source_ref):
|
||||||
|
"""A JSONL/JSON wrapper object that carries url/lang + a jsonld payload."""
|
||||||
|
cols = {k: k for k in obj.keys()}
|
||||||
|
norm = {_norm_header(k): k for k in obj.keys()}
|
||||||
|
def pick(role):
|
||||||
|
for alias in COLUMN_ALIASES[role]:
|
||||||
|
if alias in norm:
|
||||||
|
v = obj[norm[alias]]
|
||||||
|
return v
|
||||||
|
return None
|
||||||
|
payload = pick("jsonld")
|
||||||
|
raw = payload if isinstance(payload, str) else json.dumps(payload, ensure_ascii=False)
|
||||||
|
url = pick("url")
|
||||||
|
return {
|
||||||
|
"entry_id": entry_id,
|
||||||
|
"url": str(url).strip() if url else "",
|
||||||
|
"lang": str(pick("lang") or "").strip(),
|
||||||
|
"device": str(pick("device") or "").strip(),
|
||||||
|
"page_type": str(pick("page_type") or "").strip(),
|
||||||
|
"raw": raw,
|
||||||
|
"source_ref": source_ref,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _load_jsonl(path):
|
||||||
|
entries = []
|
||||||
|
with open(path, encoding="utf-8") as f:
|
||||||
|
for n, line in enumerate(f, start=1):
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
sref = f"{path}:line{n}"
|
||||||
|
try:
|
||||||
|
obj = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
# Keep the bad line so L1 reports it as a syntax error.
|
||||||
|
entries.append({"entry_id": f"{Path(path).stem}#l{n}", "url": "",
|
||||||
|
"lang": "", "device": "", "page_type": "",
|
||||||
|
"raw": line, "source_ref": sref})
|
||||||
|
continue
|
||||||
|
eid = f"{Path(path).stem}#l{n}"
|
||||||
|
if _looks_like_schema(obj):
|
||||||
|
entries.append({"entry_id": eid, "url": "", "lang": "", "device": "",
|
||||||
|
"page_type": "", "raw": line, "source_ref": sref})
|
||||||
|
else:
|
||||||
|
entries.append(_wrapper_to_entry(obj, eid, sref))
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
def _load_json(path):
|
||||||
|
with open(path, encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
entries = []
|
||||||
|
if isinstance(data, dict) and not _looks_like_schema(data) and all(
|
||||||
|
isinstance(v, (dict, list, str)) for v in data.values()
|
||||||
|
) and not any(k.startswith("@") for k in data):
|
||||||
|
# url -> jsonld map
|
||||||
|
for url, payload in data.items():
|
||||||
|
raw = payload if isinstance(payload, str) else json.dumps(payload, ensure_ascii=False)
|
||||||
|
entries.append({"entry_id": url, "url": url, "lang": "", "device": "",
|
||||||
|
"page_type": "", "raw": raw, "source_ref": f"{path}:{url}"})
|
||||||
|
elif isinstance(data, list):
|
||||||
|
for n, item in enumerate(data, start=1):
|
||||||
|
sref = f"{path}:[{n}]"
|
||||||
|
eid = f"{Path(path).stem}#{n}"
|
||||||
|
if _looks_like_schema(item) or not isinstance(item, dict):
|
||||||
|
raw = item if isinstance(item, str) else json.dumps(item, ensure_ascii=False)
|
||||||
|
entries.append({"entry_id": eid, "url": "", "lang": "", "device": "",
|
||||||
|
"page_type": "", "raw": raw, "source_ref": sref})
|
||||||
|
else:
|
||||||
|
entries.append(_wrapper_to_entry(item, eid, sref))
|
||||||
|
else:
|
||||||
|
entries.append({"entry_id": Path(path).stem, "url": "", "lang": "", "device": "",
|
||||||
|
"page_type": "", "raw": json.dumps(data, ensure_ascii=False),
|
||||||
|
"source_ref": path})
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
def _load_dir(path):
|
||||||
|
entries = []
|
||||||
|
for p in sorted(Path(path).rglob("*")):
|
||||||
|
if p.suffix.lower() in (".json", ".jsonld"):
|
||||||
|
entries.append({"entry_id": p.stem, "url": "", "lang": "", "device": "",
|
||||||
|
"page_type": "", "raw": p.read_text(encoding="utf-8"),
|
||||||
|
"source_ref": str(p)})
|
||||||
|
if not entries:
|
||||||
|
raise ValueError(f"No .json/.jsonld files found under {path}")
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
def _load_live(urls):
|
||||||
|
try:
|
||||||
|
import requests
|
||||||
|
except ImportError:
|
||||||
|
raise SystemExit("Live mode (--live) needs requests: pip install requests")
|
||||||
|
entries = []
|
||||||
|
headers = {"User-Agent": "Mozilla/5.0 (compatible; SchemaValidator/1.0)"}
|
||||||
|
for url in urls:
|
||||||
|
try:
|
||||||
|
resp = requests.get(url, headers=headers, timeout=20)
|
||||||
|
resp.raise_for_status()
|
||||||
|
except Exception as exc: # noqa: BLE001 — best-effort live fetch
|
||||||
|
entries.append({"entry_id": url, "url": url, "lang": "", "device": "",
|
||||||
|
"page_type": "", "raw": "", "source_ref": url,
|
||||||
|
"_fetch_error": str(exc)})
|
||||||
|
continue
|
||||||
|
scripts = JSONLD_SCRIPT_RE.findall(resp.text)
|
||||||
|
if not scripts:
|
||||||
|
entries.append({"entry_id": url, "url": url, "lang": "", "device": "",
|
||||||
|
"page_type": "", "raw": "", "source_ref": url,
|
||||||
|
"_no_schema": True})
|
||||||
|
continue
|
||||||
|
for i, block in enumerate(scripts, start=1):
|
||||||
|
entries.append({"entry_id": f"{url}#{i}", "url": url, "lang": "",
|
||||||
|
"device": "", "page_type": "", "raw": block.strip(),
|
||||||
|
"source_ref": f"{url} (script {i})"})
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
def load_entries(input_path, live_urls):
|
||||||
|
if live_urls:
|
||||||
|
return _load_live(live_urls)
|
||||||
|
p = Path(input_path)
|
||||||
|
if p.is_dir():
|
||||||
|
return _load_dir(p)
|
||||||
|
suffix = p.suffix.lower()
|
||||||
|
if suffix == ".csv":
|
||||||
|
return _load_csv(p)
|
||||||
|
if suffix in (".xlsx", ".xlsm"):
|
||||||
|
return _load_xlsx(p)
|
||||||
|
if suffix == ".jsonl":
|
||||||
|
return _load_jsonl(p)
|
||||||
|
if suffix in (".json", ".jsonld"):
|
||||||
|
return _load_json(p)
|
||||||
|
raise ValueError(f"Unsupported input: {input_path} (suffix {suffix!r})")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Node helpers
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def type_of(node):
|
||||||
|
"""Return the primary @type as a string (first if it's a list)."""
|
||||||
|
t = node.get("@type")
|
||||||
|
if isinstance(t, list):
|
||||||
|
return t[0] if t else ""
|
||||||
|
return t or ""
|
||||||
|
|
||||||
|
|
||||||
|
def iter_typed_nodes(parsed):
|
||||||
|
"""Yield every dict that has an @type, top-level and nested (recursively)."""
|
||||||
|
seen = []
|
||||||
|
|
||||||
|
def walk(obj):
|
||||||
|
if isinstance(obj, dict):
|
||||||
|
if "@type" in obj:
|
||||||
|
seen.append(obj)
|
||||||
|
for v in obj.values():
|
||||||
|
walk(v)
|
||||||
|
elif isinstance(obj, list):
|
||||||
|
for v in obj:
|
||||||
|
walk(v)
|
||||||
|
|
||||||
|
# @graph documents: walk the graph; otherwise walk the object/array directly.
|
||||||
|
if isinstance(parsed, dict) and "@graph" in parsed:
|
||||||
|
walk(parsed["@graph"])
|
||||||
|
else:
|
||||||
|
walk(parsed)
|
||||||
|
return seen
|
||||||
|
|
||||||
|
|
||||||
|
def all_strings(obj):
|
||||||
|
"""Yield (key, value) for every string value anywhere in the structure."""
|
||||||
|
if isinstance(obj, dict):
|
||||||
|
for k, v in obj.items():
|
||||||
|
if isinstance(v, str):
|
||||||
|
yield k, v
|
||||||
|
else:
|
||||||
|
yield from all_strings(v)
|
||||||
|
elif isinstance(obj, list):
|
||||||
|
for v in obj:
|
||||||
|
yield from all_strings(v)
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_name(s):
|
||||||
|
return re.sub(r"\s+", " ", str(s or "").strip().lower())
|
||||||
|
|
||||||
|
|
||||||
|
def first_text(value):
|
||||||
|
"""Coerce a property value to a comparable scalar (handles list/dict)."""
|
||||||
|
if isinstance(value, list):
|
||||||
|
return first_text(value[0]) if value else ""
|
||||||
|
if isinstance(value, dict):
|
||||||
|
return value.get("name") or value.get("@id") or value.get("streetAddress") or ""
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Layer 0 — Coverage
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def load_url_inventory(url_list_path):
|
||||||
|
urls = set()
|
||||||
|
p = Path(url_list_path)
|
||||||
|
suffix = p.suffix.lower()
|
||||||
|
if suffix in (".xlsx", ".xlsm"):
|
||||||
|
from openpyxl import load_workbook
|
||||||
|
wb = load_workbook(p, read_only=True, data_only=True)
|
||||||
|
for sheet in wb.worksheets:
|
||||||
|
for row in sheet.iter_rows(values_only=True):
|
||||||
|
for cell in row:
|
||||||
|
if isinstance(cell, str) and URL_RE.match(cell.strip()):
|
||||||
|
urls.add(cell.strip())
|
||||||
|
elif suffix == ".csv":
|
||||||
|
with open(p, newline="", encoding="utf-8-sig") as f:
|
||||||
|
for row in csv.reader(f):
|
||||||
|
for cell in row:
|
||||||
|
if isinstance(cell, str) and URL_RE.match(cell.strip()):
|
||||||
|
urls.add(cell.strip())
|
||||||
|
else: # plain text, one URL per line
|
||||||
|
for line in p.read_text(encoding="utf-8").splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if URL_RE.match(line):
|
||||||
|
urls.add(line)
|
||||||
|
return urls
|
||||||
|
|
||||||
|
|
||||||
|
def layer0_coverage(entries, inventory, defects):
|
||||||
|
entry_urls = {e["url"] for e in entries if e.get("url")}
|
||||||
|
missing = inventory - entry_urls
|
||||||
|
for url in sorted(missing):
|
||||||
|
defects.add("P1", "L0", "COVERAGE_MISSING",
|
||||||
|
"Inventory URL has no authored schema entry.", url=url)
|
||||||
|
orphans = entry_urls - inventory
|
||||||
|
for url in sorted(orphans):
|
||||||
|
defects.add("P2", "L0", "COVERAGE_ORPHAN",
|
||||||
|
"Entry URL is not in the canonical URL inventory "
|
||||||
|
"(typo, stale path, or missing from list).", url=url)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Layer 1 — Syntax
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def layer1_syntax(entry, rules, defects):
|
||||||
|
"""Parse + structural checks. Returns parsed object or None (fatal)."""
|
||||||
|
eid, url = entry["entry_id"], entry["url"]
|
||||||
|
if entry.get("_fetch_error"):
|
||||||
|
defects.add("P1", "L1", "FETCH_ERROR",
|
||||||
|
f"Could not fetch live URL: {entry['_fetch_error']}", eid, url)
|
||||||
|
return None
|
||||||
|
if entry.get("_no_schema"):
|
||||||
|
defects.add("P0", "L1", "NO_SCHEMA_IN_HTML",
|
||||||
|
"Live page has no application/ld+json script block.", eid, url)
|
||||||
|
return None
|
||||||
|
raw = entry["raw"]
|
||||||
|
if "<EFBFBD>" in raw:
|
||||||
|
defects.add("P1", "L1", "ENCODING_CORRUPTION",
|
||||||
|
"Replacement character (\\ufffd) present — encoding corruption.",
|
||||||
|
eid, url)
|
||||||
|
try:
|
||||||
|
parsed = json.loads(raw)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
defects.add("P0", "L1", "INVALID_JSON",
|
||||||
|
f"JSON does not parse: {exc.msg} at line {exc.lineno} col {exc.colno}.",
|
||||||
|
eid, url)
|
||||||
|
return None
|
||||||
|
|
||||||
|
nodes = iter_typed_nodes(parsed)
|
||||||
|
if not nodes:
|
||||||
|
defects.add("P1", "L1", "NO_TYPE",
|
||||||
|
"No @type found anywhere in the entry — not a usable schema object.",
|
||||||
|
eid, url)
|
||||||
|
|
||||||
|
# @context lives at the top of the document; nested nodes inherit it.
|
||||||
|
if isinstance(parsed, dict):
|
||||||
|
ctx = parsed.get("@context")
|
||||||
|
if ctx is None:
|
||||||
|
defects.add("P1", "L1", "MISSING_CONTEXT",
|
||||||
|
"Top-level @context is missing.", eid, url)
|
||||||
|
else:
|
||||||
|
ctx_urls = [ctx] if isinstance(ctx, str) else (
|
||||||
|
[c for c in ctx if isinstance(c, str)] if isinstance(ctx, list) else []
|
||||||
|
)
|
||||||
|
valid = rules["valid_contexts"]
|
||||||
|
if ctx_urls and not any(c.rstrip("/") in [v.rstrip("/") for v in valid]
|
||||||
|
for c in ctx_urls):
|
||||||
|
defects.add("P1", "L1", "WRONG_CONTEXT",
|
||||||
|
f"@context is not schema.org: {ctx_urls}.", eid, url)
|
||||||
|
return parsed
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Layer 2 — Vocabulary + value formats
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def _check_value_formats(node, rules, defects, eid, url, ntype, severity):
|
||||||
|
vf = rules["value_formats"]
|
||||||
|
|
||||||
|
def each(value):
|
||||||
|
if isinstance(value, list):
|
||||||
|
for v in value:
|
||||||
|
yield from each(v)
|
||||||
|
else:
|
||||||
|
yield value
|
||||||
|
|
||||||
|
for prop, value in node.items():
|
||||||
|
if prop.startswith("@"):
|
||||||
|
continue
|
||||||
|
if prop in vf["url_props"]:
|
||||||
|
for v in each(value):
|
||||||
|
if isinstance(v, str) and not URL_RE.match(v.strip()):
|
||||||
|
defects.add(severity, "L2", "BAD_URL",
|
||||||
|
f"'{prop}' is not an http(s) URL: {v!r}.", eid, url, ntype)
|
||||||
|
if prop in vf["date_props"]:
|
||||||
|
for v in each(value):
|
||||||
|
if isinstance(v, str) and not DATE_RE.match(v.strip()):
|
||||||
|
defects.add(severity, "L2", "BAD_DATE",
|
||||||
|
f"'{prop}' is not ISO-8601: {v!r}.", eid, url, ntype)
|
||||||
|
if prop in vf["lang_props"]:
|
||||||
|
for v in each(value):
|
||||||
|
if isinstance(v, str) and not LANG_RE.match(v.strip()):
|
||||||
|
defects.add(severity, "L2", "BAD_LANG",
|
||||||
|
f"'{prop}' is not a BCP-47 language code: {v!r}.",
|
||||||
|
eid, url, ntype)
|
||||||
|
if prop in vf["currency_props"]:
|
||||||
|
for v in each(value):
|
||||||
|
if isinstance(v, str) and not re.match(r"^[A-Z]{3}$", v.strip()):
|
||||||
|
defects.add(severity, "L2", "BAD_CURRENCY",
|
||||||
|
f"'{prop}' is not a 3-letter ISO-4217 code: {v!r}.",
|
||||||
|
eid, url, ntype)
|
||||||
|
if prop in vf["number_props"]:
|
||||||
|
for v in each(value):
|
||||||
|
if isinstance(v, str):
|
||||||
|
try:
|
||||||
|
float(v.replace(",", ""))
|
||||||
|
except ValueError:
|
||||||
|
defects.add(severity, "L2", "BAD_NUMBER",
|
||||||
|
f"'{prop}' is not numeric: {v!r}.", eid, url, ntype)
|
||||||
|
|
||||||
|
|
||||||
|
def layer2_vocabulary(node, rules, defects, eid, url, strict):
|
||||||
|
ntype = type_of(node)
|
||||||
|
known = rules["known_types"]
|
||||||
|
containers = set(rules["container_types"])
|
||||||
|
minor = "P1" if strict else "P2"
|
||||||
|
|
||||||
|
if ntype and ntype not in known and ntype not in containers:
|
||||||
|
defects.add(minor, "L2", "UNKNOWN_TYPE",
|
||||||
|
f"@type '{ntype}' is not in the curated rule set "
|
||||||
|
"(treated as a warning — add it to schema_rules.json if intended).",
|
||||||
|
eid, url, ntype)
|
||||||
|
|
||||||
|
_check_value_formats(node, rules, defects, eid, url, ntype, minor)
|
||||||
|
|
||||||
|
# Unexpected-property check is OPT-IN (--strict). Off by default to avoid the
|
||||||
|
# exact noise explosion that makes clients say "too many errors".
|
||||||
|
if strict and ntype in known:
|
||||||
|
spec = known[ntype]
|
||||||
|
allowed = set(spec["required"]) | set(spec["recommended"]) | set(spec["allowed"])
|
||||||
|
allowed |= set(rules["global_properties"])
|
||||||
|
for prop in node:
|
||||||
|
if prop.startswith("@"):
|
||||||
|
continue
|
||||||
|
if prop not in allowed:
|
||||||
|
defects.add("P1", "L2", "UNEXPECTED_PROPERTY",
|
||||||
|
f"'{prop}' is not a known property of {ntype} (strict mode).",
|
||||||
|
eid, url, ntype)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Layer 3 — Rich-result (required / recommended)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def layer3_richresult(node, rules, defects, eid, url, no_recommended):
|
||||||
|
ntype = type_of(node)
|
||||||
|
known = rules["known_types"]
|
||||||
|
if ntype not in known:
|
||||||
|
return # containers + unknown types have no required-property contract
|
||||||
|
spec = known[ntype]
|
||||||
|
|
||||||
|
for prop in spec["required"]:
|
||||||
|
if not node.get(prop):
|
||||||
|
defects.add("P0", "L3", "MISSING_REQUIRED",
|
||||||
|
f"{ntype} is missing required property '{prop}' "
|
||||||
|
"(blocks the rich result).", eid, url, ntype)
|
||||||
|
|
||||||
|
if not no_recommended:
|
||||||
|
missing_rec = [p for p in spec["recommended"] if not node.get(p)]
|
||||||
|
if missing_rec:
|
||||||
|
# Aggregate to ONE line per node — never one defect per property.
|
||||||
|
defects.add("P2", "L3", "MISSING_RECOMMENDED",
|
||||||
|
f"{ntype} is missing recommended properties: "
|
||||||
|
f"{', '.join(missing_rec)}.", eid, url, ntype)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Layer 4 — Consistency (cross-node / cross-entry)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
NAP_TYPES = {"Organization", "Corporation", "LocalBusiness", "Hotel",
|
||||||
|
"LodgingBusiness", "Resort", "Restaurant", "FoodEstablishment", "BarOrPub"}
|
||||||
|
|
||||||
|
|
||||||
|
def _address_street(node):
|
||||||
|
addr = node.get("address")
|
||||||
|
if isinstance(addr, dict):
|
||||||
|
return normalize_name(addr.get("streetAddress"))
|
||||||
|
if isinstance(addr, list) and addr and isinstance(addr[0], dict):
|
||||||
|
return normalize_name(addr[0].get("streetAddress"))
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _walk_ids(obj, defined, referenced):
|
||||||
|
"""Collect @id definitions vs pure references by walking the whole document.
|
||||||
|
|
||||||
|
A *reference* is an object whose only key is @id (e.g. {"@id": "...#org"}).
|
||||||
|
A *definition* is any object carrying @id plus other content. References live
|
||||||
|
in untyped wrapper dicts, so this must walk the raw doc — not just typed nodes.
|
||||||
|
"""
|
||||||
|
if isinstance(obj, dict):
|
||||||
|
nid = obj.get("@id")
|
||||||
|
if nid:
|
||||||
|
if set(obj.keys()) == {"@id"}:
|
||||||
|
referenced.add(nid)
|
||||||
|
else:
|
||||||
|
defined.setdefault(nid, []).append(obj)
|
||||||
|
for v in obj.values():
|
||||||
|
_walk_ids(v, defined, referenced)
|
||||||
|
elif isinstance(obj, list):
|
||||||
|
for v in obj:
|
||||||
|
_walk_ids(v, defined, referenced)
|
||||||
|
|
||||||
|
|
||||||
|
def layer4_consistency(node_index, parsed_docs, rules, defects):
|
||||||
|
"""node_index: (entry, node) for every TYPED node.
|
||||||
|
parsed_docs: (entry, parsed) for every entry that parsed — used for @id scan."""
|
||||||
|
# ---- placeholder text (P0) ----
|
||||||
|
tokens = [t.lower() for t in rules["placeholder_tokens"]]
|
||||||
|
for entry, node in node_index:
|
||||||
|
ntype = type_of(node)
|
||||||
|
for key, val in all_strings(node):
|
||||||
|
low = val.lower()
|
||||||
|
hit = next((t for t in tokens if t in low), None)
|
||||||
|
if hit:
|
||||||
|
defects.add("P0", "L4", "PLACEHOLDER_TEXT",
|
||||||
|
f"Placeholder/boilerplate token {hit!r} in '{key}': {val[:60]!r}.",
|
||||||
|
entry["entry_id"], entry["url"], ntype)
|
||||||
|
break # one placeholder defect per node is enough signal
|
||||||
|
|
||||||
|
# ---- NAP consistency (P0) ----
|
||||||
|
by_name = defaultdict(list)
|
||||||
|
for entry, node in node_index:
|
||||||
|
if type_of(node) in NAP_TYPES and node.get("name"):
|
||||||
|
by_name[normalize_name(first_text(node.get("name")))].append((entry, node))
|
||||||
|
for name, group in by_name.items():
|
||||||
|
phones = {str(first_text(n.get("telephone"))).strip()
|
||||||
|
for _, n in group if n.get("telephone")}
|
||||||
|
streets = {_address_street(n) for _, n in group if _address_street(n)}
|
||||||
|
if len(phones) > 1:
|
||||||
|
defects.add("P0", "L4", "NAP_PHONE_MISMATCH",
|
||||||
|
f"Business '{name}' has conflicting telephone values across "
|
||||||
|
f"entries: {sorted(phones)}.", entry_id="(dataset)")
|
||||||
|
if len(streets) > 1:
|
||||||
|
defects.add("P0", "L4", "NAP_ADDRESS_MISMATCH",
|
||||||
|
f"Business '{name}' has conflicting streetAddress values across "
|
||||||
|
f"entries: {sorted(streets)}.", entry_id="(dataset)")
|
||||||
|
|
||||||
|
# ---- @id duplicates + dangling references (P1) ----
|
||||||
|
defined = {} # @id -> list of definition dicts (walked across all docs)
|
||||||
|
referenced = set() # @id values used purely as references
|
||||||
|
for _, parsed in parsed_docs:
|
||||||
|
_walk_ids(parsed, defined, referenced)
|
||||||
|
for nid, defs in defined.items():
|
||||||
|
if len(defs) > 1:
|
||||||
|
# duplicate only matters if the definitions actually differ
|
||||||
|
shapes = {json.dumps(n, sort_keys=True, ensure_ascii=False) for n in defs}
|
||||||
|
if len(shapes) > 1:
|
||||||
|
defects.add("P1", "L4", "DUPLICATE_ID",
|
||||||
|
f"@id {nid!r} is defined {len(defs)} times with differing content.",
|
||||||
|
entry_id="(dataset)")
|
||||||
|
for nid in sorted(referenced - set(defined)):
|
||||||
|
defects.add("P1", "L4", "DANGLING_ID",
|
||||||
|
f"@id reference {nid!r} points to a node that is never defined.",
|
||||||
|
entry_id="(dataset)")
|
||||||
|
|
||||||
|
# ---- swapped / out-of-range geo (P1) ----
|
||||||
|
g = rules["geo"]
|
||||||
|
for entry, node in node_index:
|
||||||
|
if type_of(node) != "GeoCoordinates":
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
lat = float(first_text(node.get("latitude")))
|
||||||
|
lon = float(first_text(node.get("longitude")))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
lat_ok = g["lat_min"] <= lat <= g["lat_max"]
|
||||||
|
lon_ok = g["lon_min"] <= lon <= g["lon_max"]
|
||||||
|
if lat_ok and lon_ok:
|
||||||
|
continue # both in valid global range
|
||||||
|
# Invalid — distinguish a clean transposition from plain garbage.
|
||||||
|
swap_ok = (g["lat_min"] <= lon <= g["lat_max"]) and (g["lon_min"] <= lat <= g["lon_max"])
|
||||||
|
if swap_ok:
|
||||||
|
defects.add("P1", "L4", "GEO_SWAPPED",
|
||||||
|
f"GeoCoordinates look transposed (latitude={lat}, longitude={lon}) "
|
||||||
|
"— swapping them yields valid coordinates.",
|
||||||
|
entry["entry_id"], entry["url"], "GeoCoordinates")
|
||||||
|
else:
|
||||||
|
defects.add("P1", "L4", "GEO_OUT_OF_RANGE",
|
||||||
|
f"GeoCoordinates out of range (latitude={lat}, longitude={lon}).",
|
||||||
|
entry["entry_id"], entry["url"], "GeoCoordinates")
|
||||||
|
|
||||||
|
# ---- duplicate descriptions across entries (P1) ----
|
||||||
|
desc_groups = defaultdict(set)
|
||||||
|
for entry, node in node_index:
|
||||||
|
d = first_text(node.get("description"))
|
||||||
|
if isinstance(d, str) and len(d.strip()) >= 30:
|
||||||
|
desc_groups[d.strip()].add(entry["entry_id"])
|
||||||
|
for desc, eids in desc_groups.items():
|
||||||
|
if len(eids) >= 3:
|
||||||
|
defects.add("P1", "L4", "DUPLICATE_DESCRIPTION",
|
||||||
|
f"Identical description reused across {len(eids)} entries "
|
||||||
|
f"(e.g. {sorted(eids)[:3]}): {desc[:50]!r}…", entry_id="(dataset)")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Orchestration + output
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def run(entries, rules, inventory, strict, no_recommended):
|
||||||
|
defects = DefectLog()
|
||||||
|
if inventory is not None:
|
||||||
|
layer0_coverage(entries, inventory, defects)
|
||||||
|
|
||||||
|
node_index = []
|
||||||
|
parsed_docs = []
|
||||||
|
valid_entries = 0
|
||||||
|
for entry in entries:
|
||||||
|
parsed = layer1_syntax(entry, rules, defects)
|
||||||
|
if parsed is None:
|
||||||
|
continue
|
||||||
|
valid_entries += 1
|
||||||
|
parsed_docs.append((entry, parsed))
|
||||||
|
for node in iter_typed_nodes(parsed):
|
||||||
|
layer2_vocabulary(node, rules, defects, entry["entry_id"], entry["url"], strict)
|
||||||
|
layer3_richresult(node, rules, defects, entry["entry_id"], entry["url"],
|
||||||
|
no_recommended)
|
||||||
|
node_index.append((entry, node))
|
||||||
|
|
||||||
|
layer4_consistency(node_index, parsed_docs, rules, defects)
|
||||||
|
return defects, valid_entries, len(node_index)
|
||||||
|
|
||||||
|
|
||||||
|
def write_outputs(defects, outdir, meta):
|
||||||
|
outdir = Path(outdir)
|
||||||
|
outdir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
# defect_log.csv — the client-facing triage artifact
|
||||||
|
fields = ["entry_id", "url", "node_type", "layer", "code", "severity",
|
||||||
|
"message", "status", "owner", "note"]
|
||||||
|
rows = sorted(defects.rows, key=lambda r: (SEVERITY_ORDER[r["severity"]],
|
||||||
|
r["layer"], r["code"]))
|
||||||
|
with open(outdir / "defect_log.csv", "w", newline="", encoding="utf-8-sig") as f:
|
||||||
|
w = csv.DictWriter(f, fieldnames=fields)
|
||||||
|
w.writeheader()
|
||||||
|
w.writerows(rows)
|
||||||
|
|
||||||
|
counts = defects.counts()
|
||||||
|
gate = "PASS" if counts["P0"] == 0 else "FAIL"
|
||||||
|
by_code = Counter((r["severity"], r["code"]) for r in defects.rows)
|
||||||
|
|
||||||
|
# results.json — machine-readable
|
||||||
|
results = {
|
||||||
|
"summary": {**meta, **counts, "total": len(rows), "gate": gate},
|
||||||
|
"by_code": [{"severity": s, "code": c, "count": n}
|
||||||
|
for (s, c), n in by_code.most_common()],
|
||||||
|
"defects": rows,
|
||||||
|
}
|
||||||
|
(outdir / "results.json").write_text(
|
||||||
|
json.dumps(results, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
|
|
||||||
|
# report.md — human summary
|
||||||
|
lines = [
|
||||||
|
"# Schema Validation Report", "",
|
||||||
|
f"- Entries read: **{meta['entries']}** | parsed OK: **{meta['valid_entries']}** "
|
||||||
|
f"| nodes checked: **{meta['nodes']}**",
|
||||||
|
f"- Defects: **P0 {counts['P0']}** · **P1 {counts['P1']}** · **P2 {counts['P2']}** "
|
||||||
|
f"(total {len(rows)})",
|
||||||
|
"",
|
||||||
|
f"## Gate: **{gate}**",
|
||||||
|
("> ✅ Zero P0 — entries may advance to client review."
|
||||||
|
if gate == "PASS" else
|
||||||
|
"> ⛔ P0 present — these entries must NOT reach client review. Fix P0 first."),
|
||||||
|
"",
|
||||||
|
"## Defects by code", "",
|
||||||
|
"| Severity | Code | Count |", "|---|---|---|",
|
||||||
|
]
|
||||||
|
for (sev, code), n in by_code.most_common():
|
||||||
|
lines.append(f"| {sev} | {code} | {n} |")
|
||||||
|
|
||||||
|
p0 = [r for r in rows if r["severity"] == "P0"]
|
||||||
|
if p0:
|
||||||
|
lines += ["", "## P0 blockers (top 15)", "",
|
||||||
|
"| Entry | Type | Code | Message |", "|---|---|---|---|"]
|
||||||
|
for r in p0[:15]:
|
||||||
|
msg = r["message"].replace("|", "\\|")
|
||||||
|
lines.append(f"| {r['entry_id']} | {r['node_type']} | {r['code']} | {msg} |")
|
||||||
|
|
||||||
|
lines += ["", "## Next step",
|
||||||
|
("Triage P1 in `defect_log.csv`; client reviews the clean entries against this report."
|
||||||
|
if gate == "PASS" else
|
||||||
|
"Assign and fix every P0, re-run the validator, and only then open client review."),
|
||||||
|
""]
|
||||||
|
(outdir / "report.md").write_text("\n".join(lines), encoding="utf-8")
|
||||||
|
return gate, counts
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv=None):
|
||||||
|
ap = argparse.ArgumentParser(description="5-layer offline JSON-LD schema validator.")
|
||||||
|
ap.add_argument("dataset", nargs="?", help="xlsx/csv/jsonl/json file or a directory")
|
||||||
|
ap.add_argument("--url-list", help="canonical URL inventory (xlsx/csv/txt) → enables Layer 0")
|
||||||
|
ap.add_argument("--out", default="schema_qa_out", help="output directory")
|
||||||
|
ap.add_argument("--strict", action="store_true",
|
||||||
|
help="unexpected props on known types → P1; unknown types → P1")
|
||||||
|
ap.add_argument("--no-recommended", action="store_true",
|
||||||
|
help="drop L3 recommended (P2) findings — highest-signal gate")
|
||||||
|
ap.add_argument("--live", nargs="+", metavar="URL",
|
||||||
|
help="Mode B: validate live URLs (extract embedded JSON-LD)")
|
||||||
|
ap.add_argument("--rules", default=str(RULES_DEFAULT), help="path to schema_rules.json")
|
||||||
|
args = ap.parse_args(argv)
|
||||||
|
|
||||||
|
if not args.dataset and not args.live:
|
||||||
|
ap.error("provide a DATASET path or --live URL ...")
|
||||||
|
|
||||||
|
rules = json.loads(Path(args.rules).read_text(encoding="utf-8"))
|
||||||
|
|
||||||
|
try:
|
||||||
|
entries = load_entries(args.dataset, args.live)
|
||||||
|
except (ValueError, FileNotFoundError) as exc:
|
||||||
|
print(f"ERROR loading input: {exc}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
inventory = load_url_inventory(args.url_list) if args.url_list else None
|
||||||
|
|
||||||
|
defects, valid_entries, nodes = run(entries, rules, inventory,
|
||||||
|
args.strict, args.no_recommended)
|
||||||
|
meta = {"entries": len(entries), "valid_entries": valid_entries, "nodes": nodes,
|
||||||
|
"mode": "B-live" if args.live else "A-dataset", "strict": args.strict,
|
||||||
|
"coverage": inventory is not None}
|
||||||
|
gate, counts = write_outputs(defects, args.out, meta)
|
||||||
|
|
||||||
|
print(f"[{gate}] entries={len(entries)} nodes={nodes} "
|
||||||
|
f"P0={counts['P0']} P1={counts['P1']} P2={counts['P2']} → {args.out}/")
|
||||||
|
# Exit 1 when the gate fails so CI and `&&` chains stop on P0.
|
||||||
|
return 0 if gate == "PASS" else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
# 구조화 데이터 QA 리포트 — {{프로젝트명}}
|
||||||
|
|
||||||
|
> 클라이언트 검토용. **원본 JSON이 아니라 결함 리포트를 검토합니다.**
|
||||||
|
> 이 리포트에 오른 엔트리는 모두 기계 검증(Layer 0–4)을 통과한 **P0 0건** 상태입니다.
|
||||||
|
|
||||||
|
| 항목 | 값 |
|
||||||
|
|---|---|
|
||||||
|
| 데이터셋 | `{{dataset_파일명}}` |
|
||||||
|
| 검증 일시 | {{YYYY-MM-DD HH:MM}} |
|
||||||
|
| 검증 모드 | A — Dataset QA (배포 전) / B — Live audit (배포 후) |
|
||||||
|
| 엔트리 수 | {{entries}} (파싱 성공 {{valid_entries}}, 노드 {{nodes}}) |
|
||||||
|
| **게이트** | **{{PASS / FAIL}}** (PASS = P0 0건) |
|
||||||
|
| 결함 | P0 {{n}} · P1 {{n}} · P2 {{n}} |
|
||||||
|
| Audit ID | SCHEMA-{{YYYYMMDD}}-{{NNN}} |
|
||||||
|
|
||||||
|
## 1. 한눈에 보기
|
||||||
|
|
||||||
|
- ✅ **검토 가능 엔트리**: P0 0건을 통과한 {{n}}개 — 아래 판단 항목만 확인해 주세요.
|
||||||
|
- ⛔ **보류 엔트리**(있다면): P0 {{n}}건으로 검토 대상에서 제외. 수정 후 재검증합니다.
|
||||||
|
- 이번 검토에서 **사람의 판단이 필요한 것**은 기계가 잡지 못하는 두 가지뿐입니다:
|
||||||
|
1. 페이지에 맞는 스키마 **타입**이 선택되었는가
|
||||||
|
2. 표시되는 **문구(설명·이름)**가 사실과 정확히 일치하는가
|
||||||
|
|
||||||
|
## 2. 결함 요약 (코드별)
|
||||||
|
|
||||||
|
| 심각도 | 코드 | 건수 | 의미 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| P0 | {{CODE}} | {{n}} | {{한 줄 설명}} |
|
||||||
|
| P1 | {{CODE}} | {{n}} | {{한 줄 설명}} |
|
||||||
|
| P2 | {{CODE}} | {{n}} | {{한 줄 설명}} |
|
||||||
|
|
||||||
|
> 코드 정의: `references/defect-taxonomy.md`. 전체 목록: 첨부 `defect_log.csv`.
|
||||||
|
|
||||||
|
## 3. P0 블로커 (있을 경우 — 검토 전 수정 필수)
|
||||||
|
|
||||||
|
| 엔트리 | 타입 | 코드 | 내용 | 담당 | 상태 |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| {{entry_id}} | {{type}} | {{CODE}} | {{message}} | {{owner}} | open |
|
||||||
|
|
||||||
|
## 4. 클라이언트 확인 요청 (판단 항목)
|
||||||
|
|
||||||
|
기계가 통과시킨 엔트리 중, 사람의 확인이 필요한 항목입니다.
|
||||||
|
|
||||||
|
| # | URL / 페이지 | 확인 요청 | 비고 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | {{url}} | 이 페이지에 `{{@type}}` 타입이 맞습니까? | |
|
||||||
|
| 2 | {{url}} | 설명/이름 문구가 정확합니까? | |
|
||||||
|
|
||||||
|
## 5. 다음 단계
|
||||||
|
|
||||||
|
- **PASS인 경우**: 위 4번 판단 항목 확정 → 배포 단계(G4 안정화)로 이동, 샘플을 Google
|
||||||
|
Rich Results Test로 최종 확인.
|
||||||
|
- **FAIL인 경우**: P0 담당 배정 → 수정 → 재검증(`validate_schema.py`) → 본 리포트 갱신.
|
||||||
|
- P1 처리 방침(수정/수용)은 `decision-log.md`에 기록합니다.
|
||||||
|
|
||||||
|
---
|
||||||
|
*생성: 16-seo-schema-validator · 첨부: `report.md`, `defect_log.csv`, `results.json`*
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# P1 Decision Log — {{프로젝트명}}
|
||||||
|
|
||||||
|
P0 is non-negotiable: every P0 is fixed before launch (the gate enforces it). **P1 is
|
||||||
|
where judgement lives** — some P1s get fixed, some get consciously accepted. This log
|
||||||
|
records *which, by whom, and why*, so an accepted P1 is a decision on the record, not a
|
||||||
|
silently ignored defect. It is the G3 (테스트) deliverable.
|
||||||
|
|
||||||
|
## How to use
|
||||||
|
|
||||||
|
1. Open `defect_log.csv`, filter to `severity = P1`.
|
||||||
|
2. For each P1 (or each group of identical P1s), add a row below.
|
||||||
|
3. Decision is one of: **Fix** (will correct before launch) / **Accept** (ship as-is,
|
||||||
|
with rationale) / **Defer** (post-launch backlog).
|
||||||
|
4. An `Accept`/`Defer` needs a named approver. `Fix` needs an owner + target date.
|
||||||
|
5. Re-run the validator after the fixes; confirm the fixed P1s are gone.
|
||||||
|
|
||||||
|
## Log
|
||||||
|
|
||||||
|
| # | Code | Entry / scope | Summary | Decision | Owner / Approver | Target / Date | Rationale |
|
||||||
|
|---|---|---|---|---|---|---|---|
|
||||||
|
| 1 | {{CODE}} | {{entry_id or (dataset)}} | {{one line}} | Fix / Accept / Defer | {{name}} | {{YYYY-MM-DD}} | {{why}} |
|
||||||
|
| 2 | | | | | | | |
|
||||||
|
| 3 | | | | | | | |
|
||||||
|
|
||||||
|
## Standing decisions (apply to all entries unless overridden)
|
||||||
|
|
||||||
|
Record cross-cutting calls once here instead of per row — e.g. "MISSING_RECOMMENDED for
|
||||||
|
`starRating` is accepted group-wide: not contractually rated." Reduces log noise.
|
||||||
|
|
||||||
|
| Code | Standing decision | Approver | Date |
|
||||||
|
|---|---|---|---|
|
||||||
|
| {{CODE}} | Accept group-wide — {{reason}} | {{name}} | {{YYYY-MM-DD}} |
|
||||||
|
|
||||||
|
## Sign-off
|
||||||
|
|
||||||
|
| Stage gate | Condition | Confirmed by | Date |
|
||||||
|
|---|---|---|---|
|
||||||
|
| G3 테스트 | All P1 triaged (Fix/Accept/Defer), decisions logged above | {{name}} | {{date}} |
|
||||||
|
| G4 안정화 | P0 = 0, all "Fix" P1 closed, online validator green on sample | {{name}} | {{date}} |
|
||||||
Reference in New Issue
Block a user