ourdigital/our-claude-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(seo-skills): multi-backend Data Source Selection (#7)

Browse Source 
Replaces single-vendor (Ahrefs-only) tool defaults with a per-task
backend menu across all 14 SEO skills. Each skill now lists every
capable MCP in allowed-tools and documents how to pick between
Semrush, Ahrefs, OurSEO Agent (CLI + MCP), DataForSEO, and GSC
in its SKILL.md Data Source Selection section.

Tool stubs (~40 new files) populated per skill with capability
deltas, call patterns, and explicit "not for this skill when"
callouts so the menu is self-correcting.

Skills affected: 19-keyword-strategy, 20-serp-analysis,
21-position-tracking, 22-link-building, 23-content-strategy,
24-ecommerce, 25-kpi-framework, 26-international, 27-ai-visibility,
28-knowledge-graph, 31-competitor-intel, 32-crawl-budget,
33-migration-planner, 34-reporting-dashboard.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Yim
2026-05-14 03:15:32 +09:00
committed by GitHub
parent 1ca84f67ed
commit e527fb4b0f
71 changed files with 2647 additions and 258 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
custom-skills/32-seo-crawl-budget/desktop/SKILL.md
View File

@@ -85,13 +85,75 @@ Save Korean-language report to SEO Audit Log database.
| Found Date | Date | Analysis date (YYYY-MM-DD) |
| Audit ID | Rich Text | Format: CRAWL-YYYYMMDD-NNN |
## Data Sources
## Data Source Selection
| Source | Purpose |
|--------|---------|
| `our-seo-agent` CLI | Future primary data source; use `--input` for pre-fetched JSON |
| Notion MCP | Save audit report to database |
| WebSearch | Current bot documentation and best practices |
Crawl-budget analysis is **primarily log-based** — the authoritative signal lives in server access logs that this skill's local Python scripts parse. API backends supplement that with crawled URL inventories, index status, and third-party site-audit views. Pick per data class.
| Backend | Best for | Notes |
|---|---|---|
| **Server access logs** (via `scripts/log_parser.py` + `crawl_budget_analyzer.py`) | **Primary** — bot identification, request volume, status code distribution, waste detection, orphan pages | Requires actual server logs from the user (Nginx / Apache / CloudFront). No MCP substitute. |
| **OurSEO** (CLI + MCP) | **Default** for crawl-derived URL inventory, index status, redirect chains | CLI: `our collect crawl`, `our research google index`, `our audit tech`. MCP: `mcp__ourseo__crawl_website`, `mcp__ourseo__check_index`. Distributed crawl for large sites: `our collect distributed --workers N`. |
| **Ahrefs MCP** (`mcp__ahrefs__*`) | Orphan detection, redirect chain analysis via Ahrefs site audit | `site-audit-issues`, `site-audit-page-explorer`, `site-audit-page-content`. Useful when an Ahrefs project already exists for the domain. |
| **Semrush MCP** (`mcp__semrush__*`) | Alternative site audit when Ahrefs project doesn't exist | `siteaudit_research` → `get_report_schema` → `execute_report`. |
| **GSC** (via `our research search-console`) | First-party Googlebot crawl stats (Coverage / Crawl Stats reports) | Required for ground-truth Googlebot behaviour — server logs show what hit the origin, GSC shows what Google considers crawled. |
### How to pick
1. **User named a backend explicitly** → use it.
2. **User preference memory** — read `feedback_seo_tool_preferences.md`; honor the task-type default.
3. **Server access logs are available** → ALWAYS process them first (`log_parser.py` + `crawl_budget_analyzer.py`). They are the only source for actual bot behaviour at the origin.
4. **Sitemap vs. crawl comparison needed** → OurSEO `crawl_website` for URL inventory; cross-reference with logs.
5. **No logs available** → fall back to OurSEO crawl + GSC Coverage + Ahrefs/Semrush site audit. State this limitation explicitly in the report.
6. **Default**: server logs (when present) + **OurSEO crawl + index check** for URL inventory.
7. **Still ambiguous + non-trivial** → ask once via `AskUserQuestion`.
### Backend call patterns
**Local log analysis (primary):**
```bash
python scripts/log_parser.py --log-file access.log --json
python scripts/log_parser.py --log-file access.log.gz --streaming --json
python scripts/log_parser.py --log-file access.log --bot googlebot --json
python scripts/crawl_budget_analyzer.py --log-file access.log --sitemap https://<site>/sitemap.xml --json
python scripts/crawl_budget_analyzer.py --log-file access.log --scope waste --json
python scripts/crawl_budget_analyzer.py --log-file access.log --scope orphans --json
python scripts/crawl_budget_analyzer.py --log-file access.log --scope bots --json
```
**OurSEO CLI (URL inventory + index check):**
```bash
our collect crawl https://<site> --max-pages 5000
our collect distributed https://<site> --workers 8 --max-pages 50000
our research google index --domain <site>
our audit tech https://<site>
```
**OurSEO MCP (Claude Desktop):**
```
mcp__ourseo__crawl_website(url="<site>", max_pages=5000)
mcp__ourseo__check_index(domain="<site>")
```
**GSC (first-party crawl stats):**
```bash
our research search-console queries --site sc-domain:<site> --days 28
# See also: GSC Coverage / Crawl Stats reports (UI-only for some sections).
```
**Ahrefs MCP (third-party site audit):**
```
mcp__ahrefs__site-audit-projects()
mcp__ahrefs__site-audit-issues(project_id="<id>")
mcp__ahrefs__site-audit-page-explorer(project_id="<id>")
```
**Semrush MCP (alternative site audit):**
```
mcp__semrush__siteaudit_research(query="<site>", database="us")
```
Always record the chosen data source(s) in the report **Overview** so future audits can compare like-for-like — and explicitly state whether server logs were available.
## Output Format

10
custom-skills/32-seo-crawl-budget/desktop/skill.yaml
View File

@@ -1,8 +1,16 @@
name: seo-crawl-budget
description: |
Crawl budget optimization and log analysis. Triggers: crawl budget, log analysis, bot crawling, Googlebot, crawl waste, orphan pages, crawl efficiency.
# Allowed tools list every backend the skill can pull crawl/log data from.
# Per-task selection happens in SKILL.md > Data Source Selection — NOT here.
# Note: crawl-budget analysis is primarily LOG-based (local files), not API-based.
allowed-tools:
- mcp__ahrefs__*
- mcp__ourseo__* # default — crawl_website, check_index, audit_page
- mcp__ahrefs__* # site-audit-issues, site-audit-page-explorer (orphan, redirect chains)
- mcp__claude_ai_Ahrefs__* # Ahrefs (Claude.ai namespace)
- mcp__semrush__* # siteaudit_research (alternate site audit)
- Bash # `our` CLI + log_parser.py + crawl_budget_analyzer.py
- mcp__notion__*
- WebSearch
- WebFetch

36
custom-skills/32-seo-crawl-budget/desktop/tools/ahrefs.md
View File

@@ -1,17 +1,33 @@
# Ahrefs MCP Tools
# Ahrefs MCP
## site-explorer-pages-history
In crawl-budget work, Ahrefs Site Audit surfaces **orphans, redirect chains, parameter URLs, and soft 404s** without running your own crawler. Useful when an Ahrefs Site Audit project already exists for the domain.
Get historical page data for a domain to compare indexed pages with crawled pages.
## Key endpoints
- `site-audit-projects` — list available projects
- `site-audit-issues` — current crawl-budget-relevant issues (redirect chains, broken links, parameter URLs)
- `site-audit-page-explorer` — per-URL issue surfacing
- `site-audit-page-content` — content-level signals (thin content, duplicates)
- `site-explorer-pages-history` — page count over time vs. observed bot crawls
## Example
```
mcp__ahrefs__site-explorer-pages-history
mcp__ahrefs__site-audit-projects()
mcp__ahrefs__site-audit-issues(project_id="<id>")
mcp__ahrefs__site-audit-page-explorer(project_id="<id>")
mcp__ahrefs__site-explorer-pages-history(target="<domain>", history="weekly")
```
**Parameters**:
- `target` (string, required): Domain or URL to analyze
- `date_from` (string): Start date (YYYY-MM-DD)
- `date_to` (string): End date (YYYY-MM-DD)
- `mode` (string): "domain", "prefix", "exact", "subdomains"
## Strengths for this skill
**Use case**: Compare Ahrefs indexed page counts with server log crawl data to identify indexing gaps and crawl budget inefficiencies.
- Pre-computed orphan detection and redirect-chain surfacing.
- Pages-history helps validate "are we crawling pages that don't exist anymore?"
- Cross-reference Ahrefs indexed-page count with server log crawl data to spot indexing gaps.
## Not for this skill when
- **Actual bot behaviour at the origin** — only server access logs show this. Use `scripts/log_parser.py` + `scripts/crawl_budget_analyzer.py`.
- **First-party Googlebot crawl stats** — GSC Coverage / Crawl Stats reports.
- **Naver / Daum / Kakao bot analysis** — Ahrefs covers Googlebot well but underrepresents Korean bots; rely on logs.
- **No Ahrefs project exists for the domain** — fall back to Semrush `siteaudit_research` or OurSEO crawl.

45
custom-skills/32-seo-crawl-budget/desktop/tools/gsc.md Normal file
View File

@@ -0,0 +1,45 @@
# Google Search Console
GSC's Coverage + Crawl Stats reports are the **first-party Googlebot view** — what Google considers crawled, indexed, or excluded. Pair with server access logs (the origin view) and OurSEO crawl (the published view) for a complete crawl-budget audit.
## CLI access
```bash
our research search-console queries --site sc-domain:<domain> --days 28
our research search-console pages --site sc-domain:<domain> --days 28
```
`sc-domain:` prefix for Domain-verified properties.
Note: full **Coverage** and **Crawl Stats** report data is mostly UI-only in GSC — the API exposes performance (queries / pages) but limited crawl-stats fields. For exhaustive crawl analysis, export via the GSC UI and parse alongside log output.
## Ahrefs GSC alternates
When the site is connected as an Ahrefs project:
```
mcp__ahrefs__gsc-pages(project_id="<id>")
mcp__ahrefs__gsc-pages-history(project_id="<id>")
mcp__ahrefs__gsc-positions-history(project_id="<id>")
```
## How GSC pairs with the rest
| Source | Shows |
|---|---|
| Server logs | What hit the origin (bot behaviour at HTTP level) |
| OurSEO crawl | What URLs exist on the site today |
| Sitemap | What URLs the site claims to publish |
| **GSC** | **What Google considers indexed / excluded** |
The intersection (and differences) reveal:
- **Crawled-not-indexed**: Google saw it, won't index — often thin content or duplicates.
- **Indexed-not-crawled-recently**: bot budget moved elsewhere.
- **Discovered-not-crawled**: Google knows about the URL but hasn't fetched it.
## Not for this skill when
- **Korean bots** (Yeti, Daumoa) — GSC is Google-only.
- **Bot User-Agent + IP analysis** — server logs.
Gotcha: `our-claude-skills/custom-skills/15-seo-search-console/code/gotcha/gsc-cli-integration.md`.

53
custom-skills/32-seo-crawl-budget/desktop/tools/ourseo.md Normal file
View File

@@ -0,0 +1,53 @@
# OurSEO Agent (CLI + MCP)
OurSEO contributes the **crawled URL inventory** (vs. server log observations), **index status check**, and **distributed crawl** for large sites. Pair with `scripts/log_parser.py` + `scripts/crawl_budget_analyzer.py` for the full crawl-budget picture.
## CLI commands (Claude Code)
```bash
# URL inventory crawl
our collect crawl https://<site> --max-pages 5000
our collect crawl https://<site> --stealth --backend scrapy
# Distributed crawl for large sites
our collect distributed https://<site> --workers 8 --max-pages 50000
# Index status (Google index coverage)
our research google index --domain <site>
# Technical audit (redirects, canonicals, etc.)
our audit tech https://<site>
# MySQL batch analysis on crawl results
our analyze mysql-batch --session <id>
our analyze mysql-batch --session <id> --export issues --format excel
```
## MCP tools (Claude Desktop)
```
mcp__ourseo__crawl_website(url="<site>", max_pages=5000)
mcp__ourseo__check_index(domain="<site>")
mcp__ourseo__audit_page(url="<url>", audit_type="tech")
```
## Strengths for this skill
- **Configurable crawler** — backend choice (aiohttp, Scrapy), stealth mode, resume support.
- **Distributed crawl** scales to 50K+ URL audits.
- **`check_index`** uses Google PSE to spot-check indexed-or-not per URL.
- Crawl output drops into MySQL workspace — easy to diff against log-derived URL sets.
## How OurSEO pairs with log analysis
1. **Logs** (`scripts/log_parser.py`) → what bots actually crawled at the origin
2. **OurSEO crawl** → what URLs exist on the site today
3. **Sitemap** → what URLs the site claims to publish
4. **GSC** → what Google considers indexed
5. **Diff all four** — orphan, ghost, and waste candidates emerge from set differences
## Not for this skill when
- **Actual bot User-Agent + frequency** — only logs show this.
- **Per-bot status distribution** — only logs.
- **First-party Googlebot crawl stats** — GSC.

30
custom-skills/32-seo-crawl-budget/desktop/tools/semrush.md Normal file
View File

@@ -0,0 +1,30 @@
# Semrush MCP
Alternative third-party site audit when no Ahrefs project exists for the domain.
Call pattern: discovery → `get_report_schema` → `execute_report`.
## Key report
- `siteaudit_research` — site-wide audit catching redirect chains, parameter URLs, thin content, broken links
## Example
```
mcp__semrush__siteaudit_research(query="<domain>", database="us")
mcp__semrush__get_report_schema(report_id="<id>")
mcp__semrush__execute_report(report_id="<id>", params={...})
```
## When Semrush is useful here
- No Ahrefs Site Audit project — Semrush is the quickest alternative.
- User is already in Semrush context.
- Cross-vendor verification of orphan / redirect-chain counts.
## Not for this skill when
- **Actual bot behaviour at the origin** — server access logs (Nginx/Apache/CloudFront) only.
- **First-party Googlebot crawl stats** — GSC.
- **Korean bot analysis** (Yeti/Naver, Daumoa/Kakao) — Semrush underrepresents these. Use logs.
- **An Ahrefs project already exists** — prefer Ahrefs `site-audit-*` for historical continuity.