feat(seo-skills): multi-backend Data Source Selection (#7)
Replaces single-vendor (Ahrefs-only) tool defaults with a per-task backend menu across all 14 SEO skills. Each skill now lists every capable MCP in allowed-tools and documents how to pick between Semrush, Ahrefs, OurSEO Agent (CLI + MCP), DataForSEO, and GSC in its SKILL.md Data Source Selection section. Tool stubs (~40 new files) populated per skill with capability deltas, call patterns, and explicit "not for this skill when" callouts so the menu is self-correcting. Skills affected: 19-keyword-strategy, 20-serp-analysis, 21-position-tracking, 22-link-building, 23-content-strategy, 24-ecommerce, 25-kpi-framework, 26-international, 27-ai-visibility, 28-knowledge-graph, 31-competitor-intel, 32-crawl-budget, 33-migration-planner, 34-reporting-dashboard. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Yim
@@ -1,17 +1,33 @@
|
||||
# Ahrefs MCP Tools
|
||||
# Ahrefs MCP
|
||||
|
||||
## site-explorer-pages-history
|
||||
In crawl-budget work, Ahrefs Site Audit surfaces **orphans, redirect chains, parameter URLs, and soft 404s** without running your own crawler. Useful when an Ahrefs Site Audit project already exists for the domain.
|
||||
|
||||
Get historical page data for a domain to compare indexed pages with crawled pages.
|
||||
## Key endpoints
|
||||
|
||||
- `site-audit-projects` — list available projects
|
||||
- `site-audit-issues` — current crawl-budget-relevant issues (redirect chains, broken links, parameter URLs)
|
||||
- `site-audit-page-explorer` — per-URL issue surfacing
|
||||
- `site-audit-page-content` — content-level signals (thin content, duplicates)
|
||||
- `site-explorer-pages-history` — page count over time vs. observed bot crawls
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
mcp__ahrefs__site-explorer-pages-history
|
||||
mcp__ahrefs__site-audit-projects()
|
||||
mcp__ahrefs__site-audit-issues(project_id="<id>")
|
||||
mcp__ahrefs__site-audit-page-explorer(project_id="<id>")
|
||||
mcp__ahrefs__site-explorer-pages-history(target="<domain>", history="weekly")
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `target` (string, required): Domain or URL to analyze
|
||||
- `date_from` (string): Start date (YYYY-MM-DD)
|
||||
- `date_to` (string): End date (YYYY-MM-DD)
|
||||
- `mode` (string): "domain", "prefix", "exact", "subdomains"
|
||||
## Strengths for this skill
|
||||
|
||||
**Use case**: Compare Ahrefs indexed page counts with server log crawl data to identify indexing gaps and crawl budget inefficiencies.
|
||||
- Pre-computed orphan detection and redirect-chain surfacing.
|
||||
- Pages-history helps validate "are we crawling pages that don't exist anymore?"
|
||||
- Cross-reference Ahrefs indexed-page count with server log crawl data to spot indexing gaps.
|
||||
|
||||
## Not for this skill when
|
||||
|
||||
- **Actual bot behaviour at the origin** — only server access logs show this. Use `scripts/log_parser.py` + `scripts/crawl_budget_analyzer.py`.
|
||||
- **First-party Googlebot crawl stats** — GSC Coverage / Crawl Stats reports.
|
||||
- **Naver / Daum / Kakao bot analysis** — Ahrefs covers Googlebot well but underrepresents Korean bots; rely on logs.
|
||||
- **No Ahrefs project exists for the domain** — fall back to Semrush `siteaudit_research` or OurSEO crawl.
|
||||
|
||||
45
custom-skills/32-seo-crawl-budget/desktop/tools/gsc.md
Normal file
45
custom-skills/32-seo-crawl-budget/desktop/tools/gsc.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# Google Search Console
|
||||
|
||||
GSC's Coverage + Crawl Stats reports are the **first-party Googlebot view** — what Google considers crawled, indexed, or excluded. Pair with server access logs (the origin view) and OurSEO crawl (the published view) for a complete crawl-budget audit.
|
||||
|
||||
## CLI access
|
||||
|
||||
```bash
|
||||
our research search-console queries --site sc-domain:<domain> --days 28
|
||||
our research search-console pages --site sc-domain:<domain> --days 28
|
||||
```
|
||||
|
||||
`sc-domain:` prefix for Domain-verified properties.
|
||||
|
||||
Note: full **Coverage** and **Crawl Stats** report data is mostly UI-only in GSC — the API exposes performance (queries / pages) but limited crawl-stats fields. For exhaustive crawl analysis, export via the GSC UI and parse alongside log output.
|
||||
|
||||
## Ahrefs GSC alternates
|
||||
|
||||
When the site is connected as an Ahrefs project:
|
||||
|
||||
```
|
||||
mcp__ahrefs__gsc-pages(project_id="<id>")
|
||||
mcp__ahrefs__gsc-pages-history(project_id="<id>")
|
||||
mcp__ahrefs__gsc-positions-history(project_id="<id>")
|
||||
```
|
||||
|
||||
## How GSC pairs with the rest
|
||||
|
||||
| Source | Shows |
|
||||
|---|---|
|
||||
| Server logs | What hit the origin (bot behaviour at HTTP level) |
|
||||
| OurSEO crawl | What URLs exist on the site today |
|
||||
| Sitemap | What URLs the site claims to publish |
|
||||
| **GSC** | **What Google considers indexed / excluded** |
|
||||
|
||||
The intersection (and differences) reveal:
|
||||
- **Crawled-not-indexed**: Google saw it, won't index — often thin content or duplicates.
|
||||
- **Indexed-not-crawled-recently**: bot budget moved elsewhere.
|
||||
- **Discovered-not-crawled**: Google knows about the URL but hasn't fetched it.
|
||||
|
||||
## Not for this skill when
|
||||
|
||||
- **Korean bots** (Yeti, Daumoa) — GSC is Google-only.
|
||||
- **Bot User-Agent + IP analysis** — server logs.
|
||||
|
||||
Gotcha: `our-claude-skills/custom-skills/15-seo-search-console/code/gotcha/gsc-cli-integration.md`.
|
||||
53
custom-skills/32-seo-crawl-budget/desktop/tools/ourseo.md
Normal file
53
custom-skills/32-seo-crawl-budget/desktop/tools/ourseo.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# OurSEO Agent (CLI + MCP)
|
||||
|
||||
OurSEO contributes the **crawled URL inventory** (vs. server log observations), **index status check**, and **distributed crawl** for large sites. Pair with `scripts/log_parser.py` + `scripts/crawl_budget_analyzer.py` for the full crawl-budget picture.
|
||||
|
||||
## CLI commands (Claude Code)
|
||||
|
||||
```bash
|
||||
# URL inventory crawl
|
||||
our collect crawl https://<site> --max-pages 5000
|
||||
our collect crawl https://<site> --stealth --backend scrapy
|
||||
|
||||
# Distributed crawl for large sites
|
||||
our collect distributed https://<site> --workers 8 --max-pages 50000
|
||||
|
||||
# Index status (Google index coverage)
|
||||
our research google index --domain <site>
|
||||
|
||||
# Technical audit (redirects, canonicals, etc.)
|
||||
our audit tech https://<site>
|
||||
|
||||
# MySQL batch analysis on crawl results
|
||||
our analyze mysql-batch --session <id>
|
||||
our analyze mysql-batch --session <id> --export issues --format excel
|
||||
```
|
||||
|
||||
## MCP tools (Claude Desktop)
|
||||
|
||||
```
|
||||
mcp__ourseo__crawl_website(url="<site>", max_pages=5000)
|
||||
mcp__ourseo__check_index(domain="<site>")
|
||||
mcp__ourseo__audit_page(url="<url>", audit_type="tech")
|
||||
```
|
||||
|
||||
## Strengths for this skill
|
||||
|
||||
- **Configurable crawler** — backend choice (aiohttp, Scrapy), stealth mode, resume support.
|
||||
- **Distributed crawl** scales to 50K+ URL audits.
|
||||
- **`check_index`** uses Google PSE to spot-check indexed-or-not per URL.
|
||||
- Crawl output drops into MySQL workspace — easy to diff against log-derived URL sets.
|
||||
|
||||
## How OurSEO pairs with log analysis
|
||||
|
||||
1. **Logs** (`scripts/log_parser.py`) → what bots actually crawled at the origin
|
||||
2. **OurSEO crawl** → what URLs exist on the site today
|
||||
3. **Sitemap** → what URLs the site claims to publish
|
||||
4. **GSC** → what Google considers indexed
|
||||
5. **Diff all four** — orphan, ghost, and waste candidates emerge from set differences
|
||||
|
||||
## Not for this skill when
|
||||
|
||||
- **Actual bot User-Agent + frequency** — only logs show this.
|
||||
- **Per-bot status distribution** — only logs.
|
||||
- **First-party Googlebot crawl stats** — GSC.
|
||||
30
custom-skills/32-seo-crawl-budget/desktop/tools/semrush.md
Normal file
30
custom-skills/32-seo-crawl-budget/desktop/tools/semrush.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Semrush MCP
|
||||
|
||||
Alternative third-party site audit when no Ahrefs project exists for the domain.
|
||||
|
||||
Call pattern: discovery → `get_report_schema` → `execute_report`.
|
||||
|
||||
## Key report
|
||||
|
||||
- `siteaudit_research` — site-wide audit catching redirect chains, parameter URLs, thin content, broken links
|
||||
|
||||
## Example
|
||||
|
||||
```
|
||||
mcp__semrush__siteaudit_research(query="<domain>", database="us")
|
||||
mcp__semrush__get_report_schema(report_id="<id>")
|
||||
mcp__semrush__execute_report(report_id="<id>", params={...})
|
||||
```
|
||||
|
||||
## When Semrush is useful here
|
||||
|
||||
- No Ahrefs Site Audit project — Semrush is the quickest alternative.
|
||||
- User is already in Semrush context.
|
||||
- Cross-vendor verification of orphan / redirect-chain counts.
|
||||
|
||||
## Not for this skill when
|
||||
|
||||
- **Actual bot behaviour at the origin** — server access logs (Nginx/Apache/CloudFront) only.
|
||||
- **First-party Googlebot crawl stats** — GSC.
|
||||
- **Korean bot analysis** (Yeti/Naver, Daumoa/Kakao) — Semrush underrepresents these. Use logs.
|
||||
- **An Ahrefs project already exists** — prefer Ahrefs `site-audit-*` for historical continuity.
|
||||
Reference in New Issue
Block a user