397fa2aa5d
* Fix SEO skill 34 bugs, Korean labels, and transition Ahrefs refs to our-seo-agent P0: Fix report_aggregator.py — wrong SKILL_REGISTRY[33] mapping, missing CATEGORY_WEIGHTS for 7 categories, and break bug in health score parsing that exited loop even on parse failure. P1: Remove VIEW tab references from skill 20, expand skill 32 docs, replace Ahrefs MCP references across all 16 skills (19-28, 31-34) with our-seo-agent CLI data source references. P2: Fix Korean labels in executive_report.py and dashboard_generator.py, add tenacity to base requirements, sync skill 34 base_client.py with canonical version from skill 12. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add Claude Code slash commands for SEO skills 19-34 and fix stale paths Create 14 new slash command files for skills 19-28, 31-34 so they appear as /seo-* commands in Claude Code. Also fix stale directory paths in 8 existing commands (skills 12-18, 29-30) that referenced pre-renumbering skill directories. Update .gitignore to track .claude/commands/ while keeping other .claude/ files ignored. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add 8 slash commands, enhance reference-curator with depth/output options - Add slash commands: ourdigital-brand-guide, notion-writer, notebooklm-agent, notebooklm-automation, notebooklm-studio, notebooklm-research, reference-curator, multi-agent-guide - Add --depth (light/standard/deep/full) with Firecrawl parameter mapping - Add --output with ~/Documents/reference-library/ default and user confirmation - Increase --max-sources default from 10 to 100 - Rename /reference-curator-pipeline to /reference-curator - Simplify web-crawler-orchestrator label to web-crawler in docs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
55 lines
2.2 KiB
Markdown
55 lines
2.2 KiB
Markdown
---
|
|
description: Crawl budget optimization and log analysis
|
|
---
|
|
|
|
# SEO Crawl Budget
|
|
|
|
Server access log analysis, bot profiling, and crawl budget waste identification.
|
|
|
|
## Triggers
|
|
- "crawl budget", "log analysis", "크롤 예산"
|
|
|
|
## Capabilities
|
|
|
|
1. **Log Parsing** - Parse Nginx, Apache, CloudFront access logs (streaming for >1GB files)
|
|
2. **Bot Identification** - Googlebot, Yeti/Naver, Bingbot, Daumoa/Kakao, and others by User-Agent
|
|
3. **Per-Bot Profiling** - Crawl frequency, depth distribution, status codes, crawl patterns
|
|
4. **Waste Detection** - Parameter URLs, low-value pages, redirect chains, soft 404s, duplicate URLs
|
|
5. **Orphan Pages** - Pages in sitemap but never crawled, crawled but not in sitemap
|
|
6. **Optimization Plan** - robots.txt suggestions, URL parameter handling, noindex recommendations
|
|
|
|
## Scripts
|
|
|
|
```bash
|
|
# Parse Nginx access log
|
|
python custom-skills/32-seo-crawl-budget/code/scripts/log_parser.py \
|
|
--log-file /var/log/nginx/access.log --json
|
|
|
|
# Parse Apache log, filter by Googlebot
|
|
python custom-skills/32-seo-crawl-budget/code/scripts/log_parser.py \
|
|
--log-file /var/log/apache2/access.log --format apache --bot googlebot --json
|
|
|
|
# Parse gzipped log in streaming mode
|
|
python custom-skills/32-seo-crawl-budget/code/scripts/log_parser.py \
|
|
--log-file access.log.gz --streaming --json
|
|
|
|
# Full crawl budget analysis with sitemap comparison
|
|
python custom-skills/32-seo-crawl-budget/code/scripts/crawl_budget_analyzer.py \
|
|
--log-file access.log --sitemap https://example.com/sitemap.xml --json
|
|
|
|
# Waste identification only
|
|
python custom-skills/32-seo-crawl-budget/code/scripts/crawl_budget_analyzer.py \
|
|
--log-file access.log --scope waste --json
|
|
|
|
# Orphan page detection
|
|
python custom-skills/32-seo-crawl-budget/code/scripts/crawl_budget_analyzer.py \
|
|
--log-file access.log --sitemap https://example.com/sitemap.xml --scope orphans --json
|
|
```
|
|
|
|
## Output
|
|
- Bot request counts, status code distribution, top crawled URLs per bot
|
|
- Crawl waste breakdown (parameter URLs, redirects, soft 404s, duplicates)
|
|
- Orphan page lists (in sitemap not crawled, crawled not in sitemap)
|
|
- Efficiency score (0-100) with optimization recommendations
|
|
- Saved to Notion SEO Audit Log (Category: Crawl Budget, Audit ID: CRAWL-YYYYMMDD-NNN)
|