55 lines
2.5 KiB
Markdown
55 lines
2.5 KiB
Markdown
---
|
|
description: Crawl budget optimization and log analysis
|
|
---
|
|
|
|
# SEO Crawl Budget
|
|
|
|
Server access log analysis, bot profiling, and crawl budget waste identification.
|
|
|
|
## Triggers
|
|
- "crawl budget", "log analysis", "크롤 예산"
|
|
|
|
## Capabilities
|
|
|
|
1. **Log Parsing** - Parse Nginx, Apache, CloudFront access logs (streaming for >1GB files)
|
|
2. **Bot Identification** - Googlebot, Yeti/Naver, Bingbot, Daumoa/Kakao, and others by User-Agent
|
|
3. **Per-Bot Profiling** - Crawl frequency, depth distribution, status codes, crawl patterns
|
|
4. **Waste Detection** - Parameter URLs, low-value pages, redirect chains, soft 404s, duplicate URLs
|
|
5. **Orphan Pages** - Pages in sitemap but never crawled, crawled but not in sitemap
|
|
6. **Optimization Plan** - robots.txt suggestions, URL parameter handling, noindex recommendations
|
|
|
|
## Scripts
|
|
|
|
```bash
|
|
# Parse Nginx access log
|
|
python /Users/ourdigital/Project/our-claude-skills/custom-skills/32-seo-crawl-budget/code/scripts/log_parser.py \
|
|
--log-file /var/log/nginx/access.log --json
|
|
|
|
# Parse Apache log, filter by Googlebot
|
|
python /Users/ourdigital/Project/our-claude-skills/custom-skills/32-seo-crawl-budget/code/scripts/log_parser.py \
|
|
--log-file /var/log/apache2/access.log --format apache --bot googlebot --json
|
|
|
|
# Parse gzipped log in streaming mode
|
|
python /Users/ourdigital/Project/our-claude-skills/custom-skills/32-seo-crawl-budget/code/scripts/log_parser.py \
|
|
--log-file access.log.gz --streaming --json
|
|
|
|
# Full crawl budget analysis with sitemap comparison
|
|
python /Users/ourdigital/Project/our-claude-skills/custom-skills/32-seo-crawl-budget/code/scripts/crawl_budget_analyzer.py \
|
|
--log-file access.log --sitemap https://example.com/sitemap.xml --json
|
|
|
|
# Waste identification only
|
|
python /Users/ourdigital/Project/our-claude-skills/custom-skills/32-seo-crawl-budget/code/scripts/crawl_budget_analyzer.py \
|
|
--log-file access.log --scope waste --json
|
|
|
|
# Orphan page detection
|
|
python /Users/ourdigital/Project/our-claude-skills/custom-skills/32-seo-crawl-budget/code/scripts/crawl_budget_analyzer.py \
|
|
--log-file access.log --sitemap https://example.com/sitemap.xml --scope orphans --json
|
|
```
|
|
|
|
## Output
|
|
- Bot request counts, status code distribution, top crawled URLs per bot
|
|
- Crawl waste breakdown (parameter URLs, redirects, soft 404s, duplicates)
|
|
- Orphan page lists (in sitemap not crawled, crawled not in sitemap)
|
|
- Efficiency score (0-100) with optimization recommendations
|
|
- Saved to Notion SEO Audit Log (Category: Crawl Budget, Audit ID: CRAWL-YYYYMMDD-NNN)
|