feat(skills): bulk-add root SKILL.md to 61 skills (native Agent Skills loadability)

Run the additive migration pass from SKILL-MIGRATION-GUIDE: generate a root SKILL.md for every skill that lacked one, copied from its desktop/SKILL.md (or code/SKILL.md), with name set to the directory name and description + body preserved verbatim. - scripts/migrate_skill_root.py: the reusable, non-destructive migrator (dry-run default). - 61 new root SKILL.md (desktop source for most; code/SKILL.md for 61/62/92). - Untouched: 16/17/95 (already had root); desktop/ and code/ packaging left intact. - All 64 root SKILL.md validate: frontmatter <=1024, kebab name, description present. Still MANUAL (no SKILL.md source — commands/README only), need hand-authored root SKILL.md: 81-mac-optimizer, 90-reference-curator, 91-multi-agent-guide, 94-dintel-bootstrap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 01:05:20 +09:00
parent 5f66f57a8e
commit e519a49cc4
62 changed files with 13869 additions and 0 deletions
--- a/custom-skills/32-seo-crawl-budget/SKILL.md
+++ b/custom-skills/32-seo-crawl-budget/SKILL.md
@@ -0,0 +1,234 @@
+---
+name: 32-seo-crawl-budget
+description: |
+  Crawl budget optimization and server log analysis for search engine bots.
+  Triggers: crawl budget, log analysis, bot crawling, Googlebot, crawl waste,
+  orphan pages, crawl efficiency, 크롤 예산, 로그 분석, 크롤 최적화.
+---
+
+# Crawl Budget Optimizer
+
+Analyze server access logs to identify crawl budget waste and generate optimization recommendations for search engine bots (Googlebot, Yeti/Naver, Bingbot, Daumoa/Kakao).
+
+## Capabilities
+
+### Log Analysis
+- Parse Nginx combined, Apache combined, and CloudFront log formats
+- Support for gzip/bzip2 compressed logs
+- Streaming parser for files >1GB
+- Date range filtering
+- Custom format via regex
+
+### Bot Profiling
+- Identify bots by User-Agent: Googlebot (and variants), Yeti (Naver), Bingbot, Daumoa (Kakao), Applebot, DuckDuckBot, Baiduspider
+- Per-bot metrics: requests/day, requests/hour, unique URLs crawled
+- Status code distribution per bot (200, 301, 404, 500)
+- Crawl depth distribution
+- Crawl pattern analysis (time of day, days of week)
+- Most crawled URLs per bot
+
+### Waste Detection
+- **Parameter URLs**: ?sort=, ?filter=, ?page=, ?utm_* consuming crawl budget
+- **Redirect chains**: Multiple redirects consuming crawl slots
+- **Soft 404s**: 200 status pages with error/empty content
+- **Duplicate URLs**: www/non-www, http/https, trailing slash variants
+- **Low-value pages**: Thin content pages, noindex pages being crawled
+
+### Orphan Page Detection
+- Pages in sitemap but never crawled by bots
+- Pages crawled but not in sitemap
+- Crawled pages with no internal links pointing to them
+
+## Workflow
+
+### Step 1: Obtain Server Access Logs
+Request or locate server access logs from the target site. Supported formats:
+- Nginx: `/var/log/nginx/access.log`
+- Apache: `/var/log/apache2/access.log`
+- CloudFront: Downloaded from S3 or CloudWatch
+
+### Step 2: Parse Access Logs
+```bash
+python scripts/log_parser.py --log-file access.log --json
+python scripts/log_parser.py --log-file access.log.gz --streaming --json
+python scripts/log_parser.py --log-file access.log --bot googlebot --json
+```
+
+### Step 3: Crawl Budget Analysis
+```bash
+python scripts/crawl_budget_analyzer.py --log-file access.log --sitemap https://example.com/sitemap.xml --json
+python scripts/crawl_budget_analyzer.py --log-file access.log --scope waste --json
+python scripts/crawl_budget_analyzer.py --log-file access.log --scope orphans --json
+python scripts/crawl_budget_analyzer.py --log-file access.log --scope bots --json
+```
+
+### Step 4: Cross-Reference with External Data (Optional)
+Use `our-seo-agent` CLI or provide pre-fetched JSON via `--input` to compare indexed pages vs crawled pages. WebSearch can supplement with current indexing data.
+
+### Step 5: Generate Recommendations
+Prioritized action items:
+1. robots.txt optimization (block parameter URLs, low-value paths)
+2. URL parameter handling (Google Search Console settings)
+3. Noindex/nofollow for low-value pages
+4. Redirect chain resolution (reduce 301 → 301 → 200 to 301 → 200)
+5. Internal linking improvements for orphan pages
+
+### Step 6: Report to Notion
+Save Korean-language report to SEO Audit Log database.
+
+| Property | Type | Description |
+|----------|------|-------------|
+| Issue | Title | Report title (Korean + date) |
+| Site | URL | Audited website URL |
+| Category | Select | Crawl Budget |
+| Priority | Select | Based on efficiency score |
+| Found Date | Date | Analysis date (YYYY-MM-DD) |
+| Audit ID | Rich Text | Format: CRAWL-YYYYMMDD-NNN |
+
+## Data Source Selection
+
+Crawl-budget analysis is **primarily log-based** — the authoritative signal lives in server access logs that this skill's local Python scripts parse. API backends supplement that with crawled URL inventories, index status, and third-party site-audit views. Pick per data class.
+
+| Backend | Best for | Notes |
+|---|---|---|
+| **Server access logs** (via `scripts/log_parser.py` + `crawl_budget_analyzer.py`) | **Primary** — bot identification, request volume, status code distribution, waste detection, orphan pages | Requires actual server logs from the user (Nginx / Apache / CloudFront). No MCP substitute. |
+| **OurSEO** (CLI + MCP) | **Default** for crawl-derived URL inventory, index status, redirect chains | CLI: `our collect crawl`, `our research google index`, `our audit tech`. MCP: `mcp__ourseo__crawl_website`, `mcp__ourseo__check_index`. Distributed crawl for large sites: `our collect distributed --workers N`. |
+| **Ahrefs MCP** (`mcp__ahrefs__*`) | Orphan detection, redirect chain analysis via Ahrefs site audit | `site-audit-issues`, `site-audit-page-explorer`, `site-audit-page-content`. Useful when an Ahrefs project already exists for the domain. |
+| **Semrush MCP** (`mcp__semrush__*`) | Alternative site audit when Ahrefs project doesn't exist | `siteaudit_research` → `get_report_schema` → `execute_report`. |
+| **GSC** (via `our research search-console`) | First-party Googlebot crawl stats (Coverage / Crawl Stats reports) | Required for ground-truth Googlebot behaviour — server logs show what hit the origin, GSC shows what Google considers crawled. |
+
+### How to pick
+
+1. **User named a backend explicitly** → use it.
+2. **User preference memory** — read `feedback_seo_tool_preferences.md`; honor the task-type default.
+3. **Server access logs are available** → ALWAYS process them first (`log_parser.py` + `crawl_budget_analyzer.py`). They are the only source for actual bot behaviour at the origin.
+4. **Sitemap vs. crawl comparison needed** → OurSEO `crawl_website` for URL inventory; cross-reference with logs.
+5. **No logs available** → fall back to OurSEO crawl + GSC Coverage + Ahrefs/Semrush site audit. State this limitation explicitly in the report.
+6. **Default**: server logs (when present) + **OurSEO crawl + index check** for URL inventory.
+7. **Still ambiguous + non-trivial** → ask once via `AskUserQuestion`.
+
+### Backend call patterns
+
+**Local log analysis (primary):**
+```bash
+python scripts/log_parser.py --log-file access.log --json
+python scripts/log_parser.py --log-file access.log.gz --streaming --json
+python scripts/log_parser.py --log-file access.log --bot googlebot --json
+
+python scripts/crawl_budget_analyzer.py --log-file access.log --sitemap https://<site>/sitemap.xml --json
+python scripts/crawl_budget_analyzer.py --log-file access.log --scope waste --json
+python scripts/crawl_budget_analyzer.py --log-file access.log --scope orphans --json
+python scripts/crawl_budget_analyzer.py --log-file access.log --scope bots --json
+```
+
+**OurSEO CLI (URL inventory + index check):**
+```bash
+our collect crawl https://<site> --max-pages 5000
+our collect distributed https://<site> --workers 8 --max-pages 50000
+our research google index --domain <site>
+our audit tech https://<site>
+```
+
+**OurSEO MCP (Claude Desktop):**
+```
+mcp__ourseo__crawl_website(url="<site>", max_pages=5000)
+mcp__ourseo__check_index(domain="<site>")
+```
+
+**GSC (first-party crawl stats):**
+```bash
+our research search-console queries --site sc-domain:<site> --days 28
+# See also: GSC Coverage / Crawl Stats reports (UI-only for some sections).
+```
+
+**Ahrefs MCP (third-party site audit):**
+```
+mcp__ahrefs__site-audit-projects()
+mcp__ahrefs__site-audit-issues(project_id="<id>")
+mcp__ahrefs__site-audit-page-explorer(project_id="<id>")
+```
+
+**Semrush MCP (alternative site audit):**
+```
+mcp__semrush__siteaudit_research(query="<site>", database="us")
+```
+
+Always record the chosen data source(s) in the report **Overview** so future audits can compare like-for-like — and explicitly state whether server logs were available.
+
+## Output Format
+
+```json
+{
+  "log_file": "access.log",
+  "analysis_period": {"from": "2025-01-01", "to": "2025-01-31"},
+  "total_bot_requests": 150000,
+  "bots": {
+    "googlebot": {
+      "requests": 80000,
+      "unique_urls": 12000,
+      "avg_requests_per_day": 2580,
+      "status_distribution": {"200": 70000, "301": 5000, "404": 3000, "500": 2000}
+    },
+    "yeti": {"requests": 35000},
+    "bingbot": {"requests": 20000},
+    "daumoa": {"requests": 15000}
+  },
+  "waste": {
+    "parameter_urls": {"count": 5000, "pct_of_crawls": 3.3},
+    "redirect_chains": {"count": 2000, "pct_of_crawls": 1.3},
+    "soft_404s": {"count": 1500, "pct_of_crawls": 1.0},
+    "total_waste_pct": 8.5
+  },
+  "orphan_pages": {
+    "in_sitemap_not_crawled": [],
+    "crawled_not_in_sitemap": []
+  },
+  "recommendations": [],
+  "efficiency_score": 72,
+  "timestamp": "2025-01-01T00:00:00"
+}
+```
+
+## Korean Output Example
+
+```
+# 크롤 예산 분석 보고서 - example.com
+
+## 분석 기간: 2025-01-01 ~ 2025-01-31
+
+### 봇별 크롤 현황
+| 봇 | 요청 수 | 고유 URL | 일 평균 |
+|----|---------|---------|---------|
+| Googlebot | 80,000 | 12,000 | 2,580 |
+| Yeti (Naver) | 35,000 | 8,000 | 1,129 |
+
+### 크롤 낭비 요인
+- 파라미터 URL: 5,000건 (3.3%)
+- 리다이렉트 체인: 2,000건 (1.3%)
+- 소프트 404: 1,500건 (1.0%)
+
+### 효율성 점수: 72/100
+```
+
+## Limitations
+
+- Requires actual server access logs (not available via standard web crawling)
+- Log format auto-detection may need manual format specification for custom formats
+- CloudFront logs have a different field structure than Nginx/Apache
+- Large log files (>10GB) may need pre-filtering before analysis
+- Bot identification relies on User-Agent strings which can be spoofed
+
+## Notion Output (Required)
+
+All audit reports MUST be saved to the OurDigital SEO Audit Log:
+- **Database ID**: `2c8581e5-8a1e-8035-880b-e38cefc2f3ef`
+- **Category**: Crawl Budget
+- **Audit ID Format**: CRAWL-YYYYMMDD-NNN
+- **Language**: Korean with technical English terms (Crawl Budget, Googlebot, robots.txt)
+
+## Reference Scripts
+
+Located in `code/scripts/`:
+- `log_parser.py` — Server access log parser with bot identification
+- `crawl_budget_analyzer.py` — Crawl budget efficiency analysis
+- `base_client.py` — Shared async client utilities