description
| description |
| Technical SEO audit for robots.txt, XML sitemap, and URL crawlability |
SEO Technical Audit
Crawlability fundamentals: robots.txt validation, XML sitemap analysis, URL accessibility checking.
Triggers
- "technical SEO", "crawlability"
- "robots.txt", "sitemap validation"
Quick Start
Scripts
| Script |
Purpose |
robots_checker.py |
Parse robots.txt, test URL rules |
sitemap_validator.py |
Validate XML structure, URL count, lastmod |
sitemap_crawler.py |
Async check URL accessibility, HTTP status |
Common Issues Detected
| Issue |
Severity |
| Missing sitemap declaration |
Medium |
| Blocking CSS/JS resources |
High |
| URLs returning 404 |
High |
| Exceeds 50,000 URL limit |
High |
| Non-canonical URLs included |
Medium |
Config
CRAWL_DELAY=1.0, MAX_CONCURRENT=20, REQUEST_TIMEOUT=30
Notion Output
- Database:
2c8581e5-8a1e-8035-880b-e38cefc2f3ef
- Korean report, Audit ID: [TYPE]-YYYYMMDD-NNN