---
description: Technical SEO audit for robots.txt, XML sitemap, and URL crawlability
---

# SEO Technical Audit

Crawlability fundamentals: robots.txt validation, XML sitemap analysis, URL accessibility checking.

## Triggers
- "technical SEO", "crawlability"
- "robots.txt", "sitemap validation"

## Quick Start

```bash
python scripts/robots_checker.py --url https://example.com
python scripts/sitemap_validator.py --url https://example.com/sitemap.xml
python scripts/sitemap_crawler.py --sitemap https://example.com/sitemap.xml
```

## Scripts

| Script | Purpose |
|--------|---------|
| `robots_checker.py` | Parse robots.txt, test URL rules |
| `sitemap_validator.py` | Validate XML structure, URL count, lastmod |
| `sitemap_crawler.py` | Async check URL accessibility, HTTP status |

## Common Issues Detected

| Issue | Severity |
|-------|----------|
| Missing sitemap declaration | Medium |
| Blocking CSS/JS resources | High |
| URLs returning 404 | High |
| Exceeds 50,000 URL limit | High |
| Non-canonical URLs included | Medium |

## Config
- `CRAWL_DELAY=1.0`, `MAX_CONCURRENT=20`, `REQUEST_TIMEOUT=30`

## Notion Output
- Database: `2c8581e5-8a1e-8035-880b-e38cefc2f3ef`
- Korean report, Audit ID: [TYPE]-YYYYMMDD-NNN