# OurSEO Agent (CLI + MCP)

OurSEO contributes the **crawled URL inventory** (vs. server log observations), **index status check**, and **distributed crawl** for large sites. Pair with `scripts/log_parser.py` + `scripts/crawl_budget_analyzer.py` for the full crawl-budget picture.

## CLI commands (Claude Code)

```bash
# URL inventory crawl
our collect crawl https://<site> --max-pages 5000
our collect crawl https://<site> --stealth --backend scrapy

# Distributed crawl for large sites
our collect distributed https://<site> --workers 8 --max-pages 50000

# Index status (Google index coverage)
our research google index --domain <site>

# Technical audit (redirects, canonicals, etc.)
our audit tech https://<site>

# MySQL batch analysis on crawl results
our analyze mysql-batch --session <id>
our analyze mysql-batch --session <id> --export issues --format excel
```

## MCP tools (Claude Desktop)

```
mcp__ourseo__crawl_website(url="<site>", max_pages=5000)
mcp__ourseo__check_index(domain="<site>")
mcp__ourseo__audit_page(url="<url>", audit_type="tech")
```

## Strengths for this skill

- **Configurable crawler** — backend choice (aiohttp, Scrapy), stealth mode, resume support.
- **Distributed crawl** scales to 50K+ URL audits.
- **`check_index`** uses Google PSE to spot-check indexed-or-not per URL.
- Crawl output drops into MySQL workspace — easy to diff against log-derived URL sets.

## How OurSEO pairs with log analysis

1. **Logs** (`scripts/log_parser.py`) → what bots actually crawled at the origin
2. **OurSEO crawl** → what URLs exist on the site today
3. **Sitemap** → what URLs the site claims to publish
4. **GSC** → what Google considers indexed
5. **Diff all four** — orphan, ghost, and waste candidates emerge from set differences

## Not for this skill when

- **Actual bot User-Agent + frequency** — only logs show this.
- **Per-bot status distribution** — only logs.
- **First-party Googlebot crawl stats** — GSC.