docs(notion): add Phase 3b-i spec for Notion semantic search skill
Brainstorming output for the first sub-project derived from the original "Notion-as-RAG export" idea. After scope clarification, that vision split into two independent skills: - 3b-i (this spec): semantic search foundation - 3b-ii (separate, later): notion-to-notebooklm push Locks five architectural decisions reached during brainstorming: - Strategy C — query expansion + LLM rerank (closes the gap from Notion's keyword-only native search) - Standalone search skill, JSON output for downstream chaining - Claude Haiku 4.5 for both stages (cheap, fast, plenty good) - SHA256(query + candidate_ids) cache with 1-day TTL - Permissive degradation matching Phase 3c parser philosophy ~850 LOC + tests. ~3 days estimated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,282 @@
|
||||
# Notion Semantic Search (Skill 31) — Design
|
||||
|
||||
> **Date**: 2026-04-28
|
||||
> **Status**: Approved (brainstorming)
|
||||
> **Scope**: Phase 3b-i — semantic search across Notion workspaces with query expansion + LLM rerank
|
||||
> **Sequence**: First of two sub-projects derived from the original Phase 3b "Notion-as-RAG export". The push-to-NotebookLM skill is the second sub-project, gets its own design after this ships.
|
||||
> **Predecessor**: Phase 3c (commit `6446f00` — extended block coverage in `32-notion-writer`)
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Build a CLI skill that searches the user's Notion workspace semantically. Output is either a terminal-friendly table for browsing or JSON for piping into the (future) `notion-to-notebooklm` push skill.
|
||||
|
||||
The load-bearing problem: Notion's native search is keyword-only and weak — "AI agents" doesn't surface a page titled "Multi-agent orchestration", and Korean ↔ English queries don't cross-match. We close the gap with two LLM stages over Notion's API search: query expansion (Claude generates synonym/cross-language variants) and rerank (Claude scores candidates against the original query).
|
||||
|
||||
This is the foundation for the rest of the 31-notion-organizer vision — aggregation, move/reorganize, and exports all operate on search results, so getting search right unblocks everything else.
|
||||
|
||||
---
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **Block-level result granularity.** Page-level only in v1. Surfacing matching block snippets within pages adds complexity and most retrieval is page-level anyway. Defer until a real use case demands it.
|
||||
- **Cross-workspace search.** Notion's integration model is per-workspace; abstracting over multiple workspaces means installing the integration in each. Document the limitation, don't try to hide it.
|
||||
- **Embedding-based search.** Vector store + sync pipeline pushes this back into "build a tool" territory. Out of scope for the skill model.
|
||||
- **The push-to-NotebookLM skill.** Separate spec, separate plan. Search outputs JSON; the push skill consumes it.
|
||||
- **Automatic pagination of search results.** Cap at ~30 candidates pre-rerank; this is enough for top-10 reranked results. If a query genuinely has 100+ relevant matches, the user can constrain with `--databases` or `--filter`.
|
||||
|
||||
---
|
||||
|
||||
## Architectural decisions
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|---|---|---|
|
||||
| Search strategy | **Strategy C — query expansion + rerank** | Notion's keyword search alone is the gap we're filling. Rerank-only catches synonyms but misses cross-language. Expansion + rerank covers both. |
|
||||
| Workflow shape | **Standalone search skill** (browse-friendly), JSON output for downstream chaining | User searches frequently for memory-refresh; pushing to NotebookLM is occasional. Two independent skills, composable via JSON. |
|
||||
| LLM model | **Claude Haiku 4.5** for both expansion and rerank | Plenty good for "rank these 30 titles + properties by relevance"; ~$0.005/query, 8-15s total latency. |
|
||||
| LLM client | **`anthropic` SDK preferred, `claude -p` CLI fallback** | SDK gives structured output; CLI works without separate API key for users on Claude Code subscriptions. |
|
||||
| Caching | **SHA256(query + sorted_candidate_ids) → JSON file** in `~/.cache/notion-search/` | Cheap, no infra, invalidates naturally when Notion content changes (different candidates → cache miss). 1-day TTL. |
|
||||
| Property filter syntax | **JSON object** matching `32-notion-writer`'s `--properties` form | Consistency across the skill suite. |
|
||||
| Failure mode | **Permissive degradation** | Rerank fail → return raw expanded-search; expansion fail → original query only. Match the parser philosophy from Phase 3c. |
|
||||
|
||||
---
|
||||
|
||||
## CLI surface
|
||||
|
||||
```bash
|
||||
# Default — rerank + expand, terminal output
|
||||
notion-search "AI agents in 2026"
|
||||
|
||||
# Pipe to JSON for the future push skill
|
||||
notion-search "AI agents" --json | jq '.[].id'
|
||||
|
||||
# Constrain to specific databases
|
||||
notion-search "AI agents" --databases f8f19ede-32bd-43ac-9f60-0651f6f40afe,abc-def-...
|
||||
|
||||
# Property filter (same JSON form as notion-writer's --properties)
|
||||
notion-search "MCP" --filter '{"Status": "Done", "Topic": "AI"}'
|
||||
|
||||
# Fast paths
|
||||
notion-search "exact term" --no-rerank # Notion API only, no LLM
|
||||
notion-search "exact term" --no-expand # Skip variant generation
|
||||
notion-search "AI agents" --limit 5 # Default 10
|
||||
notion-search "AI agents" --no-cache # Bypass result cache
|
||||
```
|
||||
|
||||
**Flags:**
|
||||
|
||||
| Flag | Default | Effect |
|
||||
|---|---|---|
|
||||
| `--json` | off | Emit JSON array instead of terminal table |
|
||||
| `--databases <ids>` | all accessible | Comma-separated database/data_source IDs to constrain search |
|
||||
| `--filter <json>` | none | Property filter applied per-database (skipped for workspace-wide search) |
|
||||
| `--no-rerank` | off | Skip Claude rerank stage |
|
||||
| `--no-expand` | off | Skip query-variant generation |
|
||||
| `--limit <n>` | 10 | Number of results to return after rerank |
|
||||
| `--no-cache` | off | Bypass cache lookup AND don't write cache |
|
||||
|
||||
---
|
||||
|
||||
## Pipeline
|
||||
|
||||
### 1. Query expansion (skipped if `--no-expand`)
|
||||
|
||||
Claude Haiku takes the original query and produces 3-5 variants covering:
|
||||
- Synonyms ("AI agents" → "multi-agent orchestration", "autonomous LLM agents")
|
||||
- Cross-language KR↔EN ("AI agents" → "AI 에이전트")
|
||||
- Related concepts ("AI agents" → "agent SDK", "tool use")
|
||||
|
||||
The original query is always included. Total variants ≤ 5 to bound API calls.
|
||||
|
||||
Prompt approach: ask Claude for a JSON array of variants, parse with `json.loads`. On parse failure, fall back to original query only.
|
||||
|
||||
### 2. Search execution
|
||||
|
||||
For each variant, hit Notion API:
|
||||
- Workspace-wide (`client.search`) when no `--databases` flag
|
||||
- Per-database (`client.data_sources.query`) when `--databases` is provided. The existing `_notion_compat.resolve_data_source_id` resolves DB IDs.
|
||||
|
||||
Calls are made sequentially (no concurrent dispatch) to stay well under Notion's 3 req/sec average rate limit. With ≤5 variants × ≤2 DBs (typical case), that's at most 10 sequential calls — under 4 seconds total even at the slower end.
|
||||
|
||||
Each call returns up to 100 results; we cap candidates at 30 total (after dedup across all variants and DBs) to keep rerank costs predictable.
|
||||
|
||||
Dedupe by page ID. Preserve the highest position-rank if the same page appears for multiple variants (used as fallback ordering when rerank is skipped).
|
||||
|
||||
### 3. Property + excerpt fetch
|
||||
|
||||
For each candidate, gather:
|
||||
- Title (from page.properties[title_prop])
|
||||
- Key properties (Status, Topic, Type, Account Code, etc. — whatever exists)
|
||||
- 200-char excerpt: fetch the first text-bearing block via `client.blocks.children.list` with `page_size=5`, walk the returned blocks and pick the first paragraph/heading/quote whose rich-text concatenates to non-empty content
|
||||
|
||||
Excerpt fallback: if none of the first 5 blocks have text content (e.g., page leads with an image, table-only, or empty), excerpt is the empty string. Rerank still runs — title + properties usually carry enough signal.
|
||||
|
||||
Properties often arrive inline with `client.search` results; we only re-fetch when properties are stripped (data source queries return full properties; workspace search doesn't).
|
||||
|
||||
### 4. Rerank (skipped if `--no-rerank`)
|
||||
|
||||
Cache lookup first: SHA256(original_query + sorted_candidate_ids).
|
||||
|
||||
Cache miss → call Claude Haiku with:
|
||||
- Original (un-expanded) query
|
||||
- Numbered list of candidates: `[N] Title — Properties — Excerpt`
|
||||
- Asks for a JSON array of objects: `{index, score, why}` ordered by score descending
|
||||
|
||||
Parse, map back to candidate page objects, take top `--limit`.
|
||||
|
||||
Failure modes:
|
||||
- LLM call error → return raw expanded-search results in candidate order, warn on stderr
|
||||
- JSON parse error → same as above
|
||||
- Rerank returns fewer than requested → take what's there, warn
|
||||
|
||||
### 5. Output
|
||||
|
||||
**Terminal** (default): formatted table with title, relevance, snippet, key properties, URL. ANSI color via `rich` library if available, plain text otherwise.
|
||||
|
||||
**JSON** (`--json`): array of objects with stable schema:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "abc-def-...", // dashed UUID
|
||||
"url": "https://notion.so/...",
|
||||
"title": "MCP server architecture...",
|
||||
"relevance": 0.94, // 0.0-1.0; null if --no-rerank
|
||||
"snippet": "Direct match — covers...", // null if --no-rerank
|
||||
"excerpt": "First paragraph text...", // 200-char from page body
|
||||
"properties": { // only properties present on the page
|
||||
"Status": "Done",
|
||||
"Topic": ["AI", "MCP"],
|
||||
"Account Code": "D.intelligence"
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
The schema is stable: any future tool consuming search output (notion-to-notebooklm push, aggregation, etc.) reads this format.
|
||||
|
||||
---
|
||||
|
||||
## File structure
|
||||
|
||||
| File | Purpose | LOC |
|
||||
|---|---|---|
|
||||
| `custom-skills/31-notion-organizer/code/scripts/notion_search.py` | Main CLI + pipeline | ~400 |
|
||||
| `custom-skills/31-notion-organizer/code/scripts/test_notion_search.py` | Unit tests with mocked Claude/Notion | ~250 |
|
||||
| `custom-skills/31-notion-organizer/code/scripts/_search_llm.py` | LLM client abstraction (SDK + CLI fallback, prompt construction) | ~150 |
|
||||
| `custom-skills/31-notion-organizer/commands/notion-search.md` | Slash command definition | ~50 |
|
||||
| `custom-skills/31-notion-organizer/code/CLAUDE.md` | Add usage section | +60 lines |
|
||||
| `custom-skills/31-notion-organizer/code/scripts/requirements.txt` | Add `anthropic` (optional, falls back to CLI) | +1 line |
|
||||
|
||||
**Total: ~850 LOC + tests + docs.** ~3 days focused work.
|
||||
|
||||
The `_notion_compat` helper from Phase 2 is reused (client factory, error explanation, ID resolution). No new compatibility layer needed.
|
||||
|
||||
---
|
||||
|
||||
## LLM client (`_search_llm.py`)
|
||||
|
||||
Abstraction so callers don't care whether SDK or CLI is in use.
|
||||
|
||||
```python
|
||||
def call_claude(prompt: str, model: str = "claude-haiku-4-5", max_tokens: int = 1000) -> str:
|
||||
"""Return Claude's text response. Raises on failure."""
|
||||
if _have_anthropic_sdk() and os.getenv("ANTHROPIC_API_KEY"):
|
||||
return _call_via_sdk(prompt, model, max_tokens)
|
||||
if _have_claude_cli():
|
||||
return _call_via_cli(prompt, model)
|
||||
raise RuntimeError(
|
||||
"No Claude client available. Install `anthropic` and set "
|
||||
"ANTHROPIC_API_KEY, or install Claude Code CLI."
|
||||
)
|
||||
```
|
||||
|
||||
Two thin implementations behind it:
|
||||
- `_call_via_sdk`: standard `anthropic.Anthropic().messages.create(...)`
|
||||
- `_call_via_cli`: `subprocess.run(["claude", "-p", prompt, "--model", model], ...)`, capture stdout
|
||||
|
||||
Both return the assistant's text content as a single string. Callers (`expand_query`, `rerank_candidates`) parse JSON out of it.
|
||||
|
||||
---
|
||||
|
||||
## Caching
|
||||
|
||||
**Key:** SHA256 hex digest of `f"{query}|{','.join(sorted(candidate_ids))}"`.
|
||||
|
||||
**Storage:** `~/.cache/notion-search/<hash>.json`. Each file contains:
|
||||
```json
|
||||
{
|
||||
"query": "AI agents",
|
||||
"candidate_ids": ["abc...", "def..."],
|
||||
"results": [...], // the reranked output
|
||||
"cached_at": "2026-04-28T14:23:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Lookup:** if file exists and `now - cached_at < TTL` (default 24h), return cached results. Else run rerank and write fresh.
|
||||
|
||||
**Invalidation:** TTL-only for simplicity. `--no-cache` bypasses both read and write.
|
||||
|
||||
**Why not invalidate on Notion content changes?** That requires tracking last-edited timestamps and computing a content hash — significant complexity for marginal benefit. With a 1-day TTL, stale results are bounded; user can `--no-cache` for fresh runs.
|
||||
|
||||
---
|
||||
|
||||
## Error handling
|
||||
|
||||
| Failure | Behavior |
|
||||
|---|---|
|
||||
| Notion API: `ObjectNotFound` on `--databases` ID | Error out with friendly message via `_notion_compat.explain_api_error` |
|
||||
| Notion API: `Unauthorized` | Same |
|
||||
| Query expansion fails (LLM error or JSON parse) | Use original query only, warn on stderr, continue |
|
||||
| Rerank fails (LLM error or JSON parse) | Return raw expanded-search results in candidate order, warn on stderr, continue |
|
||||
| Cache file corrupted | Delete cache file, treat as miss |
|
||||
| Empty Notion results | Print "No matches for '<query>'" and exit 0 (not an error) |
|
||||
| LLM client unavailable (no SDK + no CLI) | Error out with setup instructions if `--no-rerank` and `--no-expand` aren't both set; otherwise proceed |
|
||||
| `--filter` JSON parse error | Error out before any API calls |
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
10 tests covering the public surface, all using mocked Claude and Notion clients (no real API calls in tests):
|
||||
|
||||
| Test | Verifies |
|
||||
|---|---|
|
||||
| `test_expand_query_returns_variants` | LLM mock returns variant list including original; expansion produces 3-5 unique strings |
|
||||
| `test_search_unions_and_dedupes` | Two variants returning overlapping pages produce deduped candidate set |
|
||||
| `test_rerank_orders_by_relevance` | Mock rerank returns scores; output sorted score-descending |
|
||||
| `test_no_rerank_returns_raw` | `--no-rerank` skips LLM, returns Notion's native order, no relevance scores |
|
||||
| `test_no_expand_uses_single_query` | `--no-expand` calls Notion once with original query, no LLM expansion |
|
||||
| `test_json_output_parseable` | `--json` emits valid JSON conforming to the schema in Section 5 |
|
||||
| `test_cache_hit_on_repeat` | Second identical query skips LLM; cache file exists |
|
||||
| `test_cache_miss_on_different_candidates` | Different candidate set hashes different → fresh rerank |
|
||||
| `test_property_filter_applied` | `--filter` JSON passed to data_sources.query verbatim |
|
||||
| `test_database_scope_constrains` | `--databases` causes per-DB queries instead of workspace search |
|
||||
|
||||
Tests live in `test_notion_search.py`. Run via `python3 test_notion_search.py` (matching the existing test_parser.py convention in 32).
|
||||
|
||||
---
|
||||
|
||||
## Out-of-scope follow-ups
|
||||
|
||||
- **3b-ii — `notion-to-notebooklm` push skill**: consumes search-output JSON, pushes pages as NotebookLM sources. Separate spec after this ships.
|
||||
- **Block-level granularity**: surface matching blocks within pages. Adds chunking logic and richer rerank prompts. Defer.
|
||||
- **Aggregation skill**: "find pages on X, write a summary page with citations." Builds on this. Future Phase 3 sub-project.
|
||||
- **Multi-workspace search**: requires per-workspace integrations and credential management. Future.
|
||||
|
||||
---
|
||||
|
||||
## Implementation transition
|
||||
|
||||
After user approval, transition to `superpowers:writing-plans` to break this into bite-sized TDD tasks. Likely shape:
|
||||
|
||||
1. `_search_llm.py` skeleton (SDK + CLI client abstraction) + unit test
|
||||
2. Query expansion module + tests
|
||||
3. Search execution module (workspace + per-DB) + tests
|
||||
4. Property + excerpt fetch + tests
|
||||
5. Rerank module (with prompt + JSON parsing) + tests
|
||||
6. Cache layer + tests
|
||||
7. CLI assembly (argparse + output formatting) + tests
|
||||
8. Slash command + 31's CLAUDE.md update + final integration smoke test
|
||||
|
||||
Each step independently testable. Total ~3 days estimated, mirrors Phase 3c's pacing.
|
||||
Reference in New Issue
Block a user