From 08e3dd7dae4b8c5467debc48f52084822f26836f Mon Sep 17 00:00:00 2001 From: Andrew Yim Date: Tue, 28 Apr 2026 07:23:30 +0900 Subject: [PATCH] docs(notion): add Phase 3b-i spec for Notion semantic search skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brainstorming output for the first sub-project derived from the original "Notion-as-RAG export" idea. After scope clarification, that vision split into two independent skills: - 3b-i (this spec): semantic search foundation - 3b-ii (separate, later): notion-to-notebooklm push Locks five architectural decisions reached during brainstorming: - Strategy C — query expansion + LLM rerank (closes the gap from Notion's keyword-only native search) - Standalone search skill, JSON output for downstream chaining - Claude Haiku 4.5 for both stages (cheap, fast, plenty good) - SHA256(query + candidate_ids) cache with 1-day TTL - Permissive degradation matching Phase 3c parser philosophy ~850 LOC + tests. ~3 days estimated. Co-Authored-By: Claude Opus 4.7 (1M context) --- ...026-04-28-notion-semantic-search-design.md | 282 ++++++++++++++++++ 1 file changed, 282 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-28-notion-semantic-search-design.md diff --git a/docs/superpowers/specs/2026-04-28-notion-semantic-search-design.md b/docs/superpowers/specs/2026-04-28-notion-semantic-search-design.md new file mode 100644 index 0000000..b85fc5c --- /dev/null +++ b/docs/superpowers/specs/2026-04-28-notion-semantic-search-design.md @@ -0,0 +1,282 @@ +# Notion Semantic Search (Skill 31) — Design + +> **Date**: 2026-04-28 +> **Status**: Approved (brainstorming) +> **Scope**: Phase 3b-i — semantic search across Notion workspaces with query expansion + LLM rerank +> **Sequence**: First of two sub-projects derived from the original Phase 3b "Notion-as-RAG export". The push-to-NotebookLM skill is the second sub-project, gets its own design after this ships. +> **Predecessor**: Phase 3c (commit `6446f00` — extended block coverage in `32-notion-writer`) + +--- + +## Goal + +Build a CLI skill that searches the user's Notion workspace semantically. Output is either a terminal-friendly table for browsing or JSON for piping into the (future) `notion-to-notebooklm` push skill. + +The load-bearing problem: Notion's native search is keyword-only and weak — "AI agents" doesn't surface a page titled "Multi-agent orchestration", and Korean ↔ English queries don't cross-match. We close the gap with two LLM stages over Notion's API search: query expansion (Claude generates synonym/cross-language variants) and rerank (Claude scores candidates against the original query). + +This is the foundation for the rest of the 31-notion-organizer vision — aggregation, move/reorganize, and exports all operate on search results, so getting search right unblocks everything else. + +--- + +## Non-goals + +- **Block-level result granularity.** Page-level only in v1. Surfacing matching block snippets within pages adds complexity and most retrieval is page-level anyway. Defer until a real use case demands it. +- **Cross-workspace search.** Notion's integration model is per-workspace; abstracting over multiple workspaces means installing the integration in each. Document the limitation, don't try to hide it. +- **Embedding-based search.** Vector store + sync pipeline pushes this back into "build a tool" territory. Out of scope for the skill model. +- **The push-to-NotebookLM skill.** Separate spec, separate plan. Search outputs JSON; the push skill consumes it. +- **Automatic pagination of search results.** Cap at ~30 candidates pre-rerank; this is enough for top-10 reranked results. If a query genuinely has 100+ relevant matches, the user can constrain with `--databases` or `--filter`. + +--- + +## Architectural decisions + +| Decision | Choice | Rationale | +|---|---|---| +| Search strategy | **Strategy C — query expansion + rerank** | Notion's keyword search alone is the gap we're filling. Rerank-only catches synonyms but misses cross-language. Expansion + rerank covers both. | +| Workflow shape | **Standalone search skill** (browse-friendly), JSON output for downstream chaining | User searches frequently for memory-refresh; pushing to NotebookLM is occasional. Two independent skills, composable via JSON. | +| LLM model | **Claude Haiku 4.5** for both expansion and rerank | Plenty good for "rank these 30 titles + properties by relevance"; ~$0.005/query, 8-15s total latency. | +| LLM client | **`anthropic` SDK preferred, `claude -p` CLI fallback** | SDK gives structured output; CLI works without separate API key for users on Claude Code subscriptions. | +| Caching | **SHA256(query + sorted_candidate_ids) → JSON file** in `~/.cache/notion-search/` | Cheap, no infra, invalidates naturally when Notion content changes (different candidates → cache miss). 1-day TTL. | +| Property filter syntax | **JSON object** matching `32-notion-writer`'s `--properties` form | Consistency across the skill suite. | +| Failure mode | **Permissive degradation** | Rerank fail → return raw expanded-search; expansion fail → original query only. Match the parser philosophy from Phase 3c. | + +--- + +## CLI surface + +```bash +# Default — rerank + expand, terminal output +notion-search "AI agents in 2026" + +# Pipe to JSON for the future push skill +notion-search "AI agents" --json | jq '.[].id' + +# Constrain to specific databases +notion-search "AI agents" --databases f8f19ede-32bd-43ac-9f60-0651f6f40afe,abc-def-... + +# Property filter (same JSON form as notion-writer's --properties) +notion-search "MCP" --filter '{"Status": "Done", "Topic": "AI"}' + +# Fast paths +notion-search "exact term" --no-rerank # Notion API only, no LLM +notion-search "exact term" --no-expand # Skip variant generation +notion-search "AI agents" --limit 5 # Default 10 +notion-search "AI agents" --no-cache # Bypass result cache +``` + +**Flags:** + +| Flag | Default | Effect | +|---|---|---| +| `--json` | off | Emit JSON array instead of terminal table | +| `--databases ` | all accessible | Comma-separated database/data_source IDs to constrain search | +| `--filter ` | none | Property filter applied per-database (skipped for workspace-wide search) | +| `--no-rerank` | off | Skip Claude rerank stage | +| `--no-expand` | off | Skip query-variant generation | +| `--limit ` | 10 | Number of results to return after rerank | +| `--no-cache` | off | Bypass cache lookup AND don't write cache | + +--- + +## Pipeline + +### 1. Query expansion (skipped if `--no-expand`) + +Claude Haiku takes the original query and produces 3-5 variants covering: +- Synonyms ("AI agents" → "multi-agent orchestration", "autonomous LLM agents") +- Cross-language KR↔EN ("AI agents" → "AI 에이전트") +- Related concepts ("AI agents" → "agent SDK", "tool use") + +The original query is always included. Total variants ≤ 5 to bound API calls. + +Prompt approach: ask Claude for a JSON array of variants, parse with `json.loads`. On parse failure, fall back to original query only. + +### 2. Search execution + +For each variant, hit Notion API: +- Workspace-wide (`client.search`) when no `--databases` flag +- Per-database (`client.data_sources.query`) when `--databases` is provided. The existing `_notion_compat.resolve_data_source_id` resolves DB IDs. + +Calls are made sequentially (no concurrent dispatch) to stay well under Notion's 3 req/sec average rate limit. With ≤5 variants × ≤2 DBs (typical case), that's at most 10 sequential calls — under 4 seconds total even at the slower end. + +Each call returns up to 100 results; we cap candidates at 30 total (after dedup across all variants and DBs) to keep rerank costs predictable. + +Dedupe by page ID. Preserve the highest position-rank if the same page appears for multiple variants (used as fallback ordering when rerank is skipped). + +### 3. Property + excerpt fetch + +For each candidate, gather: +- Title (from page.properties[title_prop]) +- Key properties (Status, Topic, Type, Account Code, etc. — whatever exists) +- 200-char excerpt: fetch the first text-bearing block via `client.blocks.children.list` with `page_size=5`, walk the returned blocks and pick the first paragraph/heading/quote whose rich-text concatenates to non-empty content + +Excerpt fallback: if none of the first 5 blocks have text content (e.g., page leads with an image, table-only, or empty), excerpt is the empty string. Rerank still runs — title + properties usually carry enough signal. + +Properties often arrive inline with `client.search` results; we only re-fetch when properties are stripped (data source queries return full properties; workspace search doesn't). + +### 4. Rerank (skipped if `--no-rerank`) + +Cache lookup first: SHA256(original_query + sorted_candidate_ids). + +Cache miss → call Claude Haiku with: +- Original (un-expanded) query +- Numbered list of candidates: `[N] Title — Properties — Excerpt` +- Asks for a JSON array of objects: `{index, score, why}` ordered by score descending + +Parse, map back to candidate page objects, take top `--limit`. + +Failure modes: +- LLM call error → return raw expanded-search results in candidate order, warn on stderr +- JSON parse error → same as above +- Rerank returns fewer than requested → take what's there, warn + +### 5. Output + +**Terminal** (default): formatted table with title, relevance, snippet, key properties, URL. ANSI color via `rich` library if available, plain text otherwise. + +**JSON** (`--json`): array of objects with stable schema: +```json +[ + { + "id": "abc-def-...", // dashed UUID + "url": "https://notion.so/...", + "title": "MCP server architecture...", + "relevance": 0.94, // 0.0-1.0; null if --no-rerank + "snippet": "Direct match — covers...", // null if --no-rerank + "excerpt": "First paragraph text...", // 200-char from page body + "properties": { // only properties present on the page + "Status": "Done", + "Topic": ["AI", "MCP"], + "Account Code": "D.intelligence" + } + } +] +``` + +The schema is stable: any future tool consuming search output (notion-to-notebooklm push, aggregation, etc.) reads this format. + +--- + +## File structure + +| File | Purpose | LOC | +|---|---|---| +| `custom-skills/31-notion-organizer/code/scripts/notion_search.py` | Main CLI + pipeline | ~400 | +| `custom-skills/31-notion-organizer/code/scripts/test_notion_search.py` | Unit tests with mocked Claude/Notion | ~250 | +| `custom-skills/31-notion-organizer/code/scripts/_search_llm.py` | LLM client abstraction (SDK + CLI fallback, prompt construction) | ~150 | +| `custom-skills/31-notion-organizer/commands/notion-search.md` | Slash command definition | ~50 | +| `custom-skills/31-notion-organizer/code/CLAUDE.md` | Add usage section | +60 lines | +| `custom-skills/31-notion-organizer/code/scripts/requirements.txt` | Add `anthropic` (optional, falls back to CLI) | +1 line | + +**Total: ~850 LOC + tests + docs.** ~3 days focused work. + +The `_notion_compat` helper from Phase 2 is reused (client factory, error explanation, ID resolution). No new compatibility layer needed. + +--- + +## LLM client (`_search_llm.py`) + +Abstraction so callers don't care whether SDK or CLI is in use. + +```python +def call_claude(prompt: str, model: str = "claude-haiku-4-5", max_tokens: int = 1000) -> str: + """Return Claude's text response. Raises on failure.""" + if _have_anthropic_sdk() and os.getenv("ANTHROPIC_API_KEY"): + return _call_via_sdk(prompt, model, max_tokens) + if _have_claude_cli(): + return _call_via_cli(prompt, model) + raise RuntimeError( + "No Claude client available. Install `anthropic` and set " + "ANTHROPIC_API_KEY, or install Claude Code CLI." + ) +``` + +Two thin implementations behind it: +- `_call_via_sdk`: standard `anthropic.Anthropic().messages.create(...)` +- `_call_via_cli`: `subprocess.run(["claude", "-p", prompt, "--model", model], ...)`, capture stdout + +Both return the assistant's text content as a single string. Callers (`expand_query`, `rerank_candidates`) parse JSON out of it. + +--- + +## Caching + +**Key:** SHA256 hex digest of `f"{query}|{','.join(sorted(candidate_ids))}"`. + +**Storage:** `~/.cache/notion-search/.json`. Each file contains: +```json +{ + "query": "AI agents", + "candidate_ids": ["abc...", "def..."], + "results": [...], // the reranked output + "cached_at": "2026-04-28T14:23:00Z" +} +``` + +**Lookup:** if file exists and `now - cached_at < TTL` (default 24h), return cached results. Else run rerank and write fresh. + +**Invalidation:** TTL-only for simplicity. `--no-cache` bypasses both read and write. + +**Why not invalidate on Notion content changes?** That requires tracking last-edited timestamps and computing a content hash — significant complexity for marginal benefit. With a 1-day TTL, stale results are bounded; user can `--no-cache` for fresh runs. + +--- + +## Error handling + +| Failure | Behavior | +|---|---| +| Notion API: `ObjectNotFound` on `--databases` ID | Error out with friendly message via `_notion_compat.explain_api_error` | +| Notion API: `Unauthorized` | Same | +| Query expansion fails (LLM error or JSON parse) | Use original query only, warn on stderr, continue | +| Rerank fails (LLM error or JSON parse) | Return raw expanded-search results in candidate order, warn on stderr, continue | +| Cache file corrupted | Delete cache file, treat as miss | +| Empty Notion results | Print "No matches for ''" and exit 0 (not an error) | +| LLM client unavailable (no SDK + no CLI) | Error out with setup instructions if `--no-rerank` and `--no-expand` aren't both set; otherwise proceed | +| `--filter` JSON parse error | Error out before any API calls | + +--- + +## Tests + +10 tests covering the public surface, all using mocked Claude and Notion clients (no real API calls in tests): + +| Test | Verifies | +|---|---| +| `test_expand_query_returns_variants` | LLM mock returns variant list including original; expansion produces 3-5 unique strings | +| `test_search_unions_and_dedupes` | Two variants returning overlapping pages produce deduped candidate set | +| `test_rerank_orders_by_relevance` | Mock rerank returns scores; output sorted score-descending | +| `test_no_rerank_returns_raw` | `--no-rerank` skips LLM, returns Notion's native order, no relevance scores | +| `test_no_expand_uses_single_query` | `--no-expand` calls Notion once with original query, no LLM expansion | +| `test_json_output_parseable` | `--json` emits valid JSON conforming to the schema in Section 5 | +| `test_cache_hit_on_repeat` | Second identical query skips LLM; cache file exists | +| `test_cache_miss_on_different_candidates` | Different candidate set hashes different → fresh rerank | +| `test_property_filter_applied` | `--filter` JSON passed to data_sources.query verbatim | +| `test_database_scope_constrains` | `--databases` causes per-DB queries instead of workspace search | + +Tests live in `test_notion_search.py`. Run via `python3 test_notion_search.py` (matching the existing test_parser.py convention in 32). + +--- + +## Out-of-scope follow-ups + +- **3b-ii — `notion-to-notebooklm` push skill**: consumes search-output JSON, pushes pages as NotebookLM sources. Separate spec after this ships. +- **Block-level granularity**: surface matching blocks within pages. Adds chunking logic and richer rerank prompts. Defer. +- **Aggregation skill**: "find pages on X, write a summary page with citations." Builds on this. Future Phase 3 sub-project. +- **Multi-workspace search**: requires per-workspace integrations and credential management. Future. + +--- + +## Implementation transition + +After user approval, transition to `superpowers:writing-plans` to break this into bite-sized TDD tasks. Likely shape: + +1. `_search_llm.py` skeleton (SDK + CLI client abstraction) + unit test +2. Query expansion module + tests +3. Search execution module (workspace + per-DB) + tests +4. Property + excerpt fetch + tests +5. Rerank module (with prompt + JSON parsing) + tests +6. Cache layer + tests +7. CLI assembly (argparse + output formatting) + tests +8. Slash command + 31's CLAUDE.md update + final integration smoke test + +Each step independently testable. Total ~3 days estimated, mirrors Phase 3c's pacing.