ourdigital/our-claude-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(notion): add Phase 3b-i spec for Notion semantic search skill

Browse Source 
Brainstorming output for the first sub-project derived from the
original "Notion-as-RAG export" idea. After scope clarification, that
vision split into two independent skills:

- 3b-i (this spec): semantic search foundation
- 3b-ii (separate, later): notion-to-notebooklm push

Locks five architectural decisions reached during brainstorming:
- Strategy C — query expansion + LLM rerank (closes the gap from
  Notion's keyword-only native search)
- Standalone search skill, JSON output for downstream chaining
- Claude Haiku 4.5 for both stages (cheap, fast, plenty good)
- SHA256(query + candidate_ids) cache with 1-day TTL
- Permissive degradation matching Phase 3c parser philosophy

~850 LOC + tests. ~3 days estimated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andrew Yim
2026-04-28 07:23:30 +09:00
parent 6446f00522
commit 08e3dd7dae
1 changed files with 282 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

282
docs/superpowers/specs/2026-04-28-notion-semantic-search-design.md Normal file
View File

@@ -0,0 +1,282 @@
# Notion Semantic Search (Skill 31) — Design
> **Date**: 2026-04-28
> **Status**: Approved (brainstorming)
> **Scope**: Phase 3b-i — semantic search across Notion workspaces with query expansion + LLM rerank
> **Sequence**: First of two sub-projects derived from the original Phase 3b "Notion-as-RAG export". The push-to-NotebookLM skill is the second sub-project, gets its own design after this ships.
> **Predecessor**: Phase 3c (commit `6446f00` — extended block coverage in `32-notion-writer`)
---
## Goal
Build a CLI skill that searches the user's Notion workspace semantically. Output is either a terminal-friendly table for browsing or JSON for piping into the (future) `notion-to-notebooklm` push skill.
The load-bearing problem: Notion's native search is keyword-only and weak — "AI agents" doesn't surface a page titled "Multi-agent orchestration", and Korean ↔ English queries don't cross-match. We close the gap with two LLM stages over Notion's API search: query expansion (Claude generates synonym/cross-language variants) and rerank (Claude scores candidates against the original query).
This is the foundation for the rest of the 31-notion-organizer vision — aggregation, move/reorganize, and exports all operate on search results, so getting search right unblocks everything else.
---
## Non-goals
- **Block-level result granularity.** Page-level only in v1. Surfacing matching block snippets within pages adds complexity and most retrieval is page-level anyway. Defer until a real use case demands it.
- **Cross-workspace search.** Notion's integration model is per-workspace; abstracting over multiple workspaces means installing the integration in each. Document the limitation, don't try to hide it.
- **Embedding-based search.** Vector store + sync pipeline pushes this back into "build a tool" territory. Out of scope for the skill model.
- **The push-to-NotebookLM skill.** Separate spec, separate plan. Search outputs JSON; the push skill consumes it.
- **Automatic pagination of search results.** Cap at ~30 candidates pre-rerank; this is enough for top-10 reranked results. If a query genuinely has 100+ relevant matches, the user can constrain with `--databases` or `--filter`.
---
## Architectural decisions
| Decision | Choice | Rationale |
|---|---|---|
| Search strategy | **Strategy C — query expansion + rerank** | Notion's keyword search alone is the gap we're filling. Rerank-only catches synonyms but misses cross-language. Expansion + rerank covers both. |
| Workflow shape | **Standalone search skill** (browse-friendly), JSON output for downstream chaining | User searches frequently for memory-refresh; pushing to NotebookLM is occasional. Two independent skills, composable via JSON. |
| LLM model | **Claude Haiku 4.5** for both expansion and rerank | Plenty good for "rank these 30 titles + properties by relevance"; ~$0.005/query, 8-15s total latency. |
| LLM client | **`anthropic` SDK preferred, `claude -p` CLI fallback** | SDK gives structured output; CLI works without separate API key for users on Claude Code subscriptions. |
| Caching | **SHA256(query + sorted_candidate_ids) → JSON file** in `~/.cache/notion-search/` | Cheap, no infra, invalidates naturally when Notion content changes (different candidates → cache miss). 1-day TTL. |
| Property filter syntax | **JSON object** matching `32-notion-writer`'s `--properties` form | Consistency across the skill suite. |
| Failure mode | **Permissive degradation** | Rerank fail → return raw expanded-search; expansion fail → original query only. Match the parser philosophy from Phase 3c. |
---
## CLI surface
```bash
# Default — rerank + expand, terminal output
notion-search "AI agents in 2026"
# Pipe to JSON for the future push skill
notion-search "AI agents" --json | jq '.[].id'
# Constrain to specific databases
notion-search "AI agents" --databases f8f19ede-32bd-43ac-9f60-0651f6f40afe,abc-def-...
# Property filter (same JSON form as notion-writer's --properties)
notion-search "MCP" --filter '{"Status": "Done", "Topic": "AI"}'
# Fast paths
notion-search "exact term" --no-rerank # Notion API only, no LLM
notion-search "exact term" --no-expand # Skip variant generation
notion-search "AI agents" --limit 5 # Default 10
notion-search "AI agents" --no-cache # Bypass result cache
```
**Flags:**
| Flag | Default | Effect |
|---|---|---|
| `--json` | off | Emit JSON array instead of terminal table |
| `--databases <ids>` | all accessible | Comma-separated database/data_source IDs to constrain search |
| `--filter <json>` | none | Property filter applied per-database (skipped for workspace-wide search) |
| `--no-rerank` | off | Skip Claude rerank stage |
| `--no-expand` | off | Skip query-variant generation |
| `--limit <n>` | 10 | Number of results to return after rerank |
| `--no-cache` | off | Bypass cache lookup AND don't write cache |
---
## Pipeline
### 1. Query expansion (skipped if `--no-expand`)
Claude Haiku takes the original query and produces 3-5 variants covering:
- Synonyms ("AI agents" → "multi-agent orchestration", "autonomous LLM agents")
- Cross-language KR↔EN ("AI agents" → "AI 에이전트")
- Related concepts ("AI agents" → "agent SDK", "tool use")
The original query is always included. Total variants ≤ 5 to bound API calls.
Prompt approach: ask Claude for a JSON array of variants, parse with `json.loads`. On parse failure, fall back to original query only.
### 2. Search execution
For each variant, hit Notion API:
- Workspace-wide (`client.search`) when no `--databases` flag
- Per-database (`client.data_sources.query`) when `--databases` is provided. The existing `_notion_compat.resolve_data_source_id` resolves DB IDs.
Calls are made sequentially (no concurrent dispatch) to stay well under Notion's 3 req/sec average rate limit. With ≤5 variants × ≤2 DBs (typical case), that's at most 10 sequential calls — under 4 seconds total even at the slower end.
Each call returns up to 100 results; we cap candidates at 30 total (after dedup across all variants and DBs) to keep rerank costs predictable.
Dedupe by page ID. Preserve the highest position-rank if the same page appears for multiple variants (used as fallback ordering when rerank is skipped).
### 3. Property + excerpt fetch
For each candidate, gather:
- Title (from page.properties[title_prop])
- Key properties (Status, Topic, Type, Account Code, etc. — whatever exists)
- 200-char excerpt: fetch the first text-bearing block via `client.blocks.children.list` with `page_size=5`, walk the returned blocks and pick the first paragraph/heading/quote whose rich-text concatenates to non-empty content
Excerpt fallback: if none of the first 5 blocks have text content (e.g., page leads with an image, table-only, or empty), excerpt is the empty string. Rerank still runs — title + properties usually carry enough signal.
Properties often arrive inline with `client.search` results; we only re-fetch when properties are stripped (data source queries return full properties; workspace search doesn't).
### 4. Rerank (skipped if `--no-rerank`)
Cache lookup first: SHA256(original_query + sorted_candidate_ids).
Cache miss → call Claude Haiku with:
- Original (un-expanded) query
- Numbered list of candidates: `[N] Title — Properties — Excerpt`
- Asks for a JSON array of objects: `{index, score, why}` ordered by score descending
Parse, map back to candidate page objects, take top `--limit`.
Failure modes:
- LLM call error → return raw expanded-search results in candidate order, warn on stderr
- JSON parse error → same as above
- Rerank returns fewer than requested → take what's there, warn
### 5. Output
**Terminal** (default): formatted table with title, relevance, snippet, key properties, URL. ANSI color via `rich` library if available, plain text otherwise.
**JSON** (`--json`): array of objects with stable schema:
```json
[
{
"id": "abc-def-...", // dashed UUID
"url": "https://notion.so/...",
"title": "MCP server architecture...",
"relevance": 0.94, // 0.0-1.0; null if --no-rerank
"snippet": "Direct match — covers...", // null if --no-rerank
"excerpt": "First paragraph text...", // 200-char from page body
"properties": { // only properties present on the page
"Status": "Done",
"Topic": ["AI", "MCP"],
"Account Code": "D.intelligence"
}
}
]
```
The schema is stable: any future tool consuming search output (notion-to-notebooklm push, aggregation, etc.) reads this format.
---
## File structure
| File | Purpose | LOC |
|---|---|---|
| `custom-skills/31-notion-organizer/code/scripts/notion_search.py` | Main CLI + pipeline | ~400 |
| `custom-skills/31-notion-organizer/code/scripts/test_notion_search.py` | Unit tests with mocked Claude/Notion | ~250 |
| `custom-skills/31-notion-organizer/code/scripts/_search_llm.py` | LLM client abstraction (SDK + CLI fallback, prompt construction) | ~150 |
| `custom-skills/31-notion-organizer/commands/notion-search.md` | Slash command definition | ~50 |
| `custom-skills/31-notion-organizer/code/CLAUDE.md` | Add usage section | +60 lines |
| `custom-skills/31-notion-organizer/code/scripts/requirements.txt` | Add `anthropic` (optional, falls back to CLI) | +1 line |
**Total: ~850 LOC + tests + docs.** ~3 days focused work.
The `_notion_compat` helper from Phase 2 is reused (client factory, error explanation, ID resolution). No new compatibility layer needed.
---
## LLM client (`_search_llm.py`)
Abstraction so callers don't care whether SDK or CLI is in use.
```python
def call_claude(prompt: str, model: str = "claude-haiku-4-5", max_tokens: int = 1000) -> str:
"""Return Claude's text response. Raises on failure."""
if _have_anthropic_sdk() and os.getenv("ANTHROPIC_API_KEY"):
return _call_via_sdk(prompt, model, max_tokens)
if _have_claude_cli():
return _call_via_cli(prompt, model)
raise RuntimeError(
"No Claude client available. Install `anthropic` and set "
"ANTHROPIC_API_KEY, or install Claude Code CLI."
)
```
Two thin implementations behind it:
- `_call_via_sdk`: standard `anthropic.Anthropic().messages.create(...)`
- `_call_via_cli`: `subprocess.run(["claude", "-p", prompt, "--model", model], ...)`, capture stdout
Both return the assistant's text content as a single string. Callers (`expand_query`, `rerank_candidates`) parse JSON out of it.
---
## Caching
**Key:** SHA256 hex digest of `f"{query}|{','.join(sorted(candidate_ids))}"`.
**Storage:** `~/.cache/notion-search/<hash>.json`. Each file contains:
```json
{
"query": "AI agents",
"candidate_ids": ["abc...", "def..."],
"results": [...], // the reranked output
"cached_at": "2026-04-28T14:23:00Z"
}
```
**Lookup:** if file exists and `now - cached_at < TTL` (default 24h), return cached results. Else run rerank and write fresh.
**Invalidation:** TTL-only for simplicity. `--no-cache` bypasses both read and write.
**Why not invalidate on Notion content changes?** That requires tracking last-edited timestamps and computing a content hash — significant complexity for marginal benefit. With a 1-day TTL, stale results are bounded; user can `--no-cache` for fresh runs.
---
## Error handling
| Failure | Behavior |
|---|---|
| Notion API: `ObjectNotFound` on `--databases` ID | Error out with friendly message via `_notion_compat.explain_api_error` |
| Notion API: `Unauthorized` | Same |
| Query expansion fails (LLM error or JSON parse) | Use original query only, warn on stderr, continue |
| Rerank fails (LLM error or JSON parse) | Return raw expanded-search results in candidate order, warn on stderr, continue |
| Cache file corrupted | Delete cache file, treat as miss |
| Empty Notion results | Print "No matches for '<query>'" and exit 0 (not an error) |
| LLM client unavailable (no SDK + no CLI) | Error out with setup instructions if `--no-rerank` and `--no-expand` aren't both set; otherwise proceed |
| `--filter` JSON parse error | Error out before any API calls |
---
## Tests
10 tests covering the public surface, all using mocked Claude and Notion clients (no real API calls in tests):
| Test | Verifies |
|---|---|
| `test_expand_query_returns_variants` | LLM mock returns variant list including original; expansion produces 3-5 unique strings |
| `test_search_unions_and_dedupes` | Two variants returning overlapping pages produce deduped candidate set |
| `test_rerank_orders_by_relevance` | Mock rerank returns scores; output sorted score-descending |
| `test_no_rerank_returns_raw` | `--no-rerank` skips LLM, returns Notion's native order, no relevance scores |
| `test_no_expand_uses_single_query` | `--no-expand` calls Notion once with original query, no LLM expansion |
| `test_json_output_parseable` | `--json` emits valid JSON conforming to the schema in Section 5 |
| `test_cache_hit_on_repeat` | Second identical query skips LLM; cache file exists |
| `test_cache_miss_on_different_candidates` | Different candidate set hashes different → fresh rerank |
| `test_property_filter_applied` | `--filter` JSON passed to data_sources.query verbatim |
| `test_database_scope_constrains` | `--databases` causes per-DB queries instead of workspace search |
Tests live in `test_notion_search.py`. Run via `python3 test_notion_search.py` (matching the existing test_parser.py convention in 32).
---
## Out-of-scope follow-ups
- **3b-ii — `notion-to-notebooklm` push skill**: consumes search-output JSON, pushes pages as NotebookLM sources. Separate spec after this ships.
- **Block-level granularity**: surface matching blocks within pages. Adds chunking logic and richer rerank prompts. Defer.
- **Aggregation skill**: "find pages on X, write a summary page with citations." Builds on this. Future Phase 3 sub-project.
- **Multi-workspace search**: requires per-workspace integrations and credential management. Future.
---
## Implementation transition
After user approval, transition to `superpowers:writing-plans` to break this into bite-sized TDD tasks. Likely shape:
1. `_search_llm.py` skeleton (SDK + CLI client abstraction) + unit test
2. Query expansion module + tests
3. Search execution module (workspace + per-DB) + tests
4. Property + excerpt fetch + tests
5. Rerank module (with prompt + JSON parsing) + tests
6. Cache layer + tests
7. CLI assembly (argparse + output formatting) + tests
8. Slash command + 31's CLAUDE.md update + final integration smoke test
Each step independently testable. Total ~3 days estimated, mirrors Phase 3c's pacing.