diff --git a/docs/superpowers/specs/2026-04-27-notion-writer-extended-block-coverage-design.md b/docs/superpowers/specs/2026-04-27-notion-writer-extended-block-coverage-design.md new file mode 100644 index 0000000..821b0ee --- /dev/null +++ b/docs/superpowers/specs/2026-04-27-notion-writer-extended-block-coverage-design.md @@ -0,0 +1,229 @@ +# Notion Writer (Skill 32) — Extended Block Coverage Design + +> **Date**: 2026-04-27 +> **Status**: Approved (brainstorming) +> **Scope**: Phase 3c — add callout, toggle, column, and page-mention block support to `notion_writer.py`'s markdown→Notion parser +> **Sequence**: First of three Phase 3 sub-projects (3c → 3b → 3a per user-approved order) +> **Predecessor**: Phase 2 commit `144a17c` (multi-source data API migration) + +--- + +## Goal + +Extend the markdown→Notion-blocks converter in `custom-skills/32-notion-writer/code/scripts/notion_writer.py` to support four block types currently unhandled: **callouts**, **toggles**, **columns**, and **inline page mentions**. The parser remains pure (no API calls during parse), preserves existing block-type behavior unchanged, and stays under the project's "small, well-tested" line-scanner architecture. + +This work also lays the foundation for Phase 3b (Notion-as-RAG export), which needs a clean blocks→markdown reverse converter; the syntax chosen here must round-trip cleanly. + +--- + +## Non-goals + +- Notion blocks not in the four-item list above (image upload, bookmark, embed, equation, breadcrumb, table-of-contents, synced block — defer until a concrete use case demands them). +- LLM-driven content distillation (belongs to Phase 3b downstream of `90-reference-curator`). +- Search-by-title page-mention resolution (rejected during brainstorming for purity reasons; explicit ID/URL only). +- Cross-block-type round-trip tests (deferred to Phase 3b when the reverse converter exists). + +--- + +## Architectural decisions (locked during brainstorming) + +| Decision | Choice | Rationale | +|---|---|---| +| Syntax style | **Hybrid**: GitHub alerts (`> [!NOTE]`) for callouts, HTML5 `
` for toggles, Pandoc fenced div (`::: columns`) for columns | Each element uses the most-portable syntax available; renders correctly on GitHub for the formats that have native support | +| Container nesting | **Full recursion** — anything legal at top level is legal inside a toggle or column, including nested toggles/columns | Dev logs commonly put code blocks and lists inside toggles; the limitation is unacceptable | +| Page-mention syntax | **ID-or-URL** — `@[Title](page-id)` and `@[Title](https://notion.so/Page-id)` both accepted | URL form is what users naturally paste; ID form is what 3b's reverse converter emits | +| Implementation shape | **Reentrant flat parser** — `markdown_to_notion_blocks` accepts string or list of lines; container detectors recurse on inner content | Smallest diff, preserves existing line-scanner architecture, additive | + +--- + +## Public API + +`markdown_to_notion_blocks` becomes reentrant by widening its parameter type: + +```python +def markdown_to_notion_blocks(content: Union[str, List[str]]) -> List[Dict[str, Any]]: + lines = content.splitlines() if isinstance(content, str) else list(content) + return _parse_lines(lines) +``` + +The current loop body moves into a new private `_parse_lines(lines: List[str])`. All existing callers continue to pass strings; nothing breaks. + +--- + +## New block factories + +Added alongside existing `create_*_block` helpers: + +| Factory | Output shape | +|---|---| +| `create_callout_block(rich_text_spans, alert_type)` | `{"type": "callout", "callout": {"rich_text": ..., "icon": {"type": "emoji", "emoji": "ℹ️"}, "color": "blue_background"}}` | +| `create_toggle_block(summary_spans, children_blocks)` | `{"type": "toggle", "toggle": {"rich_text": ..., "children": ...}}` | +| `create_column_list_block(column_blocks_lists)` | `{"type": "column_list", "column_list": {"children": [{"type": "column", "column": {"children": [...]}}]}}` | + +Page mentions are inline rich-text — no new block factory; one new pattern in `INLINE_PATTERNS`. + +--- + +## Constants + +Added at the top of the file alongside `INLINE_PATTERNS`: + +```python +ALERT_TYPES = { + 'NOTE': ('ℹ️', 'blue_background'), + 'TIP': ('💡', 'green_background'), + 'IMPORTANT': ('☝️', 'purple_background'), + 'WARNING': ('⚠️', 'yellow_background'), + 'CAUTION': ('🚨', 'red_background'), +} +ALERT_RE = re.compile(r'^\s*\[!(NOTE|TIP|IMPORTANT|WARNING|CAUTION)\]\s*$') +``` + +The icons match GitHub's render (shape and hue); colors slot into Notion's named-color palette directly. + +--- + +## Block-type details + +### Callout + +- **Detect**: blockquote line where the first content line (after `>`) matches `ALERT_RE` +- **Body**: collect subsequent contiguous `>` lines until a non-`>` line; concatenate with `\n` separators into one rich-text run (matches GitHub's render — alerts are single-block, no nested lists/code). Body text passes through `parse_rich_text` so bold/italic/code/links/mentions work inside the callout. +- **Unknown alert type** (e.g., `> [!FOO]`): fall through to existing quote handling, `[!FOO]` preserved as text +- **Notion shape**: callout block with `rich_text`, `icon.emoji`, `color` + +### Toggle + +- **Detect**: line equals `
` (whitespace-tolerant). Single-line forms like `
Xbody
` are **not supported** in v1 — require the multi-line layout (open tag on its own line). Most markdown editors that emit `
` already use multi-line. +- **Summary**: next non-blank line if it matches `(.*)`; if absent, summary is empty rich-text. Summary text passes through `parse_rich_text` so bold/italic/code/links/mentions work inside it. +- **Body**: lines between `` (or first non-summary line) and the matching `
`, depth-tracked via `details_depth` int so `
` inside `
` works correctly +- **Recurse**: body lines fed to `_parse_lines`, result attached as `children` +- **Unclosed** at EOF: emit body lines as plain paragraphs, one-line stderr warning + +### Columns + +- **Detect**: line equals `::: columns` +- **Inner separators**: `::: column` begins each child column +- **Close**: bare `:::` closes either a column or the wrapper, depending on `colon_depth` +- **Recurse**: each column's lines fed to `_parse_lines`, results attached as `children` of `column` blocks; `column` blocks attached as `children` of `column_list` +- **Validation**: ≥2 columns required by Notion; single-column `::: columns` block degrades to plain paragraphs (skip wrapper) +- **Unclosed** at EOF: emit lines as plain paragraphs, one-line stderr warning + +### Page mentions (inline rich-text) + +- **Pattern**: new entry in `INLINE_PATTERNS`: `('mention', re.compile(r'@\[([^\]]+)\]\(([^)\s]+)\)'))` +- **Resolution**: `extract_notion_id(target)` reuses existing logic — accepts raw 32-char hex, dashed UUID, or full Notion URL +- **Valid ID**: rich-text span with shape: + ```python + { + "type": "mention", + "mention": {"type": "page", "page": {"id": resolved_id}}, + "plain_text": title_text, + } + ``` +- **Invalid ID** (extract returns None): plain text `@Title` with no link annotation, no warning (graceful — many false positives are possible with `@`) + +--- + +## Recursion mechanics + +The line-scanning loop in `_parse_lines` gets three new detectors, ordered before existing ones: + +``` +for line in lines: + if in_code_fence: ... # existing — wins over all + if in_table: ... # existing + if line == '
': ... # NEW — capture to
, recurse + if line == '::: columns': ... # NEW — capture columns, recurse per column + if line.startswith('>') and ALERT_RE.match(rest): ... # NEW — collect callout body + # fall through to existing detectors (heading/list/quote/code-fence/table/etc.) +``` + +Recursion algorithm for the two recursive container types: + +1. Detect open delimiter at line *i* +2. Scan forward to matching close, tracking depth for nested same-kind delimiters (`
`/`
` for toggles, `:::`/`::: columns` for columns) +3. Slice inner lines into a sub-list (per-column sub-lists for columns) +4. Recursively call `_parse_lines(sub_lines)` +5. Wrap result in the appropriate Notion container block +6. Advance the outer loop index past the close + +State held by the scanner: two ints (`details_depth`, `colon_depth`), since the two container kinds don't share delimiters with each other or with existing scanner state. + +--- + +## Error handling + +Permissive philosophy — never fail the push, degrade to readable text on malformed input. Same posture as the Phase 2 anchor-link fix. + +| Failure | Behavior | +|---|---| +| Unclosed `
` at EOF | Emit body lines as paragraphs; one-line stderr warning | +| Unclosed `::: columns` at EOF | Same | +| Single-column `::: columns` block | Degrade to paragraphs (skip wrapper, no warning — caller's choice) | +| Unknown alert type `> [!FOO]` | Fall through to existing quote rendering, preserving `[!FOO]` text | +| Invalid mention ID | Plain text `@Title`, no warning (avoid false-positive spam from any `@[x](y)` pattern) | +| Missing `` in `
` | Empty summary, body parses normally | + +--- + +## Testing + +Twelve new tests added to `test_parser.py`, bringing total from 16 → 28. Round-trip (markdown → blocks → markdown) tests deferred to Phase 3b. + +| Test | What it verifies | +|---|---| +| `test_callout_note` | `> [!NOTE]` emits `callout` with ℹ️ icon and `blue_background` | +| `test_callout_tip` | `> [!TIP]` → 💡, green | +| `test_callout_important` | `> [!IMPORTANT]` → ☝️, purple | +| `test_callout_warning` | `> [!WARNING]` → ⚠️, yellow | +| `test_callout_caution` | `> [!CAUTION]` → 🚨, red | +| `test_callout_unknown_falls_through` | `> [!BOGUS]` becomes a quote, `[!BOGUS]` preserved as text | +| `test_toggle_basic` | `
Xbody
` → toggle with summary X and paragraph children | +| `test_toggle_nested_blocks` | Toggle containing list + code block (validates `_parse_lines` recursion) | +| `test_toggle_nested_toggle` | Toggle inside toggle (validates `details_depth` tracking) | +| `test_toggle_unclosed_falls_through` | Missing `
` produces text paragraphs, no crash | +| `test_columns_two` | Basic two-column layout with paragraphs | +| `test_columns_with_blocks` | Columns containing lists/code (validates per-column recursion) | +| `test_columns_single_degrades` | Single-column `::: columns` block emits paragraphs, no `column_list` | +| `test_mention_id` | `@[Title](32-hex-id)` produces `mention` rich-text span | +| `test_mention_url` | `@[Title](https://notion.so/Title-id)` resolves to ID | +| `test_mention_invalid_falls_back` | Bad ID → plain text `@Title`, no link/mention annotation | + +(Test count is 16 in this list because the 5 callouts are listed individually; some may collapse into parametrized tests during implementation. Total stays ≥12 new.) + +--- + +## File changes + +| File | Change | +|---|---| +| `custom-skills/32-notion-writer/code/scripts/notion_writer.py` | Refactor `markdown_to_notion_blocks` to be reentrant; add `_parse_lines`; add 3 block factories + 1 inline pattern; add `ALERT_TYPES` and `ALERT_RE` constants | +| `custom-skills/32-notion-writer/code/scripts/test_parser.py` | +12 tests; total 28 | +| `custom-skills/32-notion-writer/code/CLAUDE.md` | Document new syntax in the Markdown Support section; bump version footer to 1.2.0 with changelog entry | + +No new files. No new dependencies. The parser remains pure. + +--- + +## Out of scope (parking lot) + +- **3b — Notion-as-RAG export**: requires a blocks → markdown reverse converter that emits the same syntax this design defines. The hybrid syntax was chosen partly to round-trip cleanly. Implementation deferred. +- **3a — Metadata-aware migration**: cross-database moves with auto-mapped properties, dry-run diff, type-aware transforms. Builds on `31-notion-organizer/scripts/schema_migrator.py`. Deferred. +- **Block-type expansion** beyond the four named: image (upload), bookmark, embed, equation, synced block, breadcrumb, table-of-contents — added when concrete use cases demand them. + +--- + +## Implementation transition + +After this spec is approved by the user, transition to `superpowers:writing-plans` to produce the step-by-step implementation plan. The plan will likely break into: + +1. Refactor `markdown_to_notion_blocks` to reentrant form (no behavior change; verify all 16 existing tests still pass) +2. Add callout detector + factory + 6 callout tests +3. Add toggle detector + factory + 4 toggle tests (depth tracking covered here) +4. Add columns detector + factory + 3 columns tests (depth tracking covered here) +5. Add page-mention inline pattern + 3 mention tests +6. Update `notion_writer.py` CLAUDE.md docs + version bump +7. Run full test suite (28 expected); commit + +Each step is independently testable and revertable.