our-claude-skills/docs/superpowers/specs/2026-06-27-notion-writer-cli-enhancements-design.md

# Notion Writer (Skill 32) — CLI Enhancements: File Uploads + Markdown Engine

> **Date**: 2026-06-27
> **Status**: Approved (brainstorming)
> **Scope**: Two additive features — (Tier 1) local file/image upload via the `ntn` CLI, and (Tier 2) an opt-in "markdown engine" that writes through Notion's native enhanced-markdown endpoints.
> **Predecessor**: 2026-04-27 Extended Block Coverage (callouts/toggles/columns/mentions), CLAUDE.md v1.2.0
> **Target version**: bump CLAUDE.md footer to v1.3.0

---

## Goal

Add two capabilities to `custom-skills/32-notion-writer/code/scripts/notion_writer.py`:

1. **Tier 1 — Local media uploads.** Today `![alt](path)` is effectively broken (the inline link regex leaves a stray `!` and renders the image as a text link). Make standalone image references work: local files are uploaded via `ntn files create` and embedded as Notion `image` blocks with a `file_upload` reference; remote URLs become `external` image blocks. This fills a genuine capability gap — the parser previously had no image support at all.

2. **Tier 2 — Markdown engine.** Add `--engine markdown` to write content through Notion's native enhanced-markdown endpoints (`POST /v1/pages` with `markdown`, `PATCH /v1/pages/:id/markdown`) instead of the local block converter. A dialect translator converts the skill's existing authoring syntax (GitHub alerts, Pandoc columns, `@[mentions]`) into Notion-flavored markdown so a single source document works in both engines.

The default engine stays `blocks` — every existing call and downstream script behaves identically. Both engines share one media-upload path.

---

## Non-goals

- **Re-platforming onto `ntn`** (Tier 3, rejected). The skill keeps its token-based SDK core; `ntn` is used only as a file-upload subprocess.
- **Reading pages back as markdown** (`GET /v1/pages/:id/markdown`). Not needed for a writer; parked.
- **Workspace-mismatch hard guard.** User chose plain shell-out; preflight surfaces the `ntn` target workspace as an informational note but does not block on mismatch.
- **Generic local attachments** (`<file>`, `<pdf>`, `<video>`, `<audio>`) in v1. The upload mechanism is media-type agnostic, but v1 wires only `![]()` images. Other media use the same walk later — parked.
- **Legacy markdown update commands** (`insert_content` for arbitrary positions, `replace_content_range`). Only `insert_content{position:end}` (append) and `replace_content` (replace) are used.
- **Bidirectional dialect translation** (Notion enhanced markdown → skill dialect). One direction only.

---

## Architectural decisions (locked during brainstorming)

| Decision | Choice | Rationale |
|---|---|---|
| File-upload transport | **Shell out to `ntn files create --plain`** | User choice. The CLI does the full upload lifecycle in one command. |
| Upload-failure visibility | **Loud, not silent** — preflight checks `ntn` is installed + logged in and prints the target workspace; upload errors abort with the file path + `ntn` stderr | File uploads are workspace-scoped; a mismatch must surface, not silently misfire. |
| Markdown dialect handling | **Translate** skill dialect → Notion enhanced markdown | User choice — one authoring document works in both engines. |
| Engine selection | **`--engine {blocks,markdown}`, default `blocks`** | Zero regression for existing callers; opt-in to the new path. |
| Local-media trigger | **Auto-detect** local paths in `![]()`; remote URLs left external | User choice — most ergonomic; replaces the silent-strip behavior. |
| Parser purity | **Preserve it** — uploads happen in a post-parse block walk, never inside the parser | The 2026-04-27 design locked "parser makes no API calls"; honored by isolating impurity. |
| Markdown transport | **`client.request(...)` low-level calls**, not typed SDK methods | The markdown endpoints have no stable typed SDK methods; `request` is version-safe and avoids SDK coupling. |
| Notion API version | **`2026-03-11` for the markdown engine**; blocks engine keeps the SDK default (`2025-09-03`) | Markdown endpoints require the newer version; blocks path stays untouched. |
| Python env | **Create venv + `requirements.txt`** at the documented path | The documented venv doesn't exist; the skill can't currently run. |

---

## Module structure

Two new modules keep `notion_writer.py` from bloating; each has one purpose and is testable in isolation.

| File | Status | Role |
|---|---|---|
| `scripts/notion_writer.py` | modified | CLI surface, engine routing, post-parse media materialization, two-phase markdown writes |
| `scripts/_notion_compat.py` | modified | Version-aware `make_client`; markdown-endpoint helpers (`create_page_markdown`, `append_markdown`, `replace_markdown`) |
| `scripts/ntn_files.py` | **new** | `ntn files create` subprocess wrapper; preflight; `upload(path) -> file_upload_id` |
| `scripts/md_translate.py` | **new** | Pure functions: skill dialect → Notion enhanced markdown |
| `scripts/requirements.txt` | modified | Pin `notion-client`, `python-dotenv` |
| `scripts/venv/` | **new** | Created at documented path; not committed (gitignored) |

Dependencies between units:
- `notion_writer` depends on `_notion_compat`, `ntn_files`, `md_translate`.
- `md_translate` and the parser are pure (deterministic, no I/O).
- `ntn_files` owns all subprocess I/O.
- `_notion_compat` owns all Notion HTTP I/O.

---

## Tier 1 — Local media uploads (blocks engine)

### Parsing (pure)
Add an image detector to `_parse_lines`, ordered with the other block detectors. A **standalone** image line matches:

```python
IMAGE_RE = re.compile(r'^\s*!\[([^\]]*)\]\(([^)\s]+)\)\s*$')
```

`create_image_block(alt, target)` always emits the **external shape** regardless of whether `target` is local or remote:

```python
{"object":"block","type":"image",
 "image":{"type":"external","external":{"url": target},
          "caption": parse_rich_text(alt)}}
```

The parser stays pure and upload-unaware. Inline images inside a paragraph are out of scope (rare); only standalone image lines become image blocks.

### Materialization (impure, isolated)
After parsing and before sending, `notion_writer` runs a recursive block walk:

```python
materialize_local_media(blocks, base_dir, uploader) -> blocks
```

- Walks `blocks` and every nested `children` list (so images inside toggles/columns are covered).
- For each `image` block whose `external.url` is **not** `http(s):` and resolves to an existing file under `base_dir`:
  - `file_upload_id = uploader.upload(resolved_path)`
  - rewrite to `{"image":{"type":"file_upload","file_upload":{"id": file_upload_id}, "caption": ...}}`
- Remote/external images are left unchanged.
- A local path that does not exist on disk → abort with a clear error (`image references missing file: <path>`), since the user clearly intended a local embed.

`base_dir` = the `--file` argument's parent directory; for `--stdin`, the current working directory.

### Upload wrapper — `ntn_files.py`

```python
def preflight() -> WorkspaceInfo        # cached; once per process
def upload(path: Path) -> str           # returns file_upload_id
```

- `preflight()`:
  - `shutil.which("ntn")` → if missing, raise with install hint (`curl -fsSL https://ntn.dev | bash`).
  - `ntn api v1/users/me` (JSON) → parse `workspace_name` / `workspace_id`; print one informational line: `ntn → workspace "<name>"`. Does not block on mismatch (per locked decision). If the call fails (not logged in), raise with `ntn login` hint.
- `upload(path)`:
  - `subprocess.run(["ntn","files","create","--plain"], stdin=open(path,"rb"), capture_output=True, text=True)`
  - on non-zero exit → raise `NtnUploadError(path, stderr)`.
  - parse stdout: the upload ID is the first tab-separated field of the first line.
  - returns the ID. (Notion requires attaching within ~1 hour; the immediate block append in the same run is well inside that window.)

---

## Tier 2 — Markdown engine

### Dialect translator — `md_translate.py`

`translate(content: str) -> str` converts the skill dialect to Notion enhanced markdown. Only three constructs differ; everything else passes through verbatim.

| Skill dialect | → Notion enhanced markdown |
|---|---|
| `> [!NOTE]` + contiguous `>` body | `<callout icon="ℹ️" color="blue_bg">\n\t<body>\n</callout>` |
| `::: columns` / `::: column` / `:::` | `<columns>\n\t<column>…</column>\n\t<column>…</column>\n</columns>` |
| `<details>` / `<summary>` toggle | `<details>\n<summary>…</summary>\n\t<children>\n</details>` — children **re-indented one tab** |
| `@[Title](id\|url)` | `<mention-page url="<resolved>">Title</mention-page>` |
| headings, lists, to-dos, quotes, fenced code, `---`, **bold**/*italic*/`code`/`~~strike~~`/links, pipe tables | **pass through unchanged** |

Notes:
- Callout icon/color reuse the existing `ALERT_TYPES` map; Notion color names use the `_bg` suffix (`blue_bg`), not the block-API `blue_background`. A small adapter maps one to the other.
- Children inside `<callout>`, `<columns>`, and `<details>` are **tab-indented** one level (enhanced-markdown requirement — toggle and callout children "must be indented"). The translator emits tabs; this is why `<details>` is translated rather than passed through verbatim.
- Pipe tables pass through — Notion's own enhanced-markdown example renders a standard pipe table, so no `<table>` rewrite is needed in v1. (If live testing shows pipe tables are rejected, the fallback is a `<table>` rewrite — noted as the one open verification item.)
- Mention resolution reuses `extract_notion_id`; an unresolvable target degrades to plain `@Title` text (same posture as the blocks engine).
- The translator is line-oriented and reentrant for the two container constructs, mirroring `_parse_lines`.

### Transport helpers — `_notion_compat.py`

All via `client.request(...)` so no SDK typed-method dependency:

```python
def make_client(api_key, notion_version=None) -> Client   # version override added
def create_page_markdown(client, parent, properties, markdown) -> dict
    # POST v1/pages  body={parent, properties, markdown}
def append_markdown(client, page_id, markdown) -> dict
    # PATCH v1/pages/{id}/markdown
    # body={"type":"insert_content","insert_content":{"content":md,"position":{"type":"end"}}}
def replace_markdown(client, page_id, markdown, allow_deleting=False) -> dict
    # PATCH v1/pages/{id}/markdown
    # body={"type":"replace_content","replace_content":{"new_str":md,"allow_deleting_content":allow_deleting}}
```

**Two-client split for version safety.** The markdown *write* helpers use a client built with `notion_version="2026-03-11"`. All schema/property/upsert-lookup operations (`resolve_data_source_id`, `get_schema`, `coerce_properties`, `find_existing_page`) continue on the **default-version** client, so the 2025-09-03 data-source behavior the skill already relies on is unchanged. Only the three markdown write calls cross to the newer version. The data-source parent (`{type:data_source_id,...}`) is expected to remain valid under `2026-03-11` (confirmed in the live smoke test).

### Routing by operation

| CLI op | blocks engine (default) | markdown engine |
|---|---|---|
| Create DB row (`-d -t`) | `pages.create(parent, properties, children=blocks)` | `create_page_markdown(parent, properties, translate(content))` |
| Append to page (`-p`) | `blocks.children.append` | `append_markdown(page_id, translate(content))` |
| Replace page (`-p -r`) | delete-all-blocks + append | `replace_markdown(page_id, translate(content))` |
| Upsert (`--upsert-by`) | unchanged (property update + body) | property update via SDK + body via `replace_markdown` |

The markdown `replace_content` path replaces the brittle paginated delete-every-block logic for that engine. `--upsert-by` lookup/property-coercion logic is shared unchanged; only the body write differs.

### Markdown engine + local images (two-phase write)

The markdown endpoints take a URL for `![](url)`, and a `file_upload` id is not a URL, so local images cannot be inlined into the markdown string. Handling, reusing the Tier 1 upload + append helpers:

1. Split standalone **local** image refs out of the content (same `IMAGE_RE`, local targets only; remote `![](http…)` stay inline in the markdown); translate the remainder; write it (create / append / replace).
2. Upload the local images and append them as `image` blocks to the resulting page via `blocks.children.append`.

**Known limitation (documented):** in the markdown engine, local images land at the **end** of the page, not their original position. When image position matters, use the blocks engine. Remote images keep their position (they stay inline in the markdown).

---

## Flags & versioning

New CLI flags:

| Flag | Default | Meaning |
|---|---|---|
| `--engine {blocks,markdown}` | `blocks` | Select the write path |
| `--notion-version VERSION` | engine-dependent | Override the API version (markdown defaults to `2026-03-11`) |
| `--allow-deleting-content` | off | Permit markdown `replace_content` to delete child pages/databases |

Validation: `--engine markdown` with `--upsert-by` on a property the markdown path can't update falls back to the shared coercion logic (no new restriction). `--allow-deleting-content` is ignored by the blocks engine (warn if combined).

---

## Error handling

Permissive where the user's intent is ambiguous; loud where they clearly intended an action that failed.

| Failure | Behavior |
|---|---|
| `ntn` not installed | Abort before any write; install hint |
| `ntn` not logged in (`users/me` fails) | Abort; `ntn login` hint |
| `ntn files create` non-zero exit | Abort; print file path + `ntn` stderr |
| Local image path missing on disk | Abort; `image references missing file: <path>` |
| Markdown `replace`/selection no match | Route `validation_error` through `explain_api_error` with a markdown-specific hint |
| `replace_content` would delete child pages | Surface the API's affected-items list; suggest `--allow-deleting-content` |
| Synced-page update rejected | Clear message (synced pages can't be updated) |
| Unresolvable `@[mention]` | Degrade to plain `@Title` (both engines) |
| Two-phase image append fails after text write | Warn that text was written but images were not; non-zero exit |

`explain_api_error` gains markdown-endpoint cases; all other paths reuse existing handling.

---

## Testing

TDD. New unit suites alongside the existing `test_parser.py` (28 tests, must stay green).

| Suite | Verifies |
|---|---|
| `test_parser.py` (extended) | `![alt](url)` → external image block; `![alt](./local.png)` → external-shape with local URL pre-materialization; non-image `!` text unaffected; existing 28 stay green |
| `test_md_translate.py` (new) | callout (each alert type → icon+`_bg` color, tab-indented body); columns → `<columns>/<column>`; mention id+url+invalid; passthrough of headings/lists/code/tables/`<details>`; escaping |
| `test_ntn_files.py` (new) | `subprocess.run` mocked: command shape, `--plain` first-field parse, non-zero → `NtnUploadError`, preflight missing-`ntn`, preflight not-logged-in; local-vs-remote branch in `materialize_local_media` |
| `test_engine_routing.py` (new) | client `request` mocked: each op×engine calls the right path/method/body; markdown client built with `2026-03-11`; two-phase image append fires after text write |

**Live smoke test** (manual, end of implementation) against the "Working with AI" data source `f8f19ede-32bd-43ac-9f60-0651f6f40afe`, both engines:
1. `--test` connection.
2. Blocks engine: create a row with a local image → confirm `file_upload` image block renders.
3. Markdown engine: create a row from a doc using `[!NOTE]` + `:::columns` → confirm callout + columns render; confirm a local image appended at end.
4. Markdown engine `--replace` on the same page → confirm `replace_content` works.

This verifies the one open item (pipe-table passthrough) and the data-source parent under `2026-03-11`.

---

## File changes

| File | Change |
|---|---|
| `scripts/notion_writer.py` | `--engine`/`--notion-version`/`--allow-deleting-content` flags; image detector + `create_image_block`; `materialize_local_media` walk; markdown-engine routing; two-phase image append |
| `scripts/_notion_compat.py` | `make_client(notion_version=…)`; `create_page_markdown`/`append_markdown`/`replace_markdown`; markdown cases in `explain_api_error` |
| `scripts/ntn_files.py` | **new** — preflight + `upload` |
| `scripts/md_translate.py` | **new** — `translate` + construct translators |
| `scripts/test_md_translate.py`, `scripts/test_ntn_files.py`, `scripts/test_engine_routing.py` | **new** test suites |
| `scripts/test_parser.py` | image tests |
| `scripts/requirements.txt` | pin deps |
| `code/CLAUDE.md`, `SKILL.md` | document engines, file uploads, flags, the markdown-engine image limitation; v1.3.0 changelog |
| `.gitignore` | ensure `venv/` ignored |

---

## Out of scope (parking lot)

- Generic local `<file>`/`<pdf>`/`<video>`/`<audio>` upload (mechanism ready; not wired in v1).
- `GET /v1/pages/:id/markdown` round-trip read.
- Workspace-mismatch hard guard (escalate preflight from warn → block) if it proves necessary in practice.
- Notion enhanced markdown → skill dialect (reverse translation), which would also serve the parked Phase 3b Notion-as-RAG export.
- `<table>` rewrite for pipe tables — only if live testing shows pipe tables are rejected by the markdown endpoint.
- Inline (non-standalone) images.

---

## Implementation transition

After user approval of this spec, invoke `superpowers:writing-plans`. The plan will likely sequence:

1. Create venv + `requirements.txt`; confirm existing 28 tests run green (baseline).
2. `ntn_files.py` + `test_ntn_files.py` (subprocess mocked).
3. Image detector + `create_image_block` + `materialize_local_media` + parser image tests (blocks engine Tier 1 complete; live single-image check).
4. `md_translate.py` + `test_md_translate.py` (pure, no API).
5. `_notion_compat` markdown helpers + version-aware client + `test_engine_routing.py`.
6. Wire `--engine`/flags + routing + two-phase image append in `notion_writer.py`.
7. Docs (CLAUDE.md/SKILL.md) + v1.3.0 bump.
8. Live smoke test (both engines) against the "Working with AI" DB; resolve the pipe-table verification item.

Each step is independently testable and revertable.