Files

Andrew Yim b1c2dca080 feat(reference-curator): add --save-raw, --no-distill flags + OurSEO crawler integration

- Add --save-raw flag: saves intact crawled markdown in raw/ subdirectory
  alongside distilled content, with YAML frontmatter for traceability
- Add --no-distill flag: skip stages 4-5 (distiller + QA reviewer),
  pure archival mode for raw documentation capture
- Integrate OurSEO BaseAsyncClient + PageAnalyzer as seo-aiohttp backend:
  token-bucket rate limiting, semaphore concurrency, tenacity retries,
  full page metadata extraction (headings, links, schema, OG)
- New seo_crawler_adapter.py: crawl URLs, sitemaps, link discovery, manifests
  with progress tracking and resume support
- Update crawler selection: docs sites now default to seo-aiohttp
- Update crawl_config.yaml with seo-aiohttp routing rules
- Update pipeline orchestrator with flag resolution and skip paths
- Update README.md and USER-GUIDE.md with raw mode documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-12 23:26:50 +09:00

4.0 KiB

Raw Blame History

Markdown Exporter

Exports approved reference content as structured markdown files for project knowledge or fine-tuning datasets.

Trigger Keywords

"export references", "generate project files", "create markdown output", "export for fine-tuning", "build knowledge base"

Export Types

Type	Format	Use Case
`project`	Nested markdown	Claude Projects knowledge
`raw`	Raw markdown	Archival, full-text reference, NotebookLM sources
`finetuning`	JSONL	Model fine-tuning dataset

Workflow

Step 1: Export Project Files

uv run python scripts/exporter.py project \
  --output ~/reference-library/exports/ \
  --min-score 0.80 \
  --structure nested_by_topic

Output structure:

exports/
├── INDEX.md
├── prompt-engineering/
│   ├── _index.md
│   ├── 00-chain-of-thought.md
│   └── 01-few-shot-prompting.md
└── claude-models/
    ├── _index.md
    └── 00-model-comparison.md

Step 2: Generate INDEX

uv run python scripts/exporter.py index --output ~/reference-library/exports/INDEX.md

Step 3: Add Cross-References

uv run python scripts/exporter.py crossrefs --input ~/reference-library/exports/

Step 4: Verify Export

uv run python scripts/exporter.py verify --path ~/reference-library/exports/

Step 5: Fine-tuning Export (Optional)

uv run python scripts/exporter.py finetuning \
  --output ~/reference-library/exports/training.jsonl \
  --max-tokens 4096

JSONL format:

{
  "messages": [
    {"role": "system", "content": "You are an expert on AI and prompt engineering."},
    {"role": "user", "content": "Explain {title}"},
    {"role": "assistant", "content": "{structured_content}"}
  ],
  "metadata": {"source": "{url}", "quality_score": 0.92}
}

Step 6: Log Export Job

uv run python scripts/exporter.py log --name "April 2026 Export" --type project_files --docs 45

Raw Export (when `--save-raw` or `--no-distill`)

Raw alongside distilled (`--save-raw`)

uv run python scripts/exporter.py raw \
  --output ~/reference-library/exports/{topic-slug}/raw/ \
  --source-dir {crawl-raw-dir} \
  --bundle {topic-slug}-raw-complete.md

Output structure:

exports/{topic-slug}/
├── INDEX.md              # Includes links to raw/ section
├── 00-page-name.md       # Distilled (normal)
├── raw/
│   ├── 00-page-name.raw.md
│   ├── 01-page-name.raw.md
│   └── {topic-slug}-raw-complete.md
└── manifest.json

Raw-only mode (`--no-distill`)

uv run python scripts/exporter.py raw \
  --output ~/reference-library/exports/{topic-slug}/ \
  --source-dir {crawl-raw-dir} \
  --bundle {topic-slug}-raw-complete.md \
  --primary

Output structure (raw files are primary, no subdirectory wrapper):

exports/{topic-slug}/
├── README.md
├── 00-page-name.raw.md
├── 01-page-name.raw.md
├── {topic-slug}-raw-complete.md
└── manifest.json

The --primary flag tells the exporter that raw files are the main output. README is generated from raw file frontmatter (source URLs, crawl timestamps).

Note: When combined with --export-format fine_tuning, the JSONL exporter uses raw content directly as the assistant response with a simpler system prompt (no structured summaries available).

Scripts

Command	Purpose
`exporter.py project`	Export as nested markdown files
`exporter.py raw`	Export raw crawled markdown files with optional bundle
`exporter.py finetuning`	Export as JSONL training dataset
`exporter.py index`	Generate INDEX.md table of contents
`exporter.py crossrefs`	Add cross-reference links
`exporter.py verify`	Verify export integrity
`exporter.py log`	Log export job to DB

Integration

From	To
quality-reviewer (approved)	→
web-crawler (raw files)	Raw markdown input (--save-raw / --no-distill)
→	Project knowledge / Fine-tuning dataset

4.0 KiB Raw Blame History