Andrew Yim
341d5f5a5b
Custom Skills (ourdigital-custom-skills/): - 00-ourdigital-visual-storytelling: Blog featured image prompt generator - 01-ourdigital-research-publisher: Research-to-publication workflow - 02-notion-organizer: Notion workspace management - 03-research-to-presentation: Notion research to PPT/Figma - 04-seo-gateway-strategist: SEO gateway page strategy planning - 05-gateway-page-content-builder: Gateway page content generation - 20-jamie-brand-editor: Jamie Clinic branded content GENERATION - 21-jamie-brand-guardian: Jamie Clinic content REVIEW & evaluation Refinements applied: - All skills converted to SKILL.md format with YAML frontmatter - Added version fields to all skills - Flattened nested folder structures - Removed packaging artifacts (.zip, .skill files) - Reorganized file structures (scripts/, references/, etc.) - Differentiated Jamie skills with clear roles 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
110 lines
4.2 KiB
Markdown
110 lines
4.2 KiB
Markdown
# Research & Documentation Skill Evaluations
|
|
|
|
Evaluation scenarios for testing the Research & Documentation skill across different Claude models.
|
|
|
|
## Purpose
|
|
|
|
These evaluations ensure the Research & Documentation skill:
|
|
- Searches across Notion workspace effectively
|
|
- Synthesizes information from multiple sources
|
|
- Selects appropriate research report format
|
|
- Creates comprehensive documentation with proper citations
|
|
- Works consistently across Haiku, Sonnet, and Opus
|
|
|
|
## Evaluation Files
|
|
|
|
### basic-research.json
|
|
Tests basic research workflow with synthesis across multiple Notion pages.
|
|
|
|
**Scenario**: Research Q4 product roadmap and create summary
|
|
**Key Behaviors**:
|
|
- Searches Notion for roadmap-related pages
|
|
- Fetches multiple relevant pages (roadmap, product docs, meeting notes)
|
|
- Synthesizes information from different sources
|
|
- Selects appropriate format (Research Summary)
|
|
- Includes citations linking back to source pages
|
|
- Creates structured document with clear sections
|
|
|
|
### research-to-database.json
|
|
Tests creating research documentation in a Notion database with properties.
|
|
|
|
**Scenario**: Research competitor landscape and save to Research database
|
|
**Key Behaviors**:
|
|
- Searches for existing competitive intelligence in Notion
|
|
- Identifies Research database as target
|
|
- Fetches database schema to understand properties
|
|
- Creates page with correct property values (Research Type, Status, Date, etc.)
|
|
- Structures content with comparison format
|
|
- Includes source citations for both Notion pages and external research
|
|
|
|
## Running Evaluations
|
|
|
|
1. Enable the `research-documentation` skill
|
|
2. Submit the query from the evaluation file
|
|
3. Verify the skill searches Notion workspace comprehensively
|
|
4. Check that multiple source pages are fetched and synthesized
|
|
5. Verify appropriate format is selected (Research Summary, Comprehensive Report, Quick Brief, Comparison)
|
|
6. Confirm citations link back to sources
|
|
7. Test with Haiku, Sonnet, and Opus
|
|
|
|
## Expected Skill Behaviors
|
|
|
|
Research & Documentation evaluations should verify:
|
|
|
|
### Notion Search & Synthesis
|
|
- Searches workspace with relevant queries
|
|
- Fetches multiple source pages (3-5+)
|
|
- Synthesizes information across sources
|
|
- Identifies patterns and insights
|
|
- Handles conflicting information appropriately
|
|
|
|
### Format Selection
|
|
- Chooses correct format based on scope and depth:
|
|
- **Research Summary**: Quick overview with key findings
|
|
- **Comprehensive Report**: Deep analysis with multiple sections
|
|
- **Quick Brief**: Fast facts and takeaways
|
|
- **Comparison**: Side-by-side analysis
|
|
- Applies format structure consistently
|
|
- Uses appropriate sections and headings
|
|
|
|
### Citation & Attribution
|
|
- Includes citations for all Notion sources
|
|
- Uses mention-page tags: `<mention-page url="...">`
|
|
- Attributes findings to specific sources
|
|
- Distinguishes between Notion content and Claude research
|
|
- Links related documents
|
|
|
|
### Document Quality
|
|
- Title clearly indicates research topic and date
|
|
- Executive summary or key findings upfront
|
|
- Organized with clear hierarchy
|
|
- Actionable insights and recommendations
|
|
- Appropriate depth for the query
|
|
|
|
## Creating New Evaluations
|
|
|
|
When adding Research & Documentation evaluations:
|
|
|
|
1. **Test different research types** - Product research, competitive analysis, technical investigation, market research
|
|
2. **Vary source count** - Synthesis of 2-3 pages vs. 10+ pages
|
|
3. **Test format selection** - Does it choose the right format for the scope?
|
|
4. **Include database targets** - Not just standalone pages
|
|
5. **Test citation accuracy** - Are all sources properly attributed?
|
|
6. **Cross-workspace search** - Testing search across teamspaces if applicable
|
|
|
|
## Example Success Criteria
|
|
|
|
**Good** (specific, testable):
|
|
- "Searches Notion for 'roadmap' and 'Q4' and 'product'"
|
|
- "Fetches at least 3 different source pages"
|
|
- "Includes citation for each key finding using mention-page tags"
|
|
- "Creates page with title format 'Research: [Topic] - [Date]'"
|
|
- "Uses Research Summary format with sections: Executive Summary → Key Findings → Details → Recommendations → Sources"
|
|
|
|
**Bad** (vague, untestable):
|
|
- "Searches Notion effectively"
|
|
- "Creates comprehensive research"
|
|
- "Uses sources appropriately"
|
|
- "Good documentation"
|
|
|