directory changes and restructuring
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
# Knowledge Capture Skill Evaluations
|
||||
|
||||
Evaluation scenarios for testing the Knowledge Capture skill across different Claude models.
|
||||
|
||||
## Purpose
|
||||
|
||||
These evaluations ensure the Knowledge Capture skill:
|
||||
- Correctly identifies content types (how-to guides, FAQs, decision records, wikis)
|
||||
- Extracts relevant information from conversations
|
||||
- Structures content appropriately for each type
|
||||
- Searches and places content in the right Notion location
|
||||
- Works consistently across Haiku, Sonnet, and Opus
|
||||
|
||||
## Evaluation Files
|
||||
|
||||
### conversation-to-wiki.json
|
||||
Tests capturing conversation content as a how-to guide for the team wiki.
|
||||
|
||||
**Scenario**: Save deployment discussion to wiki
|
||||
**Key Behaviors**:
|
||||
- Extracts steps, gotchas, and best practices from conversation
|
||||
- Identifies content as How-To Guide
|
||||
- Structures with proper sections (Overview, Prerequisites, Steps, Troubleshooting)
|
||||
- Searches for team wiki location
|
||||
- Preserves technical details (commands, configs)
|
||||
|
||||
### decision-record.json
|
||||
Tests capturing architectural or technical decisions with full context.
|
||||
|
||||
**Scenario**: Document database migration decision
|
||||
**Key Behaviors**:
|
||||
- Extracts decision context, alternatives, and rationale
|
||||
- Follows decision record structure (Context, Decision, Alternatives, Consequences)
|
||||
- Captures both selected and rejected options with reasoning
|
||||
- Places in decision log or ADR database
|
||||
- Links to related technical documentation
|
||||
|
||||
## Running Evaluations
|
||||
|
||||
1. Enable the `knowledge-capture` skill
|
||||
2. Submit the query from the evaluation file
|
||||
3. Provide conversation context as specified
|
||||
4. Verify all expected behaviors are met
|
||||
5. Check success criteria for quality
|
||||
6. Test with Haiku, Sonnet, and Opus
|
||||
|
||||
## Expected Skill Behaviors
|
||||
|
||||
Knowledge Capture evaluations should verify:
|
||||
|
||||
### Content Extraction
|
||||
- Accurately captures key points from conversation context
|
||||
- Preserves specific technical details, not generic placeholders
|
||||
- Maintains context and nuance from discussion
|
||||
|
||||
### Content Type Selection
|
||||
- Correctly identifies appropriate content type (how-to, FAQ, decision record, wiki page)
|
||||
- Uses matching structure from reference documentation
|
||||
- Applies proper Notion markdown formatting
|
||||
|
||||
### Notion Integration
|
||||
- Searches for appropriate target location (wiki, decision log, etc.)
|
||||
- Creates well-structured pages with clear titles
|
||||
- Uses proper parent placement
|
||||
- Includes discoverable titles and metadata
|
||||
|
||||
### Quality Standards
|
||||
- Content is actionable and future-reference ready
|
||||
- Technical accuracy is preserved
|
||||
- Organization aids discoverability
|
||||
- Formatting enhances readability
|
||||
|
||||
## Creating New Evaluations
|
||||
|
||||
When adding Knowledge Capture evaluations:
|
||||
|
||||
1. **Use realistic conversation content** - Include actual technical details, decisions, or processes
|
||||
2. **Test different content types** - How-to guides, FAQs, decision records, meeting notes, learnings
|
||||
3. **Vary complexity** - Simple captures vs. complex technical discussions
|
||||
4. **Test discovery** - Finding the right wiki section or database
|
||||
5. **Include edge cases** - Unclear content types, minimal context, overlapping categories
|
||||
|
||||
## Example Success Criteria
|
||||
|
||||
**Good** (specific, testable):
|
||||
- "Structures content using How-To format with numbered steps"
|
||||
- "Preserves exact bash commands from conversation"
|
||||
- "Creates page with title format 'How to [Action]'"
|
||||
- "Places in Engineering Wiki → Deployment section"
|
||||
|
||||
**Bad** (vague, untestable):
|
||||
- "Creates good documentation"
|
||||
- "Uses appropriate structure"
|
||||
- "Saves to the right place"
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "Save Conversation to Wiki",
|
||||
"skills": ["knowledge-capture"],
|
||||
"query": "Save this conversation about deploying our application to production to the team wiki",
|
||||
"context": "Preceding conversation contains discussion about deployment process, including steps, gotchas, and best practices",
|
||||
"expected_behavior": [
|
||||
"Extracts key information from conversation context (deployment steps, gotchas, best practices)",
|
||||
"Identifies content type as How-To Guide based on procedural nature",
|
||||
"Structures content using How-To structure: Overview → Prerequisites → Steps (numbered) → Verification → Troubleshooting → Related",
|
||||
"Organizes information into clear sections with proper headings",
|
||||
"Includes specific commands, configurations, or examples from conversation",
|
||||
"Adds context about why/when to use this process in Overview section",
|
||||
"Notes common issues and solutions mentioned in discussion in Troubleshooting section",
|
||||
"Uses Notion:notion-search to find team wiki location or asks user",
|
||||
"Creates page using Notion:notion-create-pages with structured content and appropriate parent",
|
||||
"Uses clear, descriptive title like 'How to Deploy to Production'",
|
||||
"Applies Notion markdown formatting (headings, code blocks, bullets)",
|
||||
"Suggests tags/categories for discoverability if wiki database"
|
||||
],
|
||||
"success_criteria": [
|
||||
"Content is structured using How-To format from SKILL.md content types",
|
||||
"Key points from conversation are captured accurately (not generic)",
|
||||
"Information is organized with proper Notion markdown (##, ###, bullets, code blocks)",
|
||||
"Specific technical details (commands, configs) are preserved from conversation",
|
||||
"Document is written for future reference with clear step-by-step instructions",
|
||||
"Title is searchable and descriptive (e.g., 'How to Deploy to Production')",
|
||||
"Page is placed in appropriate wiki location (general wiki or specific section)",
|
||||
"Uses correct tool name (Notion:notion-create-pages)"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"name": "Create Decision Record",
|
||||
"skills": ["knowledge-capture"],
|
||||
"query": "Document our decision to use PostgreSQL instead of MongoDB for our new service",
|
||||
"context": "User has just explained the decision with rationale, options considered, and trade-offs",
|
||||
"expected_behavior": [
|
||||
"Recognizes this as a decision record (architectural decision) from conversation context",
|
||||
"Uses Decision structure: Context → Decision → Rationale → Options Considered (with Pros/Cons) → Consequences → Implementation",
|
||||
"Extracts from context: decision made, options considered (PostgreSQL vs MongoDB), rationale, trade-offs",
|
||||
"Creates document with proper structure including Date, Status (Accepted), and Deciders",
|
||||
"Includes both positive and negative consequences (trade-offs) in Consequences section",
|
||||
"Uses Notion:notion-search to check if decision log database exists",
|
||||
"If database exists, asks whether to add there or create standalone page",
|
||||
"If creating in database, fetches schema using Notion:notion-fetch and sets properties: Decision title, Date, Status, Domain (Architecture), Deciders, Impact",
|
||||
"Uses Notion:notion-create-pages with parent: { data_source_id } for database or { page_id } for parent page",
|
||||
"Applies proper Notion markdown formatting with sections",
|
||||
"Suggests linking from architecture docs or project pages"
|
||||
],
|
||||
"success_criteria": [
|
||||
"Document follows Decision structure from SKILL.md content types",
|
||||
"All key sections present: Context, Decision, Rationale, Options Considered (with Pros/Cons for each), Consequences, Implementation",
|
||||
"Decision is clearly stated (PostgreSQL chosen over MongoDB)",
|
||||
"Options that were considered are documented with pros/cons structure",
|
||||
"Rationale explains why PostgreSQL was chosen based on conversation context",
|
||||
"Consequences include both positive (benefits) and negative (trade-offs)",
|
||||
"If in database, properties are set correctly from schema (Decision, Date, Status: Accepted, Domain: Architecture, Impact)",
|
||||
"Document is dated and has status 'Accepted'",
|
||||
"Uses correct tool names (Notion:notion-search, Notion:notion-fetch, Notion:notion-create-pages)"
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user