diff --git a/skills/guideline-generation/references/confidence-scoring.md b/skills/guideline-generation/references/confidence-scoring.md index d5be01b..1db68f9 100644 --- a/skills/guideline-generation/references/confidence-scoring.md +++ b/skills/guideline-generation/references/confidence-scoring.md @@ -14,6 +14,8 @@ The guideline section is well-supported and actionable. - Specific, actionable instructions (not just vague principles) - No unresolved conflicts +**Example:** Voice attribute "Confident but not arrogant" appears in the official style guide, is demonstrated in email templates, and matches patterns in top performer calls. + ### Medium Confidence The section is reasonable but could benefit from more data or team confirmation. @@ -24,6 +26,8 @@ The section is reasonable but could benefit from more data or team confirmation. - Actionable but some interpretation was required - May have one unresolved conflict +**Example:** Tone for social media inferred from email templates and one Slack thread, but no official social media guidelines exist. + ### Low Confidence The section is a best-effort recommendation. Team review strongly recommended. @@ -34,6 +38,44 @@ The section is a best-effort recommendation. Team review strongly recommended. - Unresolved conflicts between sources - Limited specificity +**Example:** Competitive positioning derived from a single sales call where a competitor was discussed, with no supporting documentation. + +## Section-Level Scoring Guide + +### Voice Attributes +- **High**: Attributes appear in official brand guide AND are demonstrated in templates or calls +- **Medium**: Attributes appear in one document type only, or are inferred from multiple conversations +- **Low**: Attributes inferred from a single source or from indirect evidence + +### Messaging Framework +- **High**: Value propositions documented in official materials AND used consistently in sales conversations +- **Medium**: Documented but not observed in practice, OR observed but not documented +- **Low**: Extracted from a single pitch deck or single call + +### Tone Matrix +- **High**: Explicit tone guidance exists for the context AND matches observed behavior +- **Medium**: Tone inferred from 3+ examples of content in that context +- **Low**: Tone inferred from 1-2 examples, or extrapolated from similar contexts + +### Terminology +- **High**: Terms explicitly listed in a style guide or glossary +- **Medium**: Terms consistently used in templates and calls (pattern-based) +- **Low**: Terms observed in a single document or inferred from brand personality + +### Language Patterns (from transcripts) +- **High**: Pattern observed in 5+ calls across multiple speakers +- **Medium**: Pattern observed in 3-4 calls or from a single top performer +- **Low**: Pattern observed in 1-2 calls only + +### Transcript-Primary Scenarios + +When guidelines are generated primarily from conversational sources (no AUTHORITATIVE documents available): +- Voice Attributes derived from 5+ transcripts = **Medium** (not Low) +- Messaging Framework from consistent patterns across 5+ calls = **Medium** +- Language Patterns weight increases from 10% to 20% in aggregate calculation (subtract 10% from Voice Attributes) + +Note this in the guideline metadata: "Guidelines generated primarily from conversational sources — team review recommended to formalize." + ## Aggregate Confidence Calculate overall guideline confidence as the weighted average of section scores: @@ -48,15 +90,39 @@ Calculate overall guideline confidence as the weighted average of section scores Convert scores: High = 1.0, Medium = 0.6, Low = 0.3 +**Example:** +- Voice Attributes: High (1.0 x 0.30 = 0.30) +- Messaging: Medium (0.6 x 0.25 = 0.15) +- Tone: Medium (0.6 x 0.20 = 0.12) +- Terminology: High (1.0 x 0.15 = 0.15) +- Language: Low (0.3 x 0.10 = 0.03) +- **Overall: 0.75 = Medium-High confidence** + **Aggregate score thresholds:** - 0.85–1.0 = High - 0.60–0.84 = Medium - Below 0.60 = Low +## Presentation + +Present confidence alongside each section header: + +```markdown +## Voice Attributes (Confidence: High) +[content] + +## Tone Matrix (Confidence: Medium) +[content — note: no official social media guidelines found, tone inferred from email patterns] +``` + +For Medium and Low confidence sections, include a brief note explaining why confidence is limited and what would raise it. + ## Relationship to Open Questions +Low confidence sections should generate corresponding open questions: + - **Low confidence + conflict** = High Priority open question - **Low confidence + gap** = Medium Priority open question - **Medium confidence + minor inconsistency** = Low Priority open question -Every open question includes a recommendation that, if confirmed, would raise the section's confidence score. \ No newline at end of file +Every open question includes a recommendation that, if confirmed, would raise the section's confidence score.