chore: sync skills/guideline-generation/references/confidence-scoring.md for v1.2.1
This commit is contained in:
@@ -14,6 +14,8 @@ The guideline section is well-supported and actionable.
|
|||||||
- Specific, actionable instructions (not just vague principles)
|
- Specific, actionable instructions (not just vague principles)
|
||||||
- No unresolved conflicts
|
- No unresolved conflicts
|
||||||
|
|
||||||
|
**Example:** Voice attribute "Confident but not arrogant" appears in the official style guide, is demonstrated in email templates, and matches patterns in top performer calls.
|
||||||
|
|
||||||
### Medium Confidence
|
### Medium Confidence
|
||||||
The section is reasonable but could benefit from more data or team confirmation.
|
The section is reasonable but could benefit from more data or team confirmation.
|
||||||
|
|
||||||
@@ -24,6 +26,8 @@ The section is reasonable but could benefit from more data or team confirmation.
|
|||||||
- Actionable but some interpretation was required
|
- Actionable but some interpretation was required
|
||||||
- May have one unresolved conflict
|
- May have one unresolved conflict
|
||||||
|
|
||||||
|
**Example:** Tone for social media inferred from email templates and one Slack thread, but no official social media guidelines exist.
|
||||||
|
|
||||||
### Low Confidence
|
### Low Confidence
|
||||||
The section is a best-effort recommendation. Team review strongly recommended.
|
The section is a best-effort recommendation. Team review strongly recommended.
|
||||||
|
|
||||||
@@ -34,6 +38,44 @@ The section is a best-effort recommendation. Team review strongly recommended.
|
|||||||
- Unresolved conflicts between sources
|
- Unresolved conflicts between sources
|
||||||
- Limited specificity
|
- Limited specificity
|
||||||
|
|
||||||
|
**Example:** Competitive positioning derived from a single sales call where a competitor was discussed, with no supporting documentation.
|
||||||
|
|
||||||
|
## Section-Level Scoring Guide
|
||||||
|
|
||||||
|
### Voice Attributes
|
||||||
|
- **High**: Attributes appear in official brand guide AND are demonstrated in templates or calls
|
||||||
|
- **Medium**: Attributes appear in one document type only, or are inferred from multiple conversations
|
||||||
|
- **Low**: Attributes inferred from a single source or from indirect evidence
|
||||||
|
|
||||||
|
### Messaging Framework
|
||||||
|
- **High**: Value propositions documented in official materials AND used consistently in sales conversations
|
||||||
|
- **Medium**: Documented but not observed in practice, OR observed but not documented
|
||||||
|
- **Low**: Extracted from a single pitch deck or single call
|
||||||
|
|
||||||
|
### Tone Matrix
|
||||||
|
- **High**: Explicit tone guidance exists for the context AND matches observed behavior
|
||||||
|
- **Medium**: Tone inferred from 3+ examples of content in that context
|
||||||
|
- **Low**: Tone inferred from 1-2 examples, or extrapolated from similar contexts
|
||||||
|
|
||||||
|
### Terminology
|
||||||
|
- **High**: Terms explicitly listed in a style guide or glossary
|
||||||
|
- **Medium**: Terms consistently used in templates and calls (pattern-based)
|
||||||
|
- **Low**: Terms observed in a single document or inferred from brand personality
|
||||||
|
|
||||||
|
### Language Patterns (from transcripts)
|
||||||
|
- **High**: Pattern observed in 5+ calls across multiple speakers
|
||||||
|
- **Medium**: Pattern observed in 3-4 calls or from a single top performer
|
||||||
|
- **Low**: Pattern observed in 1-2 calls only
|
||||||
|
|
||||||
|
### Transcript-Primary Scenarios
|
||||||
|
|
||||||
|
When guidelines are generated primarily from conversational sources (no AUTHORITATIVE documents available):
|
||||||
|
- Voice Attributes derived from 5+ transcripts = **Medium** (not Low)
|
||||||
|
- Messaging Framework from consistent patterns across 5+ calls = **Medium**
|
||||||
|
- Language Patterns weight increases from 10% to 20% in aggregate calculation (subtract 10% from Voice Attributes)
|
||||||
|
|
||||||
|
Note this in the guideline metadata: "Guidelines generated primarily from conversational sources — team review recommended to formalize."
|
||||||
|
|
||||||
## Aggregate Confidence
|
## Aggregate Confidence
|
||||||
|
|
||||||
Calculate overall guideline confidence as the weighted average of section scores:
|
Calculate overall guideline confidence as the weighted average of section scores:
|
||||||
@@ -48,15 +90,39 @@ Calculate overall guideline confidence as the weighted average of section scores
|
|||||||
|
|
||||||
Convert scores: High = 1.0, Medium = 0.6, Low = 0.3
|
Convert scores: High = 1.0, Medium = 0.6, Low = 0.3
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
- Voice Attributes: High (1.0 x 0.30 = 0.30)
|
||||||
|
- Messaging: Medium (0.6 x 0.25 = 0.15)
|
||||||
|
- Tone: Medium (0.6 x 0.20 = 0.12)
|
||||||
|
- Terminology: High (1.0 x 0.15 = 0.15)
|
||||||
|
- Language: Low (0.3 x 0.10 = 0.03)
|
||||||
|
- **Overall: 0.75 = Medium-High confidence**
|
||||||
|
|
||||||
**Aggregate score thresholds:**
|
**Aggregate score thresholds:**
|
||||||
- 0.85–1.0 = High
|
- 0.85–1.0 = High
|
||||||
- 0.60–0.84 = Medium
|
- 0.60–0.84 = Medium
|
||||||
- Below 0.60 = Low
|
- Below 0.60 = Low
|
||||||
|
|
||||||
|
## Presentation
|
||||||
|
|
||||||
|
Present confidence alongside each section header:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## Voice Attributes (Confidence: High)
|
||||||
|
[content]
|
||||||
|
|
||||||
|
## Tone Matrix (Confidence: Medium)
|
||||||
|
[content — note: no official social media guidelines found, tone inferred from email patterns]
|
||||||
|
```
|
||||||
|
|
||||||
|
For Medium and Low confidence sections, include a brief note explaining why confidence is limited and what would raise it.
|
||||||
|
|
||||||
## Relationship to Open Questions
|
## Relationship to Open Questions
|
||||||
|
|
||||||
|
Low confidence sections should generate corresponding open questions:
|
||||||
|
|
||||||
- **Low confidence + conflict** = High Priority open question
|
- **Low confidence + conflict** = High Priority open question
|
||||||
- **Low confidence + gap** = Medium Priority open question
|
- **Low confidence + gap** = Medium Priority open question
|
||||||
- **Medium confidence + minor inconsistency** = Low Priority open question
|
- **Medium confidence + minor inconsistency** = Low Priority open question
|
||||||
|
|
||||||
Every open question includes a recommendation that, if confirmed, would raise the section's confidence score.
|
Every open question includes a recommendation that, if confirmed, would raise the section's confidence score.
|
||||||
|
|||||||
Reference in New Issue
Block a user