Quality measurement
Call disposition consistency study
A review method for testing whether assistants apply the same outcome codes to comparable calls and notes.
Research question
Do reviewers and assistants select the same disposition when they apply the current definitions to comparable call records? Consistency matters because follow-up queues and reports can be distorted even when every call has a code.
Method
Create a stratified sample covering common, rare, escalated, transferred, and catch-all outcomes. Remove unnecessary personal details. Have at least two trained reviewers independently code each record using the version of the taxonomy active on the call date, then reconcile disagreements.
Measures and interpretation
- Calculate exact agreement and report disagreement pairs that occur most often.
- Compare notes with the minimum evidence required by each definition.
- Separate unclear taxonomy language from training or record-quality problems.
High agreement can still preserve a poorly designed code set. Check whether each code triggers a meaningful next step and whether catch-all categories conceal actionable patterns. Do not rank individuals from small or unbalanced samples.
Operating response
Revise one ambiguous definition with inclusion and exclusion examples, then repeat the blinded review. Version the taxonomy and avoid recoding history unless the reporting policy documents that choice.
Sources
1. NIST Cybersecurity Framework 2.0 2. NIST Privacy Framework 3. SBA guidance for managing a business