Calibrating a call quality scorecard

*October 22, 2026*

A scorecard is only useful when two reviewers score the same call the same way. Without calibration, the score reflects the reviewer more than the call, and coaching becomes a debate about taste.

Choose observable criteria

Score what can be heard or read: the greeting, the verification step, the caller's stated need, the accuracy of the information, the next action, and the closing.

Avoid criteria like "sounded friendly" unless the rubric defines the behaviors that count.

Use a three-level scale

Define meets, needs improvement, and critical failure for each criterion. A critical failure is reserved for safety, privacy, or a clearly wrong commitment.

Three levels are easier to calibrate than five and produce more consistent results.

Score independently, then compare

Two reviewers score the same sample without discussing it. Compare the scores and discuss the calls with the largest gaps.

Do not average disagreements away. The gap is the finding, and it usually points to a rubric that needs a clearer example.

Anchor the rubric with examples

Keep two or three sample calls per criterion: one that meets, one that needs improvement, and one critical failure. Review the examples when a new reviewer joins.

Separate coaching from scoring

The reviewer who scores a call may deliver the coaching, but the script and the rubric should be shared before the score. Surprise criteria erode trust.

Review the scorecard quarterly

Remove criteria that nobody uses, add ones tied to a current risk, and recheck the examples. A scorecard that never changes becomes a ritual, not a quality tool.