Four team leads sit around a screen and listen to the same seven-minute call. Then each writes down a score for empathy, on a scale of 1 to 5. The scores are 2, 3, 3 and 5.
Someone laughs. The lead who gave a 5 explains that the customer calmed down by the end, so something must have worked. The lead who gave a 2 points out that the agent talked over the customer twice in the first minute. After ten minutes the room settles on 3, everyone feels the session went well, and they move on to the next call.
Nobody writes down the number that mattered most: on this criterion, four experienced people were three points apart.
AI call scoring can only be as accurate as the human agreement you measure it against, because for most criteria there is no other answer key. So measure your reviewers first: have two of them score the same calls without seeing each other’s scores, and compare them criterion by criterion. Then measure the AI in exactly the same way. Where the AI agrees with your reviewers as often as they agree with each other, you can trust it. Where your reviewers don’t agree, fix the criterion first.
Why calibration sessions hide the problem
The session in the opening is how most contact centres keep their reviewers aligned, and it does useful work: people hear how colleagues read the scorecard and argue their way to a shared view. As a measurement, though, it has two flaws.
The scores aren’t independent. Once someone senior says a call is a 3, the others tend to move towards 3. The session ends in agreement, but the room produced that agreement. It won’t be there on Tuesday afternoon, when each lead scores calls alone.
And nothing gets recorded. The session produces one agreed score for one call. It doesn’t record how far apart people were before they talked. After a year of sessions, you still won’t know which criteria your team reads the same way and which it doesn’t.
So separate the two jobs. Score first, alone and without seeing anyone else’s score. Discuss afterwards. Keep the numbers.
94% agreement can mean very little
Once two reviewers have scored the same calls blind, the obvious measure is how often they gave the same answer. For the pass/fail checks most scorecards rely on, that measure is often misleading.
Take a check that almost every call passes, such as “confirmed that the customer understood the next step”. Two reviewers each score the same 100 calls. Each marks 5 calls as missed, but only 2 of those are the same calls. They agree on the other 92, which both marked as passed. That is 94 matches out of 100: 94% agreement, which looks excellent.
Now look at the calls that matter. Between them, the reviewers marked 8 different calls as missed, and they agreed on only 2. If either of them had done the job alone, you would have found a different set of failures depending on who scored that day. The high percentage comes almost entirely from the clear passes, where nobody was ever going to disagree.
Statisticians correct for this with a measure called Cohen’s kappa, which asks how much better two reviewers agree than they would by chance. In the example, kappa is about 0.37. On the scale most researchers use, from Landis and Koch, that is well below the 0.61 they call “substantial” agreement. For 1–5 scales like empathy, a weighted version of kappa counts a one-point difference as a near miss rather than a full disagreement. You don’t need to calculate any of this by hand, but on a check that almost always passes, don’t stop at the raw percentage.
Two more things keep the numbers useful:
- Measure each criterion separately. An overall agreement figure averages your clearest criteria with your vaguest ones, and the vague ones are the ones you need to find.
- Notice what kind of criterion it is. If reviewers disagree on a factual check, such as whether a required disclosure was read, someone missed something. That is a problem with attention, not with wording. If they disagree on a judgement, such as whether the customer understood, the wording is usually the cause.
How to judge the AI’s accuracy
With a human baseline for each criterion, judging the AI is straightforward. Treat it as one more reviewer: compare its scores with your reviewers’ blind scores, criterion by criterion, using the same measures. Criteria tend to fall into three groups.
The AI agrees with your reviewers about as well as they agree with each other. Here the AI does the work of an extra reviewer, on every call instead of a sample. This is where automated scoring is worth having.
Your reviewers agree with each other, but not with the AI. Here the AI has a real problem. Read the reasons it gives for its scores. If it is applying the criterion differently from your team, the wording probably leaves out something your reviewers all take for granted. Write that down in the criterion, then measure again.
Your reviewers don’t agree with each other. Here you can’t judge the AI at all, and it isn’t the first thing to fix. A criterion like “showed empathy” covers three or four different behaviours depending on who is listening. Split it into things a reviewer can hear, such as “acknowledged the problem before offering a fix”, and agreement usually improves for people and AI together. We’ve written about how to word criteria so they can be scored consistently.
How often to recalibrate
Agreement isn’t something you establish once. A new product changes what a good answer sounds like. New team leads bring their own habits. Someone edits the scorecard. The AI model is updated. Any of these can move scores that looked stable last quarter, so the check has to become routine.
A weekly sample of blind reviews keeps it going. Use different calls for different purposes:
- Random calls measure agreement. Report agreement on these, because they show what normal calls look like.
- Disputed calls find problems. An agent who disputes a score is telling you the scorecard and the conversation didn’t fit. (It helps to have a dispute process your agents trust.)
- Calls where the AI was unsure find problems too. Disagreement is most likely there, so a reviewer’s time teaches you the most.
Keep the disputed and uncertain calls out of the headline agreement figure. They are chosen because they are hard, so including them makes agreement look worse than it is.
This is how blind reviews and AI agreement work in Future Ready: managers score a few calls each week without seeing the AI’s score, and every skill and check shows how closely the AI and your people agree, from Strong to Needs calibration. The method doesn’t depend on any tool, though. A spreadsheet and two careful reviewers are enough to start.
A routine you can start this month
- Pick 50 to 100 recent calls that cover your main call types. For checks that rarely fail, add calls you already know failed, or you won’t have enough failures to compare.
- Have two reviewers score them blind, on your current scorecard, without discussing them.
- Compare their scores criterion by criterion. For pass/fail checks, look at the failed calls, not only the overall match rate.
- Rewrite the three criteria with the worst agreement, then test the new wording on a fresh set of calls. Reviewers remember calls they have already scored, so reusing them would not be blind.
- Compare the AI with your reviewers on the same calls and the same criteria.
- Make it weekly, with a handful of random calls for measurement and the disputed and uncertain calls for finding problems.
After the first round you’ll know which criteria mean the same thing to everyone, and which ones never did. That is worth knowing whether or not you ever automate a single score.
The question to ask any vendor
If you’re evaluating AI scoring, including ours, don’t ask for a single accuracy figure. Ask for agreement on each criterion, measured blind against your own reviewers on your own calls, next to how well your reviewers agree with each other. A vendor that can show you that has measured its accuracy. One that can’t, hasn’t.
We can show you how well the AI and your reviewers agree. We’ll score a sample of your calls on your scorecard and compare the AI with your team, criterion by criterion, so you can see which criteria are ready to automate and which need rewriting first.
