- Give every item an objective criterion: one sentence that says what “yes” means.
- Use “not applicable”, tag where each item applies and keep zero-tolerance items few.
- Let code check timing and phrases; require a checked quote for every model-scored item.
- Never score feelings. Version and approve the scorecard like a policy.
A QA scorecard, also called a quality monitoring form or evaluation form, is the list of things a good call should contain. Most contact centers already have one. It usually lives in a spreadsheet, has 15 to 30 items and was written to be filled in by an experienced QA analyst who knows the business.
That last part is the problem when you move to automated scoring. An analyst knows what “showed ownership” means at your company. A language model does not, and neither does a new team lead or an agent trying to understand a score. The fix is not a different scorecard. It is a more precise one.
Start from outcomes, not habits
Before rewriting items, ask what each one protects. Every item on a scorecard should connect to one of four things: a customer outcome (the issue got solved), a compliance requirement (a disclosure was made), a business outcome (a follow-up was booked) or a known cause of complaints (the customer was put on hold without being told why).
Items that connect to none of these, such as “used the customer's name three times”, tend to produce scores that look precise and change nothing. When you move to scoring every call, a weak item stops being a minor annoyance in a few reviews and becomes noise in every agent's average. Cut it.
Write criteria, not labels
The single most important change is to give every item an objective criterion: a sentence that says what “yes” means. Compare:
| Label only | With an objective criterion |
|---|---|
| Verified the customer | Yes if the agent confirmed at least one of: order number, account email, or the last four digits of the card, before discussing account details. |
| Set expectations | Yes if the agent told the customer what happens next and when, for example “you'll get an email within 24 hours”. |
| Showed empathy | Yes if, after the customer described a problem or frustration, the agent acknowledged it in words before moving to the solution. |
| Professional closing | Yes if, in the last three turns, the agent asked whether there was anything else and closed the call politely. |
A criterion like this does three jobs. It tells the model what to look for. It tells the agent what is expected. And it gives a reviewer something concrete to check when a score is disputed.
Choose the right item type
Not every item is a simple yes or no. Using the right type avoids a lot of unfair scores.
- Yes / No. For things that should happen on every applicable call, such as a greeting.
- Yes / No / Not applicable. For things that only matter sometimes. “Offered a refund option” makes no sense on a call where the customer is placing an order. Without “not applicable”, agents lose points for situations they never faced.
- Scale of 1 to 5. Use sparingly, for items where degree really matters, and describe each level. Scales are where human and machine scores disagree most.
- Zero tolerance. For the few things that make the whole call fail, such as promising a guaranteed outcome you cannot guarantee, or skipping a legally required disclosure. A miss sets the call's score to zero, so keep this list short and unambiguous.
Decide where each item applies
An item that applies to every call is rare. Most apply only to certain call types or topics: a retention offer only on cancellation calls, a delivery-time promise only on order-status calls. Tag each item with the call types or topics it applies to. If the analysis classifies calls by topic anyway, the scorecard can use those topics directly, and agents taking difficult queues are no longer penalized for items that belong to easy ones.
The recording format matters too. Items that depend on who said something, such as “the agent stated the cancellation terms”, should apply only where the speaker is certain. On two-channel recordings the speaker is known for every sentence. On mono recordings it is inferred, and a fair system marks such items as not applicable rather than guessing.
Let code check what code can check
Some items do not need a language model at all. With a timestamped transcript, plain code can check them exactly and the same way every time:
| Check | Example item |
|---|---|
| Greeting within a time | The agent greeted the customer within the first 10 seconds. |
| Required phrase | The agent said the recording notice within the first 30 seconds. |
| Forbidden phrase | The agent never said “guaranteed approval”. |
| Closing phrase | “Anything else I can help with” or similar, in the last three turns. |
| Maximum silence | No silence longer than 45 seconds without the agent explaining a hold. |
Code checks are cheap, fast and impossible to argue with: the phrase is in the transcript at 0:07 or it isn't. Use them wherever the criterion allows, and keep the language model for items that need understanding, such as whether an explanation actually answered the customer's question.
Demand evidence for every model-scored item
For items scored by a language model, require a quote from the call for every answer, and check the quote against the transcript by code. If the quote is not found, the item gets no score, with the reason shown. It is better to leave an item unscored than to score it on a sentence nobody said.
Also do not ask the model how confident it is. Self-reported confidence is not a reliable measure. Decide which answers need a person by rule instead: for example, every “no” on a zero-tolerance item goes to review before it affects the agent's score. We explain the reasoning in Why every AI finding needs a quote and a timestamp.
Weights: keep them simple
Weights let important items count more. Two practical rules: use a small range (for example 1 to 10, with most items between 2 and 5), and make sure the weights reflect the outcome each item protects, not how easy it is to measure. Then look at the result on real calls. If agents with very different calls end up with nearly identical scores, the weights are hiding the differences that matter.
Never score feelings
A scorecard should measure what the agent did, not what the agent felt or seemed to feel. Items such as “sounded enthusiastic”, “seemed stressed” or “positive attitude” are unreliable to measure and unfair to score. They invite disputes and erode trust in the whole program. Rewrite them as behavior (“offered the next step without being asked”) or drop them. In VoiceVera, items that try to measure an agent's emotions are rejected when the scorecard is saved.
An example: e-commerce support
Here is a compact starting scorecard for an online store's support line. Adapt the wording to your policies.
| Item | Type | Applies to | Checked by | Weight |
|---|---|---|---|---|
| Greeting within 10 seconds | Yes/No | All calls | Code | 2 |
| Verified the customer before account details | Yes/No/N.A. | Order, refund, account calls | Model + quote | 5 |
| Identified the reason for the call | Yes/No | All calls | Model + quote | 3 |
| Acknowledged the problem before solving it | Yes/No/N.A. | Complaints, problems | Model + quote | 3 |
| Explained refund or return terms correctly | Yes/No/N.A. | Refund, return calls | Model + quote | 5 |
| Offered an alternative before cancelling | Yes/No/N.A. | Cancellation calls | Model + quote | 4 |
| Set expectations: what happens next and when | Yes/No | All calls with an open issue | Model + quote | 4 |
| No unexplained silence over 45 seconds | Yes/No | All calls | Code | 2 |
| Never promised a delivery date the system did not show | Zero tolerance | Order status calls | Model + quote, always reviewed | — |
| Asked if anything else was needed and closed politely | Yes/No | All calls | Code | 1 |
Download this scorecard as a template. The ten items above with their criteria, types, scope and weights, as a CSV file that opens in Excel or Google Sheets.
Download CSVVersion it, approve it, then measure it
A scorecard is a policy. Treat changes like policy changes: one person drafts, a manager who did not write it approves, and every published version is kept. Every scored call should record which version scored it, so a change in March does not quietly rewrite February's averages.
After the first month, look at three numbers per item: how often it applies, how often it is missed and how often its scores are disputed successfully. Items that never miss can probably go. Items that are often disputed need a sharper criterion. That review, done every quarter, is what keeps a scorecard useful.
Common questions
How many items should a QA scorecard have?
Start with 10 to 15 items that each connect to a customer outcome, a compliance requirement, a business outcome or a known cause of complaints. Items that protect none of these add noise once every call is scored.
Which scorecard items can be checked without AI?
Anything that depends only on the timestamped transcript: a greeting within a set time, a required or forbidden phrase, a closing phrase or a maximum silence. Plain code checks these the same way every time.
Should a scorecard include empathy or tone?
Not as a feeling. Rewrite it as behavior, for example “acknowledged the problem in words before moving to the solution”, and drop items such as “sounded enthusiastic” that try to measure an agent's emotions.
Bring your scorecard. We'll score it.
In the free pilot we turn your current scorecard into objective items and run it on your calls.