A team presents a ranked list of customers likely to contact service again. The model looks promising, but the service leader has a practical concern: the list is longer than the team can review before the next shift. Nobody has decided which existing work should wait.
That fictional situation exposes a gap in many model discussions. A prediction needs a defined operating job. The company must know what a person could do with the result, what information they would need, and what happens when the prediction is wrong.
For a retailer, a repeat-contact prediction might support a review of unclear return information. A reviewer could check whether a useful status clarification is available. But a prediction alone cannot establish whether the parcel arrived, whether a refund review is complete, or whether the customer should receive another message.
The action determines much of the value and risk. Sending a reassuring update without checking the case could create confusion. Reviewing too many weak alerts could delay attention to more urgent problems. Both consequences belong in the decision before a model becomes part of the queue.
Make the comparison useful
A model review should include the current simple rule. If a team already checks returns whose status has remained unchanged for a specified period, that is a meaningful starting point. Compare the approaches at a review volume the team can actually handle.
The example in this week's AI, Not KPI issue shows why this matters. Stillmere Home is a fictional retailer with constructed records, not a client case. A smaller model-generated queue finds more later contacts than a simple rule at the same review count. The example is useful for examining the decision, but it does not establish that any customer has been helped.
That is a separate question. A customer can be correctly identified as likely to call and still receive an ineffective response. The team needs evidence about what the intervention changes before claiming reduced contacts or financial benefit.
Ask for the next piece of evidence
A bounded shadow review can help test whether staff can use the proposed queue. Reviewers record what they would have done while normal service continues. This can expose missing information, unrealistic review time and unclear action boundaries. It cannot measure the effect of a customer intervention that never occurred.
The next decision should reflect that limit. A company may need to revise the action, improve the input data, retain its current rule or request a separately approved controlled trial. Those are useful outcomes when they are supported by evidence and have an accountable owner.
At the next review, ask someone to walk one flagged case from the information available at prediction time to a feasible action. Include the staff time, the cases displaced and the evidence needed to know whether the action helped. That discussion will tell you more about operational readiness than an impressive score on its own.
Read AI, Not KPI on Substack.
