A retention team calls 1,000 customers, and hundreds renew. The reporting calls them saved, assigns their revenue to the program, and requests a bigger budget. The missing comparison is how many would have renewed without the call.
That gap becomes more expensive when an AI model makes the account list look authoritative. A model may rank the likelihood of cancellation well. It still does not establish that a particular call, offer, or discount will change the customer's decision.
Consider two software customers. One is preparing to close its business. The other is struggling to complete a setup task that would make the subscription useful. The first may have the higher churn score, while the second has the problem a specialist can actually help resolve. Sending the same retention offer to both makes the model responsible for a decision it was not designed to answer.
Define the help before evaluating the campaign
I would start a retention review by asking the team to describe what the customer receives. A setup review, a correction to a billing problem, and a discount are different actions with different costs and possible consequences. They should not disappear into one category called intervention.
Normal obligations also need to remain clear. A test must not become an excuse to withhold service customers are entitled to receive or make cancellation harder. Customer eligibility, contact restrictions, existing complaints, and duplicate outreach all affect whether an additional action is appropriate.
The next question is whether the action changes renewal behavior. A planned randomized comparison can help answer that in the tested population. Looking only at people who accepted an offer is less informative because those customers may already be more likely to stay.
More retained accounts does not automatically mean more value
The Week 27 AI, Not KPI working example uses a fictional software company, Northline Cloud. It compares an optional specialist setup review with normal service. All numbers are teaching data; no customer experiment occurred.
The middle-risk group has 780 of 1,000 offered accounts retained at day 90, versus 700 of 1,000 comparison accounts. The eight-point difference is the estimate worth examining. Counting all 780 as incremental saves would give the program credit for customers who may have stayed anyway.
Even that difference needs careful interpretation. The observation window must be complete, assignment and outcome records must be reliable, and the estimate has uncertainty. Costs matter too. Specialist time and outreach costs apply to the program, including customers who do not renew or do not accept the offer. A revenue figure that ignores those costs will not tell finance whether to expand the work.
A question worth bringing to the budget meeting
Ask the team to take one “customer saved” claim and show the comparison that supports the word saved. If it can only show that the customer renewed after contact, describe the result that way and identify the evidence still missing.
This is useful scrutiny whether a company uses an advanced model or a spreadsheet. Better targeting should improve the work the company chooses to do, not simply make its existing save report look more scientific.
Week 27 of AI, Not KPI includes the Retention Propensity and Test Workbook, a completed fictional example, and executive review guidance. The paid materials help a team prepare a bounded test and make its assumptions visible before committing more time or money. Read AI, Not KPI and subscribe for the Week 27 working kit.
Further reading: EconML's method motivation and Microsoft Research on diagnosing sample ratio mismatch in A/B testing.
