Classification and churn
Some customers quietly stop ordering, and nobody tells you. This lesson shows you how to define churn when there is no subscription to cancel, train a classifier that gives each customer a probability, judge it properly when only 18% of customers churn, and choose who gets a coupon from what each mistake costs.
By the end you can
- Define churn for an app without subscriptions, with an as-of date, a 60-day observation window and a 30-day label window
- Build recency, frequency, monetary and frustration features that were known on the as-of date
- Train logistic regression and gradient boosting classifiers, and check that their probabilities can be trusted
- Read a confusion matrix, precision, recall, ROC AUC and PR AUC when one class is rare
- Pick a threshold from the cost of each mistake, and use a holdout to measure what the coupon really changes
The everyday picture
Think of a baker with about 300 regular customers. She notices when a regular stops coming in. But first she has to decide what "stopped" means: a man who buys bread every morning is gone after two weeks, while a woman who comes once a month for a birthday cake is not. So she only watches people who came in at least twice in the last two months, and she calls someone gone after a month without a visit.
Every week she can phone a few of them and offer a free loaf. Each loaf costs her money, so she phones the people most likely to be gone. She accepts that she will sometimes phone someone who was coming back anyway, and sometimes miss someone who leaves. How many of each mistake she accepts depends on what a loaf costs and what a regular is worth. And she learns one more thing over time: some people come back without a call, and some never come back whatever she offers, because they moved away. Her call only matters for the people in between. A churn model does her job for 700,000 customers.
Defining churn when nobody cancels
In a subscription business, such as a phone plan or a streaming service, churn is an event: the customer cancels. Mangrove's app has no subscription. A customer who stops ordering never tells you, so churn is a definition you choose, and that definition decides what the model learns. You met the same question from the product side in Retention and cohorts.
A good definition has three parts, all measured from one as-of date:
- The population. Customers with 2 or more orders in the 60 days before the as-of date, the observation window. One-time customers are left out; getting them to a second order is a different problem (activation), and they would fill the churn rate with people who were never regulars.
- The label. Churned (1) if the customer places no order in the 30 days after the as-of date, the label window. Otherwise stayed (0).
- The decision. Who gets a win-back coupon on the as-of date. The label window is long enough to see the result and short enough to act on.
About 700,000 customers order in a typical month. On a given as-of date, roughly 450,000 of them have 2 or more orders in the last 60 days, and about 18% of those, roughly 81,000, place no order in the next 30 days (approximate). That 18% is the base rate.
Why 30 days? Look at the gaps between consecutive orders of regular customers. If about 9 in 10 gaps are shorter than 30 days, a 30-day silence is unusual and means something. A 7-day window would label slow but loyal customers as churners. A 90-day window would make you wait three months to learn anything and act too late for the customers you want to keep.
Leakage comes back here, the same way it did in Features and leakage. Every feature must be known on the as-of date. The common leaks in churn work are a complaint logged during the label window, a "coupon redeemed" flag, an app uninstall noticed later, and the quiet one: "days since last order" computed from today's date when you build old training rows. For a snapshot dated 1 May, recency is measured from 1 May. And when you split by time, the label windows of your training snapshots must end before the as-of date of your validation snapshot.
Quick check
A customer ordered 3 times in May and June. The as-of date is 1 July. On 9 July she complains about a late delivery, and her next order is on 20 August. How is she treated in the 1 July snapshot?
The label window is 1 to 30 July, and she placed no order in it, so she is labelled churned; August is outside the window. The complaint happened after the as-of date, so the model would not know it on 1 July. Using it is a leak, even though it would make the model look sharper.