a probability you can count on

Two thousand refund requests came in this week. How many were abuse? Counting the ones you denied, which is what most dashboards show, misses by about 130 in the simulation at the end of this post. Adding up a model’s honest probabilities misses by about nine, which is as close as perfect knowledge gets, and needs no reviewer to look at a single ticket. Keeping them honest while traffic and model versions move is most of this post; the counting is the payoff.

A model’s probability is only worth thresholding if it’s honest: among the items it calls 0.8, about 80% should turn out true. The first post in this series ran three models over a simulated refund queue (92% of requests legitimate, $5 lost for approving an abusive one, $20 for turning a customer away, so approve when the probability of legitimate is at least 0.2). The models were a Jev-like one that knew the most but had been trained somewhere half the requests were abuse, and a general LLM whose confidence had been cooled the way preference-tuned models often are (a physicist’s intuition is why), read out two different ways. Two repairs came out of it. The base-rate correction fixes a model trained where abuse was common by adding one number to its log-odds, with no labels at all, provided only the share of abuse changed and not what abuse looks like. Platt scaling fits two numbers (a slope and an intercept on the model’s own log-odds) from a few hundred labelled examples per question. This post is how to keep those repairs current, and what they let you do.

---

the loop, in production

The contest in the last post happened once, on a fixed pile of labels. In production the traffic keeps changing, the model gets new versions, and a map fitted in March describes March, so the fitting has to become a loop, and the loop has four parts.

1. the serving path

The serving path is small. The model returns a raw probability, a table you maintain turns it into an honest one, and the cost table turns that into a decision:

// what Jev said: the probability of yes
const raw = answers[question.id].noul;
// fitted on your labels, for this wording on this model version
const key = `${question.id}:${question.definitionHash}:${modelVersion}`;
const { slope, intercept } = calibration[key];
// what it means
const p = sigmoid(slope * logit(raw) + intercept);
// from the cost table
const approve = p >= threshold[question.id];
log({ ticket, key, raw, p, approve });

The calibration table has one row per question, and “question” here means the exact wording on an exact model version, not just its id: each question is its own forecaster with its own bend, rewording one makes it a different forecaster, and a new model version bends differently again. For a yes/no question the row is a slope and an intercept. For a choice question a single temperature is a good place to start, as Guo and colleagues found for image classifiers, but it only fixes one shape of miscalibration, and a choice that’s badly bent in other ways needs a richer map. The thresholds come from each question’s cost table, and they only mean what they say because the line above them made the probability honest.

2. the annotation queue

The annotation queue is where most versions of this go wrong. The natural source of labels is review work people already do: someone looks at a denied refund and either upholds it or reverses it. But that only ever produces labels for the requests you acted on, so you learn how often your denials were right and never how many abusive requests you waved through, and the failure that costs the most (the threshold quietly drifting while abuse slips under it) is the one those labels can’t see.

The fix is to send a small random sample of everything, two or three percent, including requests you approved without a second look, to the same queue, and to record how each item got there, so the random ones can be weighted back up to stand for all the traffic they were drawn from.

Label each item per question rather than per ticket, since “this denial was wrong” is a fact about the abuse question and not about the ticket’s other answers.

3. refitting

On a schedule, or when the alarm below goes off, fit each question’s row again on the labels that have accumulated, hold back the most recent slice to check the new fit against rows it never saw, and publish it only if it does better on those rows than the one it would replace.

4. the drift alarm

The alarm watches for change, and it needs no labels. Keep the distribution of raw probabilities from the period you last fitted on, and compare each day’s against it (the population stability index is the usual measure). A new model version or a shift in what your traffic looks like moves it immediately, long before enough labels arrive to show any damage, and that’s a signal to investigate and pull fresh labels forward.

It’s an alarm, not a calibration test, though: calibration can decay while the histogram of scores stays exactly the same (when what abuse looks like changes but the scores don’t), and the histogram can move while calibration holds. That’s why the random audit from part two keeps running on a schedule regardless, since outcomes are the only thing that catch the failures a score distribution can’t show.

Pin the model version in the meantime, because an alias that moves to a new release moves every probability your table was fitted to.

There’s a Claude Code skill that sets this loop up in your own repository, with tested reference code for every part in TypeScript and SQL. Copy the folder into your repo’s .claude/skills/. The simulations behind all three calibration posts are in the same package.

---

what opens up

The title is a pun that turns out to be literal. If a model’s probabilities are honest, adding them up counts things.

Say you want to know how many of this week’s two thousand refund requests were abuse, without anyone reviewing all two thousand. An honest probability is exactly the share of a request you should expect to be abusive, so a request at 30% contributes 0.3 of an abusive request to the total, a hundred requests at 30% contribute thirty, and summing every request’s probability of abuse estimates the week’s count. The table below runs that for a hundred simulated weeks, each with about 161 abusive requests, and reports how far each method’s count typically lands from the true one (the root-mean-square miss, which is the error a summed expectation is built to keep small):

how you count	typical miss, in requests
sum the true probabilities (the best possible here)	8.7
sum the Jev-like model’s, base-rate corrected, no labels	8.8
sum the Jev-like model’s, Platt on 4,000 labels	10.9
sum the Jev-like model’s, Platt on 400 labels	23.4
sum the cooled LLM’s, unrepaired	65.9
count the requests you denied	130.5
sum the Jev-like model’s, unrepaired	280.7

Even perfect knowledge misses by about nine in this simulation, because which of the uncertain requests actually turn out to be abuse is luck. The base-rate-corrected model matches that without a single label.

Counting the requests you denied, which is what most dashboards show, misses by about 130, and the reason is the threshold. At these prices you only deny a request when you’re at least 80% sure it’s abuse, and only about one abusive request in five looks that clear; the rest look somewhat suspicious (30% here, 60% there) and get approved, correctly, because denying them would turn away too many real customers. So the denial count sees about 31 requests a week out of 161, while the sum gives each of the other 130 its partial share.

A count is also far less forgiving than a decision. The model fitted on 400 labels thinks abuse runs at 7.0% when it really runs at 8.1%, a bias of about one point on every request. That barely changes any decisions, because it only matters for requests sitting right at the threshold, so its decisions cost within 2.5% of the best. But a sum adds that point up across all two thousand requests, and one point on two thousand is about 22 requests a week, most of its miss.

The other thing that opens up is prices that vary. The cost table in these posts had a single price for an approved abuse, five dollars, but real refunds come in every size, and an honest probability lets each request get its own threshold instead of sharing one. Approving a request risks its amount times the chance it’s abuse, and denying it risks the twenty-dollar customer times the chance it was legitimate, so the cutoff for a five-dollar refund is $5/(5+20) = 0.2$, the cutoff for a two-hundred-dollar refund is $200/(200+20) \approx 0.91$, and one model, read honestly, handles both. A probability that only ranks can’t do that; each price band would need its own threshold tuned on its own labels, and you’d never quite trust any of them. One condition comes with it: the probabilities have to be calibrated within each price band, not just overall. If large refunds are abused differently from small ones, a model can be honest on average and still wrong about exactly the requests where the stakes are highest, so check calibration band by band before trusting per-price thresholds.

And each band, like each week’s count, is a random variable rather than a number. Honest probabilities give you its expected value by summing them, $\sum_i p_i$. If the outcomes are also independent given those probabilities, its variance comes almost as cheaply, $\sum_i p_i(1-p_i)$, which for a simulated week of refunds is a standard deviation of about 8.6 requests and is where the best-possible miss of 8.7 in the counting table comes from. Real traffic is rarely that tidy: related customers, coordinated abuse and a bad week for everyone all tie outcomes together and widen the spread, sometimes a lot, so treat that formula as a lower bound on your uncertainty. If that made someone on your team sit up, it’s worth having one of your nerds read Cosma Shalizi’s lecture notes.