what jev is selling

ChatGPT brought on a wave of GenAI that really was just a fancy chatbot, until structured outputs came along. All these models are black boxes that accept unstructured input and do two things: talk to humans (free text) and talk to machines (structured output). TypeSafe’s Jev cut out the part that talks to humans and improved the use of the part that talks to machines. You give it some state and a set of typed questions, and it gives back a choice, a position on a rubric, or a probability.

the request

A request is a single POST carrying the state and the questions, each under an id you choose:

{
  "state": "Help! My payouts have been failing for 3 days.",
  "model": "jev-latest",
  "questions": {
    "urgent": {
      "type": "noul",
      "instructions": "Does this convey urgency?"
    },
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": "Payments, refunds",
        "technical": "Bugs, outages",
        "sales": null
      }
    }
  }
}

The yes/no question type is called a noul, which is Bernoulli filed down to one syllable (one trial, one probability), and its answer comes back in a field of the same name: answers.urgent.noul is the probability of yes. A choice returns the favorite along with the whole distribution over the options you listed, plus a confidence field that summarizes how concentrated that distribution is; it isn’t the probability that the favorite is right, and it shouldn’t go anywhere near a threshold. (TypeSafe’s API reference has the full shapes.)

All the questions in a request are answered in parallel and in isolation against the same state, so twenty questions cost one round trip and not much more than one question, and no question can see another’s answer, which leaves the combining to you.

The typing is real but narrower than the pitch. The model can’t return an option you didn’t define, because your schema is the whole output space, but it can certainly return the wrong one, and “can’t hallucinate” means “can’t produce something unparseable,” which isn’t the same as being right.

what’s being sold

What TypeSafe is actually selling is the probability. They train with a method they call RLCD (reinforcement learning for calibrated decisions), and the description says the reward is whether the probability attached to an answer was honest rather than whether a rater liked it. My guess is that it’s something like a proper scoring rule in reinforcement learning’s clothes, but the objective isn’t published, so that’s an interpretation and not a description.

It’s also worth knowing what TypeSafe’s launch numbers measure. Their workflow evaluations score each model against the averaged answers of two large frontier models rather than against what actually happened, so they measure agreement with those models, and the launch post reports no calibration metric at all. Neither says whether Jev’s probabilities match outcomes on your traffic; only your own labels can.

The evaluation method is described in TypeSafe’s launch post and picked apart in several independent write-ups since.

We need to talk about probability, and about why “calibrated decisions” sent me down this rabbit hole. The short version is that honesty turned out to be the cheap part, and the rest of this post is why.

---

how sure is sure enough

A model is calibratedAmong all the items it called 80% likely, about 80% come true. A property of a long run of predictions, never of one., or honest in the loose sense these posts use, when its 0.8 really means four times in five. That’s the property RLCD claims to train for, and the reason to want it is what it lets you do with the number.

Say you run refunds, and a model tells you how likely each request is to be legitimate. You can approve or deny, and the request either was legitimate or it wasn’t, so there are four things that can happen, with what you did down the side and what was true across the top, and each of them has a price:

	legitimate	abusive
approve	$0	$5
deny	$20	$0

The two right calls cost nothing. Approving an abusive request loses the five dollars you refunded, and denying a legitimate one loses a customer, which I’ll price at twenty. Now take a request the model puts at 0.2, one chance in five that it’s legitimate. Approve it and four times out of five you’re out five dollars, which averages to four dollars. Deny it and one time out of five you’ve turned away a twenty-dollar customer, which also averages to four dollars. At 0.2 the two choices cost exactly the same, above it approving is cheaper, below it denying is, and so 0.2 is the threshold: the price of one mistake over the price of both, $5/(5+20)$, arrived at without looking at a single labelled example.

Most thresholds are chosen the way 0.5 is chosen (by reflex) or the way 0.7 is chosen (by someone once). To check the arithmetic I built a fake model that is honest by construction: it draws a belief first, then flips a coin weighted by that belief to decide what’s true, so among the items where it believed 0.8 the event happens 80% of the time. Here it is running a refund queue at every threshold from zero to one:

The floor of the bowl sits where the table said it would. Reaching for 0.5 out of habit costs about half again as much, on every ticket, for as long as nobody revisits it.

All of which rests on one assumption, that when the model says 0.2 it means 0.2, and most of what models report as confidence can be trusted to rank things and not much else. Some kinds of tuning cool a model’s confidence, pushing every number toward the corners without changing which item ranks above which. Cool a probability of 0.3 and it can come out near 0.155, on the other side of 0.2. Run a cooled copy of the fake model through the same queue and it costs $1.59 a ticket against the honest one’s $1.44, while knowing exactly as much; its own cheapest threshold is down near 0.04, and nothing in the arithmetic would send you there. Accuracy and AUC don’t notice, because they only look at the ranking. A physicist’s intuition is why models end up like this, and why the word for it is temperature.

---

the contest

A fair comparison needs a queue where the truth is known and several models can look at it, so I built a refund queue in which 92% of requests are legitimate and 8% are abuse, priced with the table above, which puts the threshold at 0.2. Every ticket gets a hidden score that decides whether it’s legitimate, and each model sees that score through a layer of noise, thin for a model that knows a lot and thick for one that knows less. Because the noise is Gaussian, the correct belief for each model has a closed form, so every model starts out honest by construction, and then I bend each one the way real models get bent. It’s worth being plain about what that means: I set how much each model knows. What the contest measures is what that knowledge is worth at these prices, and what one recalibration loop can and can’t give back. Two numbers from weather forecasting score each model: reliability, how far its stated probabilities sit from what actually happens (zero is perfectly honest), and resolution, how much the outcomes differ between the items it scores differently, which is to say how much it can tell apart.

The Jev-like model sees through the thinnest noise, so it knows the most, and it reports honestly, except that it was trained somewhere half the requests were abuse, so it’s honest about a world that isn’t yours. The second is a general LLM read through its log-probabilitiesThe probability the model itself gave each token of its answer, on a log scale. Some APIs return them alongside the text; many don't., which sees through thicker noise and has been cooled to $T = 0.5$, the way preference-tuned models are often measured to be. The third is the same LLM with the same beliefs, asked instead to state its confidence, which it does in round numbers from a short list, so that its answers collapse onto twelve values.

The third is there to test a claim. Not every frontier API exposes log-probabilities, so the honest baseline from a general model is often to ask it how confident it is and take the number it says, and stated confidences pile up on a handful of round values like 0.8, 0.9 and 0.95. Items that share a value can never be told apart again, by any recalibration, so if ties destroy resolution, a model that returns a real probability has an edge that no amount of fitting on your side can give back.

Xiong and colleagues (ICLR 2024) found stated confidences clustering between 80 and 100 percent, mostly in multiples of five.

Before any repair, the Jev-like model, believing half of all customers might be frauds, denies anyone it isn’t sure about. The two LLM readouts sit right at the $0.40 of approving everyone, and a model that somehow saw the hidden score exactly would cost $0.33, so the whole prize at these prices is seven cents a ticket.

The Jev-like model’s trouble is the cheapest miscalibration there is. It isn’t that it was calibrated on your data; it was calibrated on someone else’s, where half the requests were abuse, and the only thing wrong is that its sense of how common abuse is (its prior, or base rate) is off. When only the base rate differs between training and use (more precisely, when the share of abuse changes but abusive and legitimate requests each still look the way they did in training, which statisticians call label shift), the correction is exact and needs no labelled examples at all: add the difference between the two base rates in log-odds, here 2.44, to every logitA probability rewritten as log-odds, log(p/(1-p)). It runs from minus to plus infinity, with 0.5 at zero, which is where a correction like this is just addition.. Call it the base-rate correction. That one number takes it from $1.07 a ticket to $0.354 and its reliability from 0.053 to 0.00001, from the worst model in the contest to about two cents off the oracle, without looking at a single labelled ticket. Here I assumed you know both base rates. If you don’t, estimating the new one from unlabelled traffic is possible but needs its own assumptions and some labelled reference data; Lipton, Wang and Smola is the place to start.

Then the same loop for all three. Take some number of labelled tickets, fit Platt scaling (a logistic regression on the model’s own logit, so just two parameters, a slope and an intercept), and score the result on two hundred thousand tickets the fit never saw, averaging thirty draws of the labelled set at each size:

These are synthetic models, not measurements of Jev or of any LLM. How much each one knows, and each one’s flaws (including the Jev-like model’s wrong base rate, which says nothing about whether the real Jev has one), were assigned by construction.

At four thousand labels the two continuous readouts have both become honest, with reliability of 0.00005 for the Jev-like model and 0.00007 for the LLM’s log-probs, and their costs have spread out in the order of what each model knew: $0.353 and $0.393 a ticket, against resolutions of 0.033 and 0.022. The stated-confidence readout never quite gets there. Two parameters can’t fit twelve steps, so it stalls at a reliability of 0.0013 and at $0.404 never beats approving everyone, and even a step-shaped fit (isotonic regression) brings it only to $0.395, level with the log-probs and no further.

The result I didn’t expect runs the other way, and it isn’t on the chart, which starts after the first fifty labels. Before any fitting at all, the stated confidences were the more honest of the two LLM readouts. Rounding caps them at 0.95 and 0.05, so they can’t be as extreme as the cooled log-probabilities, which run right out to the corners, and a readout that can’t be extreme can’t be extremely overconfident. But the same rounding throws away part of the ranking, and the ranking is the part that’s worth money:

same LLM, read out as	reliability, before fitting	reliability, after 4,000 labels	resolution	cost per ticket
log-probabilities, cooled	0.0021	0.00007	0.022	$0.393
a stated confidence	0.0011	0.0013	0.019	$0.404

The stated confidence started out more honest, which matches what’s been reported for real preference-tuned models, and ended less honest and more expensive, because honesty was the cheap part to fix and the distinctions its rounding threw away can’t be recovered at all.

Tian and colleagues, “Just Ask for Calibration” (EMNLP 2023), found that for preference-tuned models, stated confidences were often better calibrated than the token probabilities.

How much resolution is worth depends on the prices. Take approving everyone as the baseline, since it needs no model at all, and a model that saw the hidden score exactly as the best anyone could do. The gap between them is all the money any model could save you, and the question is how much of that gap each recalibrated model actually closes. Holding a wrongly denied customer at $20 and changing what an approved abuse costs (the column headings), here is what each saves per ticket over approving everyone, with its share of the possible savings in parentheses:

savings per ticket	abuse costs $5	$20	$50
a perfect model	7.1¢	59.9¢	$2.21
Jev-like	4.7¢ (67%)	50.5¢ (84%)	$1.97 (89%)
LLM, log-probs	1.0¢ (14%)	23.7¢ (40%)	$1.24 (56%)
LLM, stated	1.0¢ (14%)	21.9¢ (37%)	$1.24 (56%)

So at five dollars the Jev-like model saves 4.7 of the 7.1 cents a ticket that were there to be saved, which is 67%.

At five dollars abuse is cheap enough that the right move is to approve nearly everyone, so a model earns its keep only on the requests it can be sure are abuse, and telling those few apart from everything else is what the Jev-like model’s extra information buys, which is why the gap between the models is widest there.

So in this setup reliability converged, cost followed resolution, and the Jev-like model’s lead is whatever its extra information buys at your prices. That’s a result about this setup, not a rule for picking models. Resolution is an average over every threshold, and one model can separate cases better overall while another does better right at the boundary your prices care about, so compare candidates at your own threshold, on your own labels. The claim about ties held more weakly than I expected. Rounding cost the LLM about a seventh of its resolution and, with the right fitter, almost nothing at the decision. But my rounding is the kindest version there is (a clean snap of the log-probability to the nearest round number), and real stated confidences also wobble from one asking to the next, which I didn’t model, so the penalty here is a floor rather than an estimate.

---

what honesty is worth

The contest puts a price on the pitch. Recalibrating a model means learning, from some of your own labelled examples, a map from what the model says to how often it turns out right. A map that keeps distinct scores distinct (and the usual ones are monotone: they stretch the numbers but never swap two of them) can’t change which items the model tells apart, which is what resolution measures; only merging scores together, by rounding or by a fitter that creates ties, throws that information away. So recalibration doesn’t make models equally good. It makes them comparable, by stripping off the part that’s cheap to repair and leaving the part that isn’t. The same goes for a frontier model read through its log-probabilities, which recalibrates just as well.

What a model sells you, underneath, is what it can tell apart. What a model that arrives already calibrated adds is two things: it needs fewer labels (sometimes none) to become honest on your traffic, where a badly bent one needs a more flexible fit and thousands of labels, and it makes sensible decisions from the first day, before any of those labels exist, which can be worth real money on its own. Against a general model that only states its confidence, it may also keep distinctions that rounding throws away, though in my setup that edge was real and small.

The way I’d put the whole bargain: calibrated probabilities let software connect predictions to costs and to expected counts. Jev promises to make those probabilities cheaper and easier to get, but what they’re worth still depends on what the model can tell apart and on whether its calibration holds on your traffic.

---

where this goes

Jev’s first contender posted on X two hours after the announcement. Open-source versions are multiplying. The main providers will have their own soon. Build the recalibration loop now because it works on whichever model wins.

Everything here happened once, on a fixed pile of labels. The next post is what it takes to keep a model honest when traffic changes and model versions move, and what an honest probability lets you do besides decide, starting with counting. A physicist’s intuition is for anyone who wants the why: what kind of number a probability is, the Boltzmann distribution hiding inside softmax, and how preference tuning can cool a model without changing what it knows.