Jev vs. Claude Sonnet 5: The model to beat was a formula

I tested Jev and Claude Sonnet 5 on two decisions my product actually makes. On the subject-ranking task, Jev delivered on its pitch: about 1% of Sonnet’s cost and an answer in 0.3 seconds. Yet in both decisions, the production answer was neither model. It was a formula.

On September 15, TypeSafe AI released Jev, the first of what they call System One models. Instead of generating text, it answers closed questions with a probability for every option, at a fraction of the cost and latency of an LLM. The launch benchmarks are TypeSafe’s own, so I wanted numbers from my own product.

I run an exam-prep platform for the UNAM admission exam, the entrance test for Mexico’s largest public university. It already uses Claude Sonnet 5 to write a personalized analysis after each practice exam. I took two decisions the platform actually makes and ran each one through Jev, through Sonnet 5, and through the simplest formulas I could write.

This note covers what I found and what it means for choosing between a decision model, an LLM and plain code. Methodology, full tables and caveats are in the appendix.

The two decisions

1. Which subjects should the student study first? The exam has 120 questions split across 9 or 10 subjects, weighted differently depending on the area the student applies to. After a practice exam, the platform ranks the subjects by study priority. Today that ranking comes from a hard-coded rule on the results screen (lowest percentage first) plus the LLM analysis. I tested it on 30 real graded exams.

• MATHEMATICS26
• SPANISH18
• PHYSICS16
• BIOLOGY10
• CHEMISTRY10
• GEOGRAPHY10
• LITERATURE10
• MEXICAN HISTORY10
• WORLD HISTORY10

120 QUESTIONS TOTAL · EACH AREA WEIGHS THEM DIFFERENTLY · A HUMANITIES APPLICANT SWAPS IN PHILOSOPHY AS A TENTH SUBJECT

2. What will next round’s cutoff score be? For each program, campus and modality, the cutoff is the score of the last admitted applicant, and the platform shows it to students as a reference point. It moves slowly, in small steps. I backtested on 201 combinations of program, campus and modality: every method saw the history through 2024 and predicted the first round of 2025, which none of them had seen.

CORRECT ANSWERS OUT OF 120 · TWO ROUNDS A YEAR, ONE OF THE LONGEST HISTORIES IN THE SET · THE MEDIAN SERIES HAS FOUR POINTS · 2025-1 IS THE ROUND EVERY METHOD HAD TO PREDICT

Results at a glance

Decision 1: the models agreed, and so did a formula

On cost and speed, Jev delivered exactly what TypeSafe promises. Ranking subjects for 1,000 exams costs about $0.10 with Jev and $8.94 with Sonnet, 93 times more. Jev answered in 0.3 seconds; Sonnet took 5.1. And they mostly agreed: both picked the same top subject in 24 of 29 comparable exams.

Then I compared both against a one-line formula: the points left on the table in each subject, (1 − share correct) × number of questions. Both models land close to it. Each picks the same top subject as the formula in 83% of exams, with a rank correlation of 0.92 for Sonnet and 0.87 for Jev on the full ranking.

That’s not a coincidence, and it’s the main lesson. My prompts told both models to weigh how far the student is from mastering each subject and how many questions it carries. Those are exactly the two numbers in the formula. When you can describe a decision in terms of numbers you already have, you’ve already written the formula, and the model is a slower, more expensive way to evaluate it. TypeSafe’s own documentation says the same thing: don’t ask the model anything code can compute exactly.

The finding I didn’t expect was about our current rule. Sorting by lowest percentage ignores how much each subject is worth. Take a student with 1 of 10 in World History and 9 of 26 in Math. The current rule says History first. But there are 9 points left on the table in History and 17 in Math. In 17 of the 30 exams, the two rules put a different subject at the top. The most valuable output of the experiment was a one-line change that costs nothing to run.

Two more details from this decision:

• The shape of the answer. In 1 of 30 exams, Sonnet’s ranking came back in a form my parser couldn’t match to the student’s subjects, so that answer was unusable. Jev can’t fail that way: the options are a closed set and every answer includes a probability for each one. To be fair, a stricter schema on the Sonnet side would likely prevent most of these failures, but it still wouldn’t give you per-option probabilities.
• How you ask matters. Asking Jev a single question with all the subjects as options was 1.8 times cheaper and picked the same top subject as the formula 80% of the time, but its leftover probabilities didn’t make a usable ranking. One narrow question per subject, all in the same call, worked much better. That matches TypeSafe’s guidance to keep each question atomic.

Decision 2: a forecast is not a decision

For the cutoff forecast, the baseline was the simplest possible method: predict the same value as the last round. It missed by 4.0 points on average, out of 120. Methods that fit a trend did worse (5.4 for a linear fit, 6.0 for a damped Holt model): with a median of four data points per series there’s no trend to estimate, only noise to extrapolate. That’s also why neural networks weren’t on the list. Four points per series won’t train one; the size of the history ruled out most candidates before any model was chosen.

Jev missed by 3.9, statistically indistinguishable from repeating the last value. This was expected. Jev doesn’t produce numbers; it chooses among options. I asked it which movement band the cutoff would fall into (down a lot, down a little, flat, up a little, up a lot) and turned its probabilities into a number. TypeSafe’s documentation explicitly lists numeric precision as a weak spot; I ran it anyway to see the failure mode. Its confidence didn’t help either: the third of predictions where Jev was most confident landed in the right band 31% of the time, less often than the middle third (40%).

Sonnet missed by 3.6, and the improvement over repeating the last value held up statistically. But the breakdown tells a different story. For the 161 programs with one admission round per year, Sonnet and the last-value rule were tied (3.58 vs. 3.61). The entire advantage came from 40 open and distance-learning programs, which have two rounds per year. In those, the second round of the year tends to run higher than the first, so repeating the last value copies a high late-year number into the next early-year round and overshoots. Sonnet noticed the pattern and said so in more than half of its one-line explanations for those programs.

FIRST ROUND OF THE YEAR SECOND ROUND

Repeating the last value would have predicted 72. The same round a year earlier said 65. It came in at 66. Sonnet read the pattern out loud on this series, “the -1 rounds are lower, with a slightly upward trend”, and still overshot the correction, landing on 58.

CORRECT ANSWERS OUT OF 120 · 40 OF THE 201 SERIES RUN TWO ROUNDS A YEAR

Once you know the pattern, you don’t need a model to apply it. A seasonal version of the baseline, repeating the value from the same round a year earlier, misses by 3.7 overall and lands within 3 points as often as Sonnet does. The remaining difference between Sonnet and that formula isn’t statistically meaningful. (I added this baseline after the run, prompted by Sonnet’s explanations; the appendix has it with that caveat.)

Here the LLM was genuinely useful, just not as the production predictor: it was the analyst that found the pattern. Running it in production would cost $0.0034 and 1.7 seconds per prediction to match a formula that’s free and instant.

Where each one fits

The rule of thumb I’m taking from this:

1. Can you write the decision as arithmetic on data you already have? Write the formula. Both of my decisions ended here, and an LLM is still a great tool for finding the formula.
2. Is it a judgment over text or messy input, with a closed set of answers, running at volume or where latency matters? That’s where a decision model like Jev earns its place.
3. Does a person read the output? Use an LLM. The post-exam analysis should stay with one; it just shouldn’t be the one deciding the ranking it writes about.

Where Jev’s opportunity is

My two decisions turned out to be arithmetic in disguise, which is the least favorable ground for any model. The decisions Jev is built for look different: which syllabus topic a question belongs to, whether a student’s written answer is correct, whether two questions in the bank are near duplicates. There’s no formula for those, the answers come from closed sets, and they run constantly. That’s the next experiment.

What this says about choosing a model

The market sells this as a ranking. A new model ships, the benchmarks say it beats the last one, and the question becomes which one to buy. In both of my decisions the ranking decided nothing. What decided was the shape of the decision and the data sitting behind it.

Where the history is long enough to carry a pattern, the cheapest method is usually the one to beat: repeating the last value missed by 4 points out of 120, both models landed within half a point of it, and the seasonal version of that same rule erased what was left. Where the job is a closed, repetitive judgment at volume, a decision model is a different economic proposition altogether: about 1% of the LLM’s cost, an answer in under a second, and no way to return something outside the options you defined. Where a person reads the output, only an LLM does the job at all.

None of that makes one of them better. Each is the right answer to a different shape of problem, and plenty of problems need no model at all. The shape is set by your data, not by a launch post, which is why you cannot buy the class from a benchmark: you measure it on your own decision, with your own history, with the boring baseline in the race. Mine cost about a dollar in API calls, and it changed what I am going to ship.

What this doesn’t show

• Small samples, one run. 30 exams and 201 series, run once on September 21, 2026.
• One LLM. I didn’t test GPT-5.6 or other providers, so I have no data on them. Claude Haiku 4.5 costs half as much per token as Sonnet 5, which would narrow the cost gap without closing it; I didn’t measure its quality here.
• No ground truth for decision 1. There’s no single correct study order, so I measured agreement, not accuracy.
• No extended thinking. Sonnet ran with thinking disabled, the same configuration the platform uses in production.

The expensive part of an AI feature often isn’t the model. It’s skipping the step where you check whether you need one.

---

Appendix: how the experiment was run

Setup

• Models. jev-1.13.0 through TypeSafe’s API. claude-sonnet-5 through Anthropic’s API, with a forced tool call and extended thinking disabled.
• Prices. Jev: $0.042 per million input tokens, output free. Sonnet 5: $2 per million input tokens, $10 per million output tokens. Costs are computed from the token counts each API returned.
• Same input for both models. Each case builds one JSON state, sent as-is to Jev and serialized into the user message for Sonnet, byte for byte, so the comparison measures the models and not the prompt wording.
• Isolation and privacy. The scripts run outside the application code. Only the exam area and per-subject results leave the database; no student identifiers.
• Cost of a full run. About $0.95 on Anthropic and about $0.01 on TypeSafe. That is the run reported here: both providers’ dashboards show roughly twice as much for that day, which also covers the pilot runs I threw away.

Decision 1: subject priority

Data. 30 submitted practice exams in the UNAM format (120 questions): 10 in area 1, 12 in area 2, 5 in area 3 and 3 in area 4. Overall scores ranged from 35% to 91%. Each area has 9 subjects (10 in area 4) with fixed question counts that add up to 120.

State. Area, overall result, unanswered questions and, for each subject, correct answers, questions, percentage and weight in the exam. Subjects are listed from lowest to highest percentage; if anything, that order nudges the models toward the percentage rule, not away from it.

Arms.

The priority levels were written as actions (“start here”, “study soon”, “review later”, “don’t spend time on it now”) rather than grades. Both model prompts named the same two factors: distance from mastery and weight in the exam.

Cost and latency (average per exam)

Part of Sonnet’s output is the one-sentence reason per subject that its tool asks for, so a leaner schema would cost less. Even so, Sonnet’s input tokens alone cost 38 times what Jev charges for the whole call.

Agreement. A study order has no ground truth, so the metrics measure agreement: Spearman’s ρ on the full ranking, subjects shared in the top 3, and how often the top subject matches.

Notes:

• One Sonnet response (1 of 30) produced no subject names the parser could match, so that case has no usable ranking and is excluded from Sonnet’s pairs. The raw response wasn’t stored, so I can’t say exactly what went wrong.
• jev-single picks a reasonable top subject, but a single Choice spreads probability to express uncertainty about the winner, not to rank the rest. Reading a full ranking from it is a misuse of the primitive.

Decision 2: cutoff forecast

Data. 201 program × campus × modality series with at least four usable rounds: 161 in-person (one round per year), 21 open and 19 distance (two rounds per year). Training series have 3 to 8 points, median 4. Each method predicts the next usable round, which is the first round of 2025 for almost every series.

Excluded rounds. The rounds where the exam moved online: the first round of 2026 for in-person programs, and the second round of 2025 plus the first of 2026 for open and distance programs. Median scores jumped by double digits in those rounds, so predicting them would measure the change of instrument, not the forecasting method.

Methods.

Accuracy. Errors are in points out of 120. MAE is the average of |prediction − actual| across the 201 series; RMSE averages the squared errors instead, so a few big misses weigh more. The difference against naive and its interval come from a paired bootstrap over series.

A negative bias means under-prediction: 2025 cutoffs came in slightly higher than the history suggested. The bootstrap uses 10,000 paired resamples of series with a fixed seed, so every interval here can be reproduced exactly. Sonnet’s p = 0.027 is one of five planned comparisons against naive; with a Bonferroni correction (α = 0.01) it wouldn’t clear the bar.

By modality (MAE)

TWO ROUNDS / YEAR · n=40 ONE ROUND / YEAR · n=161

The seasonal rule captures most of the pattern for free. What is left of Sonnet’s overall edge is not statistically meaningful.

201 PROGRAM × CAMPUS × MODALITY SERIES · ERRORS IN POINTS OUT OF 120

In two-round programs, the second round of a year averaged 4.1 points above the first (median 2, across 153 same-year pairs). Sonnet referred to specific rounds in its one-line reason in 22 of those 40 series, and in none of the 161 one-round series. Against seasonal-naive, Sonnet’s overall difference is -0.11 [-0.39, 0.15] (p = 0.42) and Jev’s is +0.19 [-0.13, 0.49] (p = 0.24). Within the 40 two-round series, both models beat naive clearly (Sonnet by 1.98 points, Jev by 1.59, both p < 0.01), and neither beats seasonal-naive (Sonnet -0.45 [-1.27, 0.38], p = 0.29).

Direction. Classifying each real change as up, down or flat (within 1 point): flat 71, up 78, down 52. Always guessing “up” is right 39% of the time. Jev got the direction right 40% of the time and Sonnet 48%. In the two-round subset, both reached 60%, and seasonal-naive reached 65%.

Confidence. Jev’s confidence had a median of 0.51 (range 0.11 to 1.0). By thirds:

TypeSafe defines confidence as a summary of how concentrated the probability distribution is, not as the probability of being right, so this isn’t a calibration curve. The point is narrower: on this task, higher confidence didn’t mean better predictions. The full probabilities weren’t stored, so a proper reliability analysis would need a re-run. Sonnet’s coarse self-rating did separate errors (high: 2.38 over 13 series; medium: 3.58 over 177; low: 5.00 over 11), though the high and low groups are small.

Cost.

Reproducibility

Three scripts: one per decision, plus a report generator that recomputes every statistic from the stored results without calling any API. All model calls were real API calls made on September 21, 2026. The seasonal baseline, the modality breakdown and the confidence analysis were added to the report script afterwards and computed from the same stored results, with no new model calls.

Sources: TypeSafe’s launch post, and its documentation on known limitations of jev-1.13 and confidence.