Ask any language model whether a comment is toxic and it will happily answer. That is not the useful question. The useful question — the one that decides whether you can automate a single decision — is whether the number attached to that answer means anything. If the model says a comment is 80% likely to be toxic, is it toxic about 80% of the time? A model that is fluent and confident and wrong about its own confidence can't be gated on; a model whose probabilities track reality can. That property is calibration, and it is the whole game in moderation.

So we tested it, on data with a real answer key: 1,500 comments from a public dataset where each one carries a human toxicity score — the fraction of annotators who judged it toxic. We asked a calibrated decision model (TypeSafe's Jev)1 a small battery of typed questions about each comment — is it toxic, is it a threat, does it attack someone's identity, how severe — and compared its numbers to the humans'.

It reads the text well

First, can it tell toxic from not at all? Yes, and clearly. Its toxicity score ranks a toxic comment above a clean one at an AUC of 0.865, and for identity attacks — a category where the signal is unusually explicit in the words — 0.95. This is a sharp contrast with a credit-risk experiment we ran earlier, where reading the application text scored a weak 0.57 because the real signal lived in a credit bureau the text never mentions. Here the signal is in the text, so the model is genuinely strong. The value of a text-reading model scales with how much of the judgment actually lives in the words.

But confident is not calibrated

Being a strong classifier is not the same as being trustworthy about your own certainty. Out of the box the model over-predicts: it is more sure a comment is toxic than the panel of humans is. The reliability curve sits below the diagonal — when it says 0.9, the human toxicity fraction is closer to 0.5.

000.50.511perfectJev predicted P(toxic)human toxicity fractionmonotone but over-confident (ECE 0.28) → recalibrates to 0.013
Predicted toxicity vs the human label. The curve is monotone — higher scores really do mean more toxic — but it sits below the diagonal: the raw score is over-confident (ECE 0.28). Because the ordering is right, one recalibration step maps it onto the line (ECE 0.013).

This is the same lesson as every other calibrated-model experiment we've run: the raw score is mis-levelled, but it is monotone, so the fix is a one-parameter recalibration in code that maps the model's ordering onto true frequencies — here, expected calibration error from 0.28 down to 0.013. You don't argue with the model to get a trustworthy number; you take its ranking and calibrate it against your own data, where you can test it. (Note the direction is the opposite of a credit-risk model we tested, which under-predicted — the fix is the same either way.)

How much can you automate?

A calibrated score turns into an operating decision: auto-allow the confidently-clean, auto-remove the confidently-toxic, and send only the uncertain middle to a human. The only number that matters for that is how much you can take off the human's desk without letting harm through.

0%15%30%45%292<.10 / >.80342<.15 / >.85382<.20 / >.90auto-handled %toxic that slips through %auto-allow / auto-remove thresholds
Auto-handling the confident ends of the queue. At a 0.15/0.85 gate the model resolves about a third of the queue on its own while roughly 2% of genuinely toxic comments slip past — a dial you set by how much risk each side of the mistake carries.

At a middle setting the gate auto-handles around a third of the queue — the obvious-clean and the obvious-toxic — and passes only about 2% of the truly toxic comments, leaving the human moderator to spend their attention on the genuinely ambiguous cases instead of the easy ones. Push the thresholds and you trade a little more automation for a little more slippage; that is a policy dial, and because the score is calibrated, the dial means what it says. (In practice you gate on the recalibrated score, so a 0.85 threshold really is 85% likely toxic.)

The point

Moderation is the clean case of a pattern that shows up wherever a queue is too big for the people on it: the decision that matters is a judgment about text, and the thing that makes it automatable is not a smarter-sounding model but an honest one — a model whose confidence you can put a threshold on. A calibrated decision model gives you that, cheaply, as a typed number your code can route on. The model reads; your code decides; and because the number is calibrated, you know exactly how much you're trusting it. That is what we mean, at BIS, by decisions engineered.

Built in the open — with TypeSafe

An independent experiment on TypeSafe's System One model, Jev1 — not sponsored, not affiliated. The data is public and human-labelled; the urgency here is the model's calibrated reading of the text, measured against the human panel. Caveat: one dataset, English comments, and toxicity is itself a contested label — treat the numbers as a controlled measurement, not a universal benchmark.