Here is a tempting idea that is mostly wrong. You take a large language model, paste in a loan application — the amount, the income, the debt ratio, and the little paragraph where the borrower explains, in their own words, why they need the money — and you ask: “will this person pay it back?” The model answers 0.78, sounds confident, and you feel like you have just replaced a credit bureau with an API call.
I wanted to know how much of that is real. So I ran the experiment properly, on 1,500 real loans with known outcomes, and measured the one thing that actually matters for lending: not whether the answer sounds smart, but whether the number is true. The result was more interesting than either the hype or the backlash. The raw score was wrong — and that turned out to be exactly why it is useful.
The setup, and one honest constraint
The data is a public set of 1,500 loans from Lending Club’s 2007–2011 vintage, each with the fields a lender sees at application time and, crucially, the borrower’s own free-text description of the loan — and each with a resolved outcome: paid in full, or charged off. The base default rate is 15.3%. I held out 450 loans the model never influenced, and measured everything there.2
The honest constraint: I deliberately withheld the bureau’s own answer. Lending Club’s letter grade and interest rate already encode a credit-history-based risk model — the FICO score, the bureau pull, the payment record. Feeding those to the model would be letting it copy the answer off the back of the book. So the model saw only the application: purpose, amount, term, income, debt-to-income, employment, home ownership, and the borrower’s description. Text and soft numbers. Nothing from the bureau.
The model is not a chatbot. It is a “System One” model — a new category built for structured decisions rather than prose. You give it typed questions and a piece of state, and it returns typed answers with probabilities: a yes/no question comes back as a calibrated probability, not a paragraph. It is trained specifically so those probabilities mean something — a claim I was about to test hard.1
The raw score was wrong
I asked the simplest possible question — “based on this application, will the borrower repay in full and on time?” — and compared the model’s probability against what actually happened. Two numbers tell the story. Discrimination, measured by AUC, came in at 0.57, where 0.5 is a coin flip and 1.0 is perfect: real signal, but weak on its own. And calibration was frankly bad: an expected calibration error of 0.25. The model kept saying “about 55–65% likely to repay” for a population that actually repaid 85% of the time. It was systematically, confidently pessimistic.
For lending, a probability that is off by twenty points is not a rounding error — it is the difference between a portfolio that prices risk and one that quietly loses money. If you had trusted the raw number, you would have declined a lot of people who were going to pay you back.
This is the part the hype skips. Out of the box, reading only the application text and soft fields, the model is not a credit bureau and does not pretend to be one. If you paste applications into a general model and trust the number it gives you, this is what you are actually deploying.
But wrong in a fixable way
Here is the turn. The raw scores were mis-levelled, but they were monotone: bin the applications by the model’s score and the observed repayment rate climbs right along with it. The ordering was right; only the scale was off. And a wrong scale with a right order is the easiest problem in applied statistics to fix. One classical recalibration step — a single logistic fit on a holdout, the same trick used to turn any well-ordering score into a probability — collapsed the calibration error from 0.25 to 0.013. Essentially perfect, and it is about ten lines of code that never touch the model.34
This is the whole philosophy of building with a decision model, stated in one experiment: the model gives you a calibrated relative judgment, and your code owns the deterministic mapping into a real probability. You do not argue with the model to get a trustworthy number. You take its ordering and calibrate it — in code, on your own data, where you can test it and version it.
“Won’t a bigger chat model just do this?”
The obvious objection, so I tested it. I ran the same 450 applications through a strong general-purpose chat model — Anthropic’s Claude Haiku — asked the naive way: “return a probability from 0 to 1 that this borrower repays.” It was no better. Its AUC was 0.55, essentially a coin flip at telling a defaulter from a payer, and it did something very characteristic: it piled two-thirds of its answers into the 0.7–0.8 range. Confident-sounding, narrow, and low-resolution.
This is the trap in the tempting idea. A general model’s probability is a shape of language, not a measured frequency; it can be fluent and calibrated-looking and still carry almost no information. The point of a System One model is not that it is magically smarter at reading a loan — on raw discrimination the two were close — it is that its probabilities are built to be recalibrated into true frequencies and its answers come as typed, inspectable values instead of a sentence you have to parse and trust.
What the text knows that the number doesn’t
So the text alone will not out-predict a credit bureau — and it shouldn’t; the bureau is built on hard payment history the application text simply doesn’t contain. The right question is not “can text replace the score?” but “does text add anything the score misses?” To find out, I fit two small models on the training loans: one on the bureau’s grade and rate alone, and one on the bureau plus the model’s reading of the text. Then I scored the held-out set.
The gain is modest — 0.677 to 0.695 — and I am not going to inflate it. On the whole portfolio, the bureau does most of the work. But an average hides the interesting part, because the text’s value is not spread evenly. It is concentrated exactly where a lender is most blind.
The surprise: inside the “safest” band
A credit grade sorts borrowers into bands, and inside a band everyone looks the same to the numbers — same grade, similar rate, statistically interchangeable. That is precisely where a lender has no remaining signal. So I asked: within a single grade, can the borrower’s own words still separate who defaults from who doesn’t?
Inside Grade A — the borrowers the bureau calls safest, where fewer than one in sixteen defaults and the numbers offer a lender nothing to choose between — the model’s reading of the text ranked the eventual defaulters at an AUC of 0.71. Not from the income, not from the ratio; those are held constant within the band. From the words. This is the entire thin-file thesis in one bar: when the structured data runs out of things to say, the unstructured story is where the remaining risk lives — and it is the part every classical scorecard throws away.
Why a decision model, and not a prompt
None of this works if the model is a black box that emits a mood. It works because a System One model is built to be engineered around, and three properties made the difference:
- Typed, decomposable questions. I didn’t ask one vague “is this risky?” — I asked separate, atomic questions: is the income stable, is the stated purpose sound, is there financial distress, is there a credible repayment plan. Each comes back as its own calibrated number, which means every decision decomposes into named reasons — the raw material of an adverse-action notice a regulator will accept, produced by construction rather than reverse-engineered.5
- Calibrated probabilities you can gate on. Because the numbers mean something after recalibration, you can route on them: auto-decide the confident cases, and send only the genuinely uncertain middle to a human underwriter — with the threshold set tighter for a larger loan.
- Many questions, effectively free. All the questions run in one call, in parallel, so asking more of them costs no extra time. Scoring all 1,500 applications — five questions each — took 75 seconds and cost four cents.
That last number is not a flex; it is the point. A feature that costs four cents per fifteen hundred applications and returns calibrated, auditable, per-reason signal is not a research toy. It is a component you can put into a real underwriting pipeline next to the bureau pull.
What this actually means for credit
The headline is not “AI reads applications and gives you a credit score.” It doesn’t, and anyone selling that is selling the 0.25-calibration-error version and hoping you won’t measure it. The real, defensible result is narrower and more useful: a calibrated decision model turns the unstructured half of an application — the half every scorecard discards — into a recalibratable, auditable feature that adds signal on top of the bureau, most where the bureau is blind. Thin files, first-time borrowers, small businesses, anyone whose numbers are thin but whose story is not.
And the way to deploy it is the way this experiment was built: the model reads the text and returns calibrated judgments; your code owns the recalibration, the weights, the thresholds, and the audit trail. The model supplies the signal the numbers can’t; the engineering makes it trustworthy, explainable, and cheap. That division of labour — a model for judgment, code for control — is what we mean, at BIS, by decisions engineered.
Built in the open — with TypeSafe, and an open invitation
One note on what this is, and isn’t. I ran this as an independent experiment on TypeSafe’s System One model, Jev1 — I’m not affiliated with TypeSafe and this wasn’t sponsored; I built it because a decision model trained for calibrated probabilities is a genuinely new tool and worth testing in the open, on real data, with the caveats left in. If you’re on the TypeSafe team, or building on System One yourself, I’d love to compare notes — and if you want a system like this designed, calibrated, and shipped into a real underwriting pipeline, that is exactly the work we do at BIS. The numbers and the method are above; the rest is a conversation.