Look closely at almost any AI system that runs an operation and you will find two models stacked on top of each other. First a prediction: how much demand next week, how long this route will take, how likely this claim is to be fraud. Then a decision that uses it: how much stock to order, which vehicles to dispatch, whether to auto-approve. Forecast, then act. Predict, then optimize.
This split is so natural that most teams never question it. The data scientists build the best forecast they can. The operations team — or a solver — takes that forecast as truth and optimizes against it. Two clean handoffs, two teams, two metrics. And a quiet, expensive flaw runs right down the seam between them.
The flaw: accuracy and good decisions are not the same goal
A forecasting model is trained to minimize prediction error — mean squared error, MAPE, something that says “be close to the true number, on average.” That objective treats every mistake the same: being 100 units high is exactly as bad as being 100 units low, on a cheap item exactly as bad as on a critical one. But the decision downstream does not see it that way at all.
Consider stocking a shelf. Under-forecast a high-margin product you can restock tomorrow and you lose a little upside. Over-forecast a perishable good with no salvage value and you eat the whole cost. Same size of error, wildly different consequences. The forecast that scores best on average error is not the forecast that leads to the best ordering decision — because the optimizer amplifies some errors and shrugs off others, and the training objective was blind to which was which.
A model can have lower forecast error across the board and still produce worse decisions — because not every error costs the same once a decision is made on top of it.
This is not a rounding-error effect. When decisions are constrained — a fixed fleet, a warehouse that only holds so much, a budget that must balance — the cost of an error can jump sharply right at the boundary the optimizer cares about, and stay flat everywhere else. An accuracy metric spreads the model’s attention evenly across all its predictions. The decision needs it concentrated exactly where a mistake would flip the answer.
The reframe: measure regret, not error
The fix starts with changing the scoreboard. Instead of asking “how far was the forecast from the truth?”, ask “how much worse was the decision we made than the decision we would have made with perfect information?” That gap has a name — decision regret — and it is the number that actually shows up in the P&L.
Once regret is the metric, a different way of building the system follows naturally: train the prediction model so that the decisions it leads to are good, not so that the numbers it emits are close. This is the idea behind a line of research known as decision-focused learning, or “smart predict-then-optimize” (Elmachtoub and Grigas, 2022) — you push the downstream decision cost back through training, so the model learns to be accurate where accuracy changes the decision and is allowed to be sloppy where it does not.
The intuition is almost obvious once stated: a forecast only matters insofar as it changes what you do. Effort spent shaving error off predictions that never move the decision is wasted; effort spent near the decision boundary pays off many times over.
Why it is hard, in one honest paragraph
If this is so sensible, why isn’t it the default? Because the decision layer is usually an optimizer — an argmin — and argmin is a brick wall for gradient-based training: nudge the forecast slightly and the optimal decision either doesn’t move at all or jumps discontinuously, so there is no smooth signal to learn from. The research advances are mostly clever ways around this wall: convex surrogate losses that behave like regret but can be differentiated, and differentiable optimization layers that let a smoothed decision pass gradients back to the model. This is real engineering, not a config flag — which is exactly why it is a competence, not a checkbox.
When it is worth it — and when the simple pipeline is fine
Decision-focused learning earns its keep when three things are true: the downstream costs are asymmetric or the decision is tightly constrained, the prediction is genuinely uncertain, and the decision is repeated often enough that regret compounds. Inventory and assortment, energy commitment, routing under uncertain travel times, credit and fraud thresholds where false positives and false negatives cost very different amounts — these are its home turf.
And when is the plain two-stage pipeline the right call? More often than a purist would admit. If your forecasts are already excellent, or the costs really are symmetric, or the decision is a one-off, the elaborate machinery buys you little. There is a much cheaper 80% of the benefit available first: feed the optimizer honest uncertainty instead of a single point forecast, and make sure the costs in the optimization model reflect the real asymmetry. Many “bad AI decisions” are not a modeling-sophistication problem at all — they are a point estimate being treated as certainty, and a cost function that quietly assumes every error is equal.
Why this is one job, not two
The deeper lesson is organizational. The predict-then-optimize seam is where a lot of enterprise AI silently underperforms: the model team optimizes accuracy, the operations team optimizes the plan, and no one owns the number that matters — the quality of the decision the two produce together. Treating the forecast and the optimizer as one system, designed against decision regret, is precisely the operations-research discipline that generic “add an ML model” projects skip.
That is the line we work on. Whether the answer is full decision-focused training or simply an optimizer fed calibrated uncertainty and honest costs, the question we start from is never “how accurate is the forecast?” It is “how good is the decision?” — because that is the only one your operation actually pays for.