A new class of tool has quietly become useful: you hand it a spreadsheet, ask a question in plain English — “which region is actually driving revenue?” — and it answers with a chart, a table, and a paragraph of interpretation, right there in the chat. What makes it more than a party trick is a single, unglamorous rule: it does not guess the numbers. It writes real Python, runs it over your real data, and reports only what actually came back.
That rule sounds small. It is the whole architecture. Once you insist that every figure be computed rather than predicted, the interesting engineering stops being the language model and starts being everything around it: how you run untrusted code without getting owned, how you keep the model from ever inventing a value, how you stream results back, and how you let something other than a human — an agent — drive the whole loop. I recently built a deep, production-grade reference implementation of exactly this to work through those problems end to end1. This is the dissection.
The hard part is not the chat
It is tempting to think the difficulty is the conversation — parsing the question, phrasing the answer. It is not. The dangerous, load-bearing problem is this: a language model writes a program, and you have to execute that program over a user’s data. The code is untrusted by definition. Nobody reviewed it. It might loop forever, try to reach the network, read files it should not, or simply crash. And you have to run it anyway, fast, and turn the result into an answer — all without ever letting a fabricated number slip through.
The model’s output is not text to display. It is code you are about to run on your own infrastructure. Treat it as exactly that dangerous, and the rest of the design follows.
Grounding: the model never guesses a number
The grounding rule has to be enforced by the control flow, not by asking the model nicely. The loop is: turn the question into Python, execute it in the sandbox, and write the interpretation only from what actually printed to standard output. A run that never succeeds is never interpreted — there is no code path on which the model gets to narrate a result it did not compute. That closes the door through which invented numbers would otherwise walk.
Errors are treated as data, not dead ends. When the generated code throws — a misspelled column, a dtype mismatch, the ordinary failures of code written against a schema it has only seen described — the traceback is fed straight back to the model with a request to repair its own code, and the fixed version is retried, up to a small fixed budget. Most first-try failures are trivial and vanish the moment the model can see its own stack trace. A bounded repair loop turns the majority of them into a correct answer a half-second later, instead of a 500.
The sandbox: isolation at the boundary, not in Python
This is where an AI data analyst is won or lost. The reference build runs each analysis in one ephemeral container — a fresh “docker run --rm” from a prebuilt image, spawned per request and torn down the instant it finishes. Crucially, the isolation is enforced at the container boundary, not by trusting anything inside Python. The controls are boring on purpose:
- No network at all (--network none) — the single most important flag. Untrusted code cannot exfiltrate data, call home, or reach internal services.2
- A read-only filesystem with a small writable scratch space, dropped Linux capabilities, no-new-privileges, and an unprivileged user — so a hostile script has nothing to escalate into.
- Hard memory, CPU, and process-count caps to stop fork bombs and runaway loops, and a wall-clock timeout that kills the container outright.
- The dataset mounted read-only, so an analysis can read your data but can never mutate the source.
Inside, a small trusted harness — not the model’s code — loads the dataframe, runs the generated code with output captured, and emits exactly one structured envelope: did it succeed, what printed, and any artifacts (the chart as an image, the result as a table). Because the harness owns the output framing, the untrusted code physically cannot corrupt the result it hands back; a crash becomes a captured traceback rather than a thrown exception, which is precisely what the repair loop needs.
Being honest about the threat model is part of the engineering. Spawning containers means something holds the Docker socket, and that socket is powerful — so in a real deployment you move execution to a managed, remote sandbox with no local daemon exposure3, and the genuinely optimal next step is stronger per-run isolation still: a user-space kernel like gVisor, or a Firecracker microVM, which defend against the container escapes that capability-dropping alone does not45. The point of a clean sandbox seam is that this is a runtime swap, not a rewrite.
One core, many clients
The most consequential architectural decision has nothing to do with AI. There is exactly one implementation of the analyst pipeline — upload, profile, generate, execute, interpret — exposed as a single callable core, with its dependencies (the model provider, the sandbox, storage, the database) injected rather than hard-wired. Every surface is a thin client of that core: the web UI, a command-line driver, and — the interesting one — an agent-facing server all call into the same engine.
This is the difference between a demo and a system. The moment you add a second way to drive the product, a coupled codebase forces you to duplicate the logic, and the two paths drift until they disagree. Keeping the core free of any one client from day one means the agent surface and the human UI provably produce the same answer, because they are the same code. When people say “staff-level,” this seam — not the framework, not the model — is usually what they mean.
A platform an agent can drive
Because the core is client-agnostic, exposing it to autonomous agents is almost free — and it is a real capability most tools in this category lack. They tend to be consumers of external connectors, with no public interface an agent could call to drive them. The reference build ships the opposite: an MCP server that exposes the whole capability surface — upload a dataset, ask a question, pull back the chart, the table, and the generated code — each tool a thin adapter over the same core the UI uses6.
The detail that matters is exposing the code, not just the answer. An analyst an agent can query for the exact Python behind a result is auditable and composable; an analyst that returns only a conclusion is a black box the agent has to trust. Handing back the code is what lets one automated step check, reuse, or build on another — the whole point of making a platform agent-drivable rather than merely agent-adjacent.
Bring your own model — even an agent as the model
Nothing in the pipeline hard-codes a model. The provider is configuration, resolved in layers — a deployment default plus an optional per-request override — so the same analyst can point at a fully local model for privacy, a fast hosted endpoint for throughput, or something stranger. The strange one is worth naming: because the model is just an injected dependency, the model can be an agent host itself. Each generate/repair/interpret request is placed on a queue and answered by a connected agent over the same protocol that drives the platform — so an agent can not only pilot the analyst, it can be its brain. None of that touches the core; it is one more provider behind the same seam.
Streaming, and why the transport stays dumb
A good analyst feels alive: you watch it move through phases — generating, executing, repairing if it must, interpreting — and then the written answer types itself out as the model produces it, with the chart and table landing as the code finishes. That is server-sent events carrying phase events and, where the provider supports it, token-by-token text. The core emits these through a simple hook so the transport carries dumb events and never contains logic; providers that cannot stream degrade gracefully to a single final payload through the exact same path.
Measured, not asserted
For a tool whose entire promise is “the numbers are real,” “it works” is a claim you have to measure, not assert. So the reference build scores two things over a golden set on every change. Execution-based correctness: do the expected values appear in the actually-executed output — proof the generated code computed the right thing, not that the prose sounded right. And grounding faithfulness: is every substantive figure in the written interpretation genuinely present in the executed output? That second metric turns “never fabricate” from a property of the code path into a property you can put a number on and regression-test in continuous integration, for free, with no paid model calls in the gate.
What this is really about
Strip away the category and a pattern remains that applies far beyond data analysis. The interesting engineering in “AI that does X” is rarely the model. It is the boundary where the model’s output stops being text and becomes an action in the real world — code that runs, a number that lands in a decision, a tool an agent invokes. Making that boundary safe, grounding it in something true, measuring it honestly, and exposing it so other systems can build on it: that is the work. The model is a component; the system is the product.
The full reference implementation — the sandbox, the core seam, the agent server, the eval harness — is open source, and there is a static walkthrough of the loop on this site if you would rather see it than read about it1. It runs on a toy dataset; a real one would run on your data, in your environment, against the questions your team actually asks — with the same rule underneath: no number reaches a decision unless code produced it. That is what we mean, at BIS, by decisions engineered.