The Guardrails Are the Product. Running Dex against a live warehouse, with an LLM on the other end
I did not adopt Dex for its features. I adopted it because it refuses things. What follows is what that refusal looked like in production, including where it did not hold.
Data Engineer, Principal Architect & Independent Researcher at CatInCloud Labs
I run a production analytics stack myself, and pay for it myself: one VPS, encrypted at rest, orchestrated with Dagster, transformed with Polars, landed in BigQuery. It exists so claims about data tooling can be tested against a real warehouse with a real bill attached, rather than asserted. Silent reliability is the whole point.
Then I put an interactive Slack agent in front of the warehouse, and the calculus changed. Someone types a question in a channel, a language model decides what to ask, and the warehouse bills for the answer. On one side of that pipe is a probabilistic system that will confidently propose a query scanning every byte you own; on the other, a metered database that charges for every byte it touches. Everything interesting lives in the layer between them.
That layer is exmergo-dex-core, wrapped in an internal HTTP service that is backend-only and never exposed to the internet. Scheduled jobs and the agent both drive it over HTTP, and it opens exactly three doors: read-only exploration, drift detection against a pinned baseline, and a gated plan-then-apply path into an isolated dev dataset.
I did not adopt Dex for its features. I adopted it because it refuses things. What follows is what that refusal looked like in production, including where it did not hold.
Cost: the number you cap is not the number you were shown
BigQuery charges for bytes scanned, so a cost guard is the first thing that has to work. Dex surfaces an estimate before spending and enforces a per-request budget. I set a 1 GB ceiling on any single call and a 5 GB cumulative cap per UTC day, and considered the problem handled.
It was not handled, for two reasons that took live traffic to find.
The first is that an estimate and a charge are different numbers, and a guardrail that sums the wrong one is decorative. My events carried both: a forecast field preferring the preflight estimate, and a settlement field reading only actual spend. A dry run that bills nothing still reports a large forecast, which is right for a “what might this cost” dashboard and wrong for a cap. The spend guard now sums settled bytes only, and emits that field on every call, including zero for free routes, refusals, and crashes. That last part matters more than it sounds: my log query silently drops lines missing the field, so an omitted zero would have quietly undercounted the cap rather than failing loudly. A guardrail that fails permissive is worse than none, because it also hands you a number you trust.
The second was worse. Dex resolves configuration per repository root, with no inheritance. I run two roots (one for exploration and drift, one isolated for model building) and the second never saw the first one’s budget block. The per-command budget still applied, because my service passes it explicitly on every billed call. The cumulative cap did not exist there at all. The ledger held seven builds at exactly 100 MB each, roughly 700 MB total, recorded faithfully and settled against nothing.
Nothing broke. That is the point. Seven legal builds ran against a daily cap structurally incapable of binding, and I found it by reading the ledger, not by an alert.
Where Dex fell short, and what shipped: the multi-table profile estimate originally omitted BigQuery’s per-query billing floor, so on a warehouse of small tables actual spend came in around 1.7x the forecast. Under-forecasting a cost guard is a real defect, but the Dex team at Exmergo fixed it upstream.
PII: the false positives are the point
Dex flags personal data by column name and value shape, and refuses to project anything it flags. Early on it refused day_name and month_name on my date dimension, the single most-joined table in the warehouse. A *_name suffix scores high enough on the name channel to cross the block threshold, so an entirely ordinary query got a 422.
My first instinct was that the detector was too aggressive. My second, better instinct was that a detector tuned to stop annoying me is one that will eventually let something through. The false positive is not a flaw in the design; it is the design being wrong in the only direction that is safe.
So the fix is not a looser threshold, it is a recorded human decision. Each cleared column sits in version-controlled config with its justification beside it: seven distinct values, the weekday names. Twelve distinct values, the month names. I used the exact, fully-qualified form rather than the pattern form, because a pattern would silently clear any same-named column added to a matching table later. Every entry is a claim a reviewer can check in a pull request instead of assume.
Where Dex fell short, and what shipped: the name channel used to fire on non-string columns, so an integer column counting email addresses scored as an email at 0.9 confidence. Names are evidence, but an INT64 cannot hold an address, and that combination is now de-rated upstream. The override list also gained a pattern form for teams whose column count makes the exact form impractical.
What a leaked credential can reach
The Slack agent needs to read. It never needs to spend and it never needs to mutate state, so those are different credentials: the read token reaches five metadata and query routes and gets a 403 on anything billed or state-changing. A leaked read token cannot materialize a BigQuery table. It cannot even run the drift check, because that one costs money.
Auth also fails closed. With no token configured the service answers 503 on every route rather than serving them all unprotected, so the failure mode of a missing secret is refusal rather than openness.
The part I am most glad about is smaller. The post-deploy smoke test asserts that an unauthenticated request gets a 401. Not that the service is up: that it still says no. A deploy that silently disabled authentication would pass every liveness check ever written, and fails this one.
The gap I hit hardest: refusals had no name
Dex returns a structured envelope, but for a long time the reason for a failure existed only as English prose. My service converts that into an HTTP status the automation branches on, and those statuses are instructions: 409 means “run setup and retry”, 422 means “stop, this is a policy violation”. Getting the mapping wrong does not produce a wrong answer. It produces a wrong behavior.
So I was pattern-matching on error text, and a message reading connection refused: could not connect to bigquery matched my refusal pattern. A transient network fault was scored as a permanent policy violation, which the worker was explicitly forbidden to retry. A blip became an outage by classification.
The immediate fix was mine: anchor the patterns to the exact shapes dex emits rather than searching for substrings anywhere in the message, plus a canary test asserting my anchor still matches upstream’s own constant, so a reword upstream fails loudly in my CI instead of silently downgrading a retry into a stop.
The real fix was upstream from the Dex team at Exmergo, and it has landed. Dex-core 1.4.2 gives every refusal a name: a typed error hierarchy a host can catch, so a consumer branches on RequestError versus ConfigurationError instead of inferring intent from a sentence. It reaches me when I move off the CLI subprocess and onto the programmatic API, which is now the work in front of me. The same release made the storage seam a public extension point, and that is the other half of my two-root problem, since one injected store is what unifies two ledgers.
I filed each of those with file, line, pinned version, and whether I had reproduced it live or only read the source. “That is upstream’s bug” is a claim about code I did not write, and it moves work off my plate, so it deserves more evidence than a claim about my own.
The question that actually matters
Every safety mechanism in this stack is listed in one inventory, and each row answers one question:
If this silently stopped working, what would tell us?
Nine controls here have been found looking active while doing nothing. Not one was found by a failure. Every single one was found by someone going to look.
Two of them are the same incident from opposite ends. An image prune logged success while reclaiming nothing. A disk alert delivered correctly five times with an empty message body. Both were green. Both were, in the sense that matters, off.
So the inventory tracks “believed fine” as its own status, alongside “proven” and “known-blind”, with the date each control was last observed actually firing. A second inventory asks the orthogonal question about features: has this ever been driven against the real thing, and how much of it? Today that answer is 134 of 219 automation nodes have ever executed, and exactly one automated test in the stack crosses a process boundary. Publishing that is uncomfortable, and it is the only version of the document worth keeping.
A control can be active, correct, and still change nothing. That is the sentence I would hang above the desk of anyone wiring a language model to a billed database.
What Dex gave me is somewhere to put the controls that count, close enough to the warehouse that the agent cannot route around them, from an engine that treats refusing as a first-class result rather than an error path. What I gave back is a list of the places that boundary leaked. Both halves are the job.
Resources