`dex maintain` as long-term memory for agents doing analytics engineering

Three artifacts, three kinds of memory

The .dex/ directory is not a cache directory that happens to be committed. The three files do genuinely different jobs, and conflating them is how teams end up with a drift baseline nobody trusts.

.dex/config.yml: adjudicated policy

This is the memory of decisions a human already made, written down so nobody has to make them again.

Most of it is scope and governance: which datasets the tooling may read, which project, which dbt target, what the spend ceiling is. The part that earns the "memory" description is pii_overrides.

PII detection is automatic and, on a schemaless source, noisy in a specific way. In this warehouse the CDC changelogs carry a document_name column holding a resource path, something like enterprises/{id}/members/{user_id}. A name-based scanner sees _name and flags it as personal data. It is not; it is a path. Somebody sat down, looked at the values, and adjudicated it.

Without a durable home, that adjudication dies at the end of the session. The next re-profile re-flags the column, the next agent hits a refusal on a perfectly reasonable probe, and somebody works out the same answer again. With pii_overrides, the verdict lives in a git-reviewable file, carries a reason string pointing at the documentation that explains it, and survives every subsequent profile.

That is the shape of the whole idea in miniature: a human judgment, recorded once, applied forever, visible in review. The alternative that must never happen is hand-editing the cache to clear a flag, which produces the same short-term result and no memory at all.

.dex/cache.json: the world as last observed

This is the expensive one. It holds, per object, the column profiles, the inferred grain and candidate keys, the PII flags, the relationship graph, and the data-quality notes, each stamped with when it was profiled.

Concretely, from this session's rebuild: 201 objects across twelve datasets, covering raw, staging, intermediate, and marts layers in three environments, carrying 84 inferred relationships, 155 PII-flagged columns, and 49 data-quality notes. Building it from an empty cache took about twenty-two minutes and billed 5.12 GB against an 8.69 GB estimate.

Two things follow from that cost.

The first is that it has to be committed. Nobody is paying five gigabytes of scan every time a new session starts, and no agent should be asked to work from a partial picture because the full one was too expensive to obtain. Committing the cache means the next session, the next teammate, and CI all start from the same understanding for free.

The second is that its freshness is a real property, not an implementation detail. Each profile carries its own timestamp, so staleness is visible instead of silently assumed away. When a re-map skips an object, the prior profile is carried forward with its original timestamp rather than quietly vanishing.

.dex/snapshot.json: the state someone vouched for

The cache says what the world looks like. The snapshot says what the world looked like when a human last confirmed it was correct.

That distinction is the entire mechanism. Drift is not "the warehouse changed since some arbitrary point"; it is "the warehouse no longer matches the state a person signed off on". maintain snapshot is the act of signing off. It pins the current cache, so the grain baseline is the exact-distinct verdicts already computed rather than a fresh guess, and it fingerprints the dbt project's own layers alongside it.

From this session's snapshot: 201 datasets, 201 with full column detail, 169 grain baselines, 84 relationships, plus the project fingerprint covering 92 files, 47 model nodes, 15 sources, 11 semantic models, and 27 metrics.

The "201 of 201 with column detail" number is the one to look at. A baseline can only compare columns for objects it has columns for, so a snapshot taken over a partial map is a baseline with holes in it, and the holes do not announce themselves.

The loop, as it actually ran

The session that produced these numbers was a materialization change: thirteen staging models over CDC changelogs moved from views to incremental merge tables, because the raw partitions expire after ninety days and a view over an expiring window silently loses any document whose last event ages out.

That change is close to a worst case for a drift baseline. It altered the relation type of fourteen objects (view becomes table), added three bookkeeping columns to each of thirteen models, and added two more columns to two of them. Every one of those is precisely what a drift detector is built to shout about, which meant the old baseline would report the entire staging layer as drifted until it was re-pinned.

The loop:

1. Build and verify the change. Thirty-eight models, 265 passing tests, seventeen warnings, zero failures.
2. Clear the cache and re-map in full. 201 of 201 profiled, zero skipped, zero carried forward, zero dropped.
3. Re-pin the baseline. maintain snapshot, with no thin-cache or staleness warning in the envelope.
4. Prove the new baseline is clean. maintain check across all four axes.

The result of step 4:

0 schema drifts

0 volume drifts

0 grain drifts

0 semantic drifts

The happy path. But the semantic zero is the one worth dwelling on. Eleven semantic models and twenty-seven metrics sit on marts that were entirely rebuilt from re-materialized staging tables, and not one entity, measure, dimension, or metric definition moved. That is a claim nobody could make by reading a diff, because the diff shows SQL, not whether the meaning survived. It came from comparing a pinned definition fingerprint against a freshly computed one.

Cost of the confidence: about 0.87 GB for the check, on top of the re-map.

What this buys

A new session starts informed. An agent opening this repository tomorrow reads a committed cache and knows the grain of every table, which columns are keys, which are PII, and how the objects join, without spending a byte. That is the difference between an agent that asks "what is in this warehouse" and one that asks "what changed in this warehouse".

A human's adjudications outlive the conversation they happened in. The false-positive PII verdict, the budget, the dataset allowlist. All of it is policy the agent inherits rather than re-litigates.

Drift becomes a review artifact. Because the baseline is a committed file, a change to it shows up in a pull request. Re-pinning a baseline is a visible act, which means quietly accepting drift is also a visible act. That property is lost the moment these files live in a temp directory or a user-level cache.

A colleague inherits the same memory. Pull the branch, get the understanding. There is no separate onboarding step where a person or an agent rebuilds context that already existed.

Two honest failure modes

A use case that only lists benefits is marketing. These both happened here.

A poisoned baseline is worse than a stale one

On an earlier occasion the baseline was re-pinned from a default map rather than a full one. Past fifty objects, a default map profiles only the top twenty-five by rank and enters the rest as metadata alone. Snapshotting that produced 1,548 false column_added findings, against 373 from the stale baseline it replaced, plus three false HIGH table_dropped findings for relations that had been dropped minutes earlier and were carried forward as ghosts.

Thousands of low-severity findings and a few false criticals are exactly where a real drop or retype hides. The discipline that came out of it is now followed every time: remove the cache first, map in full, snapshot, then check to zero before committing. Verify the cache before snapshotting, because the cache's completeness is the baseline's quality.

The spend guard can forbid its own hygiene

Re-pinning is not free, and in this session it was refused outright.

The cumulative daily spend ceiling is checked against each command's estimate. A full re-map estimates 8.69 GB. A full project build estimates 6.70 GB. Those two commands together cannot fit under a 10 GB ceiling, no matter that they billed 5.12 GB and 3.45 GB respectively. So on any day the project was built, re-baselining was structurally impossible, which is precisely the day a change altering relation types and columns makes it necessary.

The real money at stake in that standoff was around three cents.

The estimate side of this has its own cause worth naming. dbt has no dry run for a MERGE, so an incremental model whose filter prunes to a three-day partition slice is priced as though it read the whole table. Across this conversion the billed cost fell 27% (4.75 GB to 3.45 GB) while the estimate moved 3% (6.90 GB to 6.70 GB). The guard enforces against the number that did not move.

Both are filed upstream as #324 (gate the cumulative ceiling on billed rather than estimated bytes) and #323 (dry-run the compiled batch statement for incremental nodes). The immediate resolution was to raise the ceiling and treat it as a runaway backstop, keeping the per-command budget as the real control.

When to reach for maintain

The trigger is not a phrasing, it is a situation: something that used to work has started behaving differently, and the cause is more likely upstream than in the code someone just wrote.

• A test began failing with no code change.
• A dashboard's numbers moved and nobody knows why.
• A model is suddenly empty, or suddenly duplicated.
• A primary key acquired duplicates.
• A metric definition no longer matches what the business means by it.
• Someone is about to promote a change and wants proof the semantic layer survived it.

The everyday entry point is maintain check, which sweeps all four axes and ranks findings by blast radius. Schema, volume, and the reference half of semantic are free, because they read metadata and the snapshot. Grain and dimension cardinality scan the warehouse and take a cost handshake first.

For learning an unfamiliar warehouse, explore is the right tool. For authoring models and metrics, transform. maintain is the one that answers a narrower and more valuable question: is what I believed about this warehouse still true?

An agent cannot answer that from the repository alone. It needs a record of what it believed, when it believed it, and who agreed. That record is three files, in git, next to the code they describe.