# Dex passes 57.3% of data-eng-bench, and 94.7% of what the verifiers check. > Dex + Claude Sonnet 5 resolves 59 of 103 tasks (57.3%) on Snowflake's data-eng-bench at k=1, the highest published figure for any Sonnet 5 configuration, and passes 94.7% of the verifiers' 2,850 individual assertions. The highest published figure for any Claude Sonnet 5 configuration on Snowflake's 103-task suite, measured at k=1: one attempt per task, all 103 tasks attempted, zero errored trials, and no best-of selection. - Benchmark: [data-eng-bench](https://www.exmergo.com/benchmarks/data-eng-bench) (Data Engineering Agent Benchmark), published by Snowflake - Run date: 2026-08-11 - Headline result: 59 / 103 tasks resolved (57.3%) with Claude Sonnet 5 - Benchmark repository: https://github.com/snowflake-labs/data-eng-bench - Raw results: https://github.com/exmergo/dex/tree/main/benchmarks/data-eng-bench - Conditions: k=1, one attempt per task; 103 of 103 tasks; 0 errored trials; 0 retries; Claude Sonnet 5 at xhigh; Dex on 98% of trials ## Headline numbers | Metric | Value | Detail | | --- | --- | --- | | Pass rate, k=1 | 57.3% | 59 of 103 tasks, 95% CI 47.6 to 66.4 | | Verifier assertions | 90.6% | task-weighted mean, 95% CI 86.4 to 94.3 | | Dex trigger rate | 98% | 101 of 103 trials | | Cost per trial | $2.74 | list pricing, $1.82 introductory | | Errored trials | 0 | and zero retries | ## How we ran it data-eng-bench drops an agent into a 2,356-model dbt project with a ticket, then scores whether the task's own pytest verifier runs clean afterwards. This was one full pass over all 103 tasks, one attempt each, with no re-runs and no best-of selection. - **Run**: full-dex-20260810-221947. Started 2026-08-10 22:19:47 UTC, main pass finished 2026-08-11 09:20:36 UTC. - **Tasks**: All 103, one attempt each (k=1). 84 build tasks and 19 fix tasks; 3 easy, 47 medium, 45 hard, 8 very hard. - **Model**: anthropic/claude-sonnet-5 at reasoning_effort: xhigh - **Agent**: claude-code 2.1.226, median 49 steps per trial - **Dex**: v1.6.3 at commit 1554d248, exmergo-dex-core 1.6.3, warm cache in the image - **Image**: ghcr.io/snowflake-labs/data-eng-bench-base:1.0.0, dex arm - **Dataset**: snowflake-labs/data-eng-bench@v1.0, task digests unmodified - **Throughput**: 11.01 h wall clock at n=2 concurrency, 9.4 trials per hour, 22 core-hours total - **Headroom**: Longest agent execution 2,026 s against a 4,000 s ceiling. Nothing was truncated. - **Reliability**: Zero errored trials and zero retries. One trial hit a host-side network failure during agent setup, 38 s in and before any tokens were spent, and was re-run to completion on a provably identical configuration. ## Results Fifty-nine of 103 tasks scored a clean pass, with zero errored trials and zero retries. Underneath that binary reward, 2,699 of 2,850 individual verifier assertions passed, and 19 of the 44 failed tasks were blocked by exactly one assertion. The binary number is what a leaderboard records; the assertion record is where the remaining work actually is. Snowflake published their own data-eng-bench numbers on 6 August 2026, five days before this run. Their Pass@1 is the comparable measure, and on it this run lands at the top of the Sonnet 5 group by 0.7 points, a margin with no statistical content at this sample size. ## Against the published board Snowflake define Pass@1 as the mean single-attempt pass rate over a k=3 run, which is the same quantity a k=1 pass rate estimates, measured on three times the sample. Pass^3 counts a task only when all three attempts succeed, and a k=1 run cannot produce one. | Harness | Model | Attempts | Pass@1 | Pass^3 | Tokens/trial | | --- | --- | --- | --- | --- | --- | | Snowflake CoCo | Opus 5 | k=3 | 73.8% | 64.1% | 1,070,515 | | Claude Code | Opus 5 | k=3 | 69.6% | 60.2% | 4,810,868 | | Snowflake CoCo | GPT 5.6 Sol | k=3 | 64.1% | 55.3% | 436,236 | | Codex | GPT 5.6 Sol | k=3 | 60.5% | 49.5% | 812,306 | | **Claude Code + dex** (ours) | Claude Sonnet 5 | k=1 | 57.3% | not measured | 5,385,156 | | Snowflake CoCo | Claude Sonnet 5 | k=3 | 56.6% | 40.8% | 2,879,678 | | Claude Code | Claude Sonnet 5 | k=3 | 56.6% | 40.8% | 7,914,293 | This run is first among every published Sonnet 5 configuration, and fifth of seven overall. The margin over the two Snowflake published at 56.6% is 0.7 points, a single task on a 103-task suite, so the honest reading is that all three are statistically indistinguishable and this one is nominally at the top. A genuinely 56.6% agent produces a draw at least this good about half the time. Snowflake publish identical figures for both of their Sonnet 5 configurations, so treat those two rows as one reference point rather than two. Source: Snowflake engineering blog, 6 August 2026. Their rows are k=3; ours is k=1. ## Where Dex shines What the run shows about the tool, separated from what it shows about the model. Every figure here carries its bound in the same place as the number. ### Dex fired on 98% of trials (101 of 103) This is a measurement of Dex rather than of a tool that happened to be installed. The CLI carried 90% of the trials and the Skill wrapper 18%, for roughly 1,200 invocations in total. Every one of them was `dex explore`: the agent used it to verify real columns, types, null rates, grain and join keys before writing SQL. *Caveat: Usage is counted from the agent transcript, not from the installed skill files. A naive scan of the agent directory reports 100% on any run where the skills were merely present.* ### The more Dex ran, the more tasks passed (71% against 50%) Trials that made twenty or more dex calls passed at 71%, against 50% for trials that made none at all. The gradient is monotonic from five calls upward, and the heaviest-usage bucket is 67% hard or harder yet still passed above the run average. *Caveat: Observational, and not a causal claim. There is no control arm in this run, the strata are small, and the arrow could point either way: an agent that has already lost the thread may also stop reaching for tools, which would make low usage a symptom rather than a cause.* ### On hard tasks the gap is widest (70% against 20%) Stratified by difficulty, so the gradient is not simply easier tasks attracting more tool use. Among hard tasks, trials that leaned on Dex passed at 70% against 20% for trials that barely touched it. Among medium tasks it is 63% against 44%. *Caveat: Same caution as above. The thin cell here is three passes out of fifteen, so the 20% figure is a very small denominator.* ### Nearly right almost everywhere (94.7%) 2,699 of 2,850 individual verifier assertions passed. Even across the 44 tasks that failed, 87.6% of assertions still passed, 19 of them were blocked by exactly one assertion, and 29 by two or fewer. `marketing-campaigns-harbor` passed 218 of 219: the single failure compared a date against a timestamp. ### The failure surface is semantic, not structural (38 of 44) In 38 of the 44 failed tasks the agent produced a buildable, correctly shaped model and then lost on a boundary condition or a reading of an ambiguous spec. Only six never produced a working deliverable at all. Getting the shape right is the part that depends on knowing what is actually in the warehouse. ### 32% fewer tokens for the same score (5.39 M) Against the identical harness and model on the published board, this run used 5.39 M tokens per trial against 7.91 M, while running at `reasoning_effort: xhigh`, a setting that pushes token use up rather than down. 97.8% of the prompt volume was cache reads. *Caveat: Snowflake's published cost and token columns do not reconcile with each other, so at least one of them is on a basis we cannot see. We state our own basis and make no cost comparison.* ### Pass rate by dex calls per trial Each bar is a bucket of the 103 trials, split by how many `dex` subcommand invocations appear in the agent transcript. | Bucket | Value | Trials | | --- | --- | --- | | 0 calls | 50% | n = 10 | | 1 to 4 | 28% | n = 18 | | 5 to 9 | 59% | n = 27 | | 10 to 19 | 67% | n = 27 | | 20 or more | 71% | n = 21 | *Caveat: Bucket sizes are small and the 1-to-4 bucket sits below the zero-call bucket, so read the overall shape rather than any single pair.* ## Two scales, one run The reward is binary and total: a task scores only when pytest exits with zero failures and zero skips. Because the verifiers are written as many independent components, the run also produced a much finer record underneath the binary one, and reading it changes what the 57.3% means. | Measure | Value | What it counts | | --- | --- | --- | | Task reward, binary | 57.3% | 59 of 103 tasks. What a leaderboard records. | | Assertions, pooled | 94.7% | 2,699 of 2,850 individual verifier assertions. | | Assertions, task-weighted | 90.6% | Mean of per-task assertion fractions, each task counted once. | Across the 44 failed tasks alone, 87.6% of assertions still passed. Nineteen of the 44 were blocked by exactly one assertion and 29 by two or fewer. The binary reward is doing what it is designed to do, but it compresses “built the wrong thing” and “got one boundary condition wrong on a 219-assertion suite” into the same zero. *Two cautions on reading this. These are not free points: a near-miss only converts if its cause is stochastic, and most of these look systematic, so the assertion rate describes where the remaining work is rather than a pass rate in waiting. And prefer the task-weighted 90.6% to the pooled 94.7% when quoting a single figure, because pooling overweights the one task that carries 219 assertions.* ## Where the other 44 lost Every failed trial was classified from its verifier output. The distribution is lopsided in a useful direction: the agent almost always produced a buildable, correctly shaped model and then lost on semantics. | Bucket | Failure mode | Tasks | Within two assertions | Description | | --- | --- | --- | --- | --- | | E | Classification threshold or tie-break | 12 of 44 | 9 of 12 | Scores correct, the bucket they land in is not. Percentile edges, tier waterfalls, ties. | | D | Population boundary (row count) | 8 of 44 | 8 of 8 | Right shape, wrong row count. Off by an edge case, not by a formula. | | A | Deliverable missing or unbuildable | 6 of 44 | 0 of 6 | The model, or a model it depends on, was never produced. Nothing about the SQL was judged. | | B | Order-status population semantics | 6 of 44 | 2 of 6 | Cancelled, returned, or failed orders included or excluded against what the verifier expects. | | C | Grain and join fan-out | 4 of 44 | 4 of 4 | A join multiplied rows, so totals inflate by a roughly integer factor. | | F | NULL handling | 4 of 44 | 3 of 4 | A NULL survived into a column the verifier requires to be populated. | | G | Type, precision, or format | 2 of 44 | 2 of 2 | Numerically right, typed or truncated wrong. | | H | Stateful algorithm | 2 of 44 | 1 of 2 | FIFO and LIFO layer allocation. Sequential state the SQL has to carry. | Buckets D and E together are 20 of the 44, and both are boundary problems rather than modeling problems. Bucket B deserves its own attention: six separate tasks turned on whether cancelled, returned, or failed orders belong in the population. The tickets frequently do not say, and the verifiers always have an opinion. Bucket A is the only group where nothing about the SQL was judged, and it is six tasks. ## Reading the numbers - This is a k=1 run: one attempt per task. The 57.3% carries a Wilson 95% interval of 47.6% to 66.4%, roughly plus or minus 9.4 points, and a single task is 0.97 points. Any comparison narrower than about ten points is noise at this sample size. - The assertion-level interval is about two and a half times tighter than the binary one on the same run, 7.9 points against 19.4. Where a single figure is needed, the task-weighted 90.6% is the more sensitive instrument. - The benchmark's own difficulty labels barely separate the outcomes. Medium tasks passed at 60% and hard at 53%, a seven-point spread inside a plus or minus nine-point interval. The failure taxonomy above predicts this model's misses far better than the label does. - Build and fix tasks scored 58.3% (49 of 84) and 52.6% (10 of 19). Snowflake characterise the suite as 84 build and 19 fix but do not publish the split, so this is a first. At n=19 the fix-task interval runs from 31.7% to 72.7% and carries almost no information. - Cost per trial was essentially flat between passes ($2.64) and failures ($2.87), so the failures are not agents burning budget while flailing. - Every trial's assertion tally, cost, step count and dex call count is in the table at the foot of this page, and the raw verifier output is committed alongside the harness configuration. ## Reading this run The arithmetic here is easy for anyone to check, so everything that qualifies the number sits on this page rather than in a footnote. - This is a k=1 run: one attempt per task, below the leaderboard's k=3 minimum. It sizes a submission; it is not one. - There is no baseline arm in this run, so no delta is attributable to Dex. Nothing here measures what the same loop would have scored without it, and the usage gradient above is observational. - Snowflake's published rows are k=3 and ours is k=1. Both estimate the same quantity, but ours is the noisier estimate on roughly a third of the sample. - We publish no Pass^3 figure. A k=1 run cannot produce one, and no reanalysis will conjure it. - Our arm modifies the base image. It adds uv, a warm dex-core cache, the dex CLI on PATH, and an /app/CLAUDE.md, reproduced verbatim below. - The model ran at reasoning_effort: xhigh. Snowflake do not state their setting. - Different day, different hardware, different operator, and no shared control arm. This is a cross-report comparison, not a controlled experiment. - Task digests match snowflake-labs/data-eng-bench@v1.0. Nothing under tasks/ was modified. - Cost is reported at list pricing ($3 and $15 per MTok, $3.75 cache write, $0.30 cache read), which reconstructs the harness figure of $282.12 exactly. The same token ledger at the introductory pricing running through 2026-08-31 comes to $187.95. - Snowflake's published cost and token columns do not reconcile with each other, so we make no cost comparison against the board. ### The CLAUDE.md this arm added 94 of the 103 tickets already instruct the agent to run `echo $DB_TYPE`, and 100 of 103 already instruct it to create a `profiles.yml` with profile name `retail_dw_master`. Steps 1, 2 and 5 therefore restate what the tickets already say, and the marginal prompt content is step 3. ```text # Project setup sequence Every task in this project follows the same order. Do not skip a step. 1. `echo $DB_TYPE` to determine the backend. 2. Write `profiles.yml` for that backend, profile name `retail_dw_master`. 3. **`dex explore profile ` for every source table the ticket names.** This returns real columns, types, null rates, grain and verified join keys. 4. Write or edit the model SQL. 5. `dbt run -s ` to verify it builds. Profiling tells you what the data *is*. It does not tell you what the ticket wants. Where the two seem to disagree, the ticket and the existing model are the specification: implement the change that was asked for, and leave the surrounding logic alone. ## dex commands dex explore profile
columns, types, null rates, grain, PII dex explore inventory --rank what exists, ranked by relevance dex explore relationships --verify join keys, verified against the data dex explore query "