Dex passes 57.3% of data-eng-bench, and 94.7% of what the verifiers check.
The highest published figure for any Claude Sonnet 5 configuration on Snowflake's 103-task suite, measured at k=1: one attempt per task, all 103 tasks attempted, zero errored trials, and no best-of selection.
Run on
k=1, one attempt per task
103 of 103 tasks
0 errored trials
0 retries
Claude Sonnet 5 at xhigh
Dex on 98% of trials
Pass rate on data-eng-bench
0.0%
0 / 103 tasks resolved
95% CI 47.6 to 66.4
Verifier assertions passed
94.7%
2,699 of 2,850
90.6% task-weighted
Dex + Claude Sonnet 5 on Snowflake’s data-eng-bench.
Snowflake define Pass@1 as the mean single-attempt pass rate over a k=3 run, which is the same quantity a k=1 pass rate estimates, measured on three times the sample. Pass^3 counts a task only when all three attempts succeed, and a k=1 run cannot produce one.
HarnessModelPass@1Pass^3Tokens/trial
Snowflake CoCok=3
Opus 5
73.8%
64.1%1.07 M
Claude Codek=3
Opus 5
69.6%
60.2%4.81 M
Snowflake CoCok=3
GPT 5.6 Sol
64.1%
55.3%0.44 M
Codexk=3
GPT 5.6 Sol
60.5%
49.5%0.81 M
Claude Code + dexk=1
Claude Sonnet 5Ours
57.3%
n/a5.39 M
Snowflake CoCok=3
Claude Sonnet 5
56.6%
40.8%2.88 M
Claude Codek=3
Claude Sonnet 5
56.6%
40.8%7.91 M
This run is first among every published Sonnet 5 configuration, and fifth of seven overall. The margin over the two Snowflake published at 56.6% is 0.7 points, a single task on a 103-task suite, so the honest reading is that all three are statistically indistinguishable and this one is nominally at the top. A genuinely 56.6% agent produces a draw at least this good about half the time. Snowflake publish identical figures for both of their Sonnet 5 configurations, so treat those two rows as one reference point rather than two.
Snowflake engineering blog, 6 August 2026. Their rows are k=3; ours is k=1.
Where Dex shines
What the run shows about the tool, separated from what it shows about the model. Every figure here carries its bound in the same place as the number.
101 of 103
Dex fired on 98% of trials
This is a measurement of Dex rather than of a tool that happened to be installed. The CLI carried 90% of the trials and the Skill wrapper 18%, for roughly 1,200 invocations in total. Every one of them was dex explore: the agent used it to verify real columns, types, null rates, grain and join keys before writing SQL.
Usage is counted from the agent transcript, not from the installed skill files. A naive scan of the agent directory reports 100% on any run where the skills were merely present.
71% against 50%
The more Dex ran, the more tasks passed
Trials that made twenty or more dex calls passed at 71%, against 50% for trials that made none at all. The gradient is monotonic from five calls upward, and the heaviest-usage bucket is 67% hard or harder yet still passed above the run average.
Observational, and not a causal claim. There is no control arm in this run, the strata are small, and the arrow could point either way: an agent that has already lost the thread may also stop reaching for tools, which would make low usage a symptom rather than a cause.
70% against 20%
On hard tasks the gap is widest
Stratified by difficulty, so the gradient is not simply easier tasks attracting more tool use. Among hard tasks, trials that leaned on Dex passed at 70% against 20% for trials that barely touched it. Among medium tasks it is 63% against 44%.
Same caution as above. The thin cell here is three passes out of fifteen, so the 20% figure is a very small denominator.
94.7%
Nearly right almost everywhere
2,699 of 2,850 individual verifier assertions passed. Even across the 44 tasks that failed, 87.6% of assertions still passed, 19 of them were blocked by exactly one assertion, and 29 by two or fewer. marketing-campaigns-harbor passed 218 of 219: the single failure compared a date against a timestamp.
38 of 44
The failure surface is semantic, not structural
In 38 of the 44 failed tasks the agent produced a buildable, correctly shaped model and then lost on a boundary condition or a reading of an ambiguous spec. Only six never produced a working deliverable at all. Getting the shape right is the part that depends on knowing what is actually in the warehouse.
5.39 M
32% fewer tokens for the same score
Against the identical harness and model on the published board, this run used 5.39 M tokens per trial against 7.91 M, while running at reasoning_effort: xhigh, a setting that pushes token use up rather than down. 97.8% of the prompt volume was cache reads.
Snowflake's published cost and token columns do not reconcile with each other, so at least one of them is on a basis we cannot see. We state our own basis and make no cost comparison.
Pass rate by dex calls per trial
Each bar is a bucket of the 103 trials, split by how many dex subcommand invocations appear in the agent transcript.
0 calls
50%n = 10
1 to 4
28%n = 18
5 to 9
59%n = 27
10 to 19
67%n = 27
20 or more
71%n = 21
Bucket sizes are small and the 1-to-4 bucket sits below the zero-call bucket, so read the overall shape rather than any single pair.
Two scales, one run
The reward is binary and total: a task scores only when pytest exits with zero failures and zero skips. Because the verifiers are written as many independent components, the run also produced a much finer record underneath the binary one, and reading it changes what the 57.3% means.
Task reward, binary
57.3%
59 of 103 tasks. What a leaderboard records.
Assertions, pooled
94.7%
2,699 of 2,850 individual verifier assertions.
Assertions, task-weighted
90.6%
Mean of per-task assertion fractions, each task counted once.
0%50%100%
Across the 44 failed tasks alone, 87.6% of assertions still passed. Nineteen of the 44 were blocked by exactly one assertion and 29 by two or fewer. The binary reward is doing what it is designed to do, but it compresses “built the wrong thing” and “got one boundary condition wrong on a 219-assertion suite” into the same zero.
Two cautions on reading this. These are not free points: a near-miss only converts if its cause is stochastic, and most of these look systematic, so the assertion rate describes where the remaining work is rather than a pass rate in waiting. And prefer the task-weighted 90.6% to the pooled 94.7% when quoting a single figure, because pooling overweights the one task that carries 219 assertions.
Where the other 44 lost
Every failed trial was classified from its verifier output. The distribution is lopsided in a useful direction: the agent almost always produced a buildable, correctly shaped model and then lost on semantics.
Failure modeTasksWithin two assertions
EClassification threshold or tie-break
Scores correct, the bucket they land in is not. Percentile edges, tier waterfalls, ties.
12 of 44
9 of 12
DPopulation boundary (row count)
Right shape, wrong row count. Off by an edge case, not by a formula.
8 of 44
8 of 8
ADeliverable missing or unbuildable
The model, or a model it depends on, was never produced. Nothing about the SQL was judged.
6 of 44
0 of 6
BOrder-status population semantics
Cancelled, returned, or failed orders included or excluded against what the verifier expects.
6 of 44
2 of 6
CGrain and join fan-out
A join multiplied rows, so totals inflate by a roughly integer factor.
4 of 44
4 of 4
FNULL handling
A NULL survived into a column the verifier requires to be populated.
4 of 44
3 of 4
GType, precision, or format
Numerically right, typed or truncated wrong.
2 of 44
2 of 2
HStateful algorithm
FIFO and LIFO layer allocation. Sequential state the SQL has to carry.
2 of 44
1 of 2
Buckets D and E together are 20 of the 44, and both are boundary problems rather than modeling problems. Bucket B deserves its own attention: six separate tasks turned on whether cancelled, returned, or failed orders belong in the population. The tickets frequently do not say, and the verifiers always have an opinion. Bucket A is the only group where nothing about the SQL was judged, and it is six tasks.
How we ran it
data-eng-bench drops an agent into a 2,356-model dbt project with a ticket, then scores whether the task's own pytest verifier runs clean afterwards. This was one full pass over all 103 tasks, one attempt each, with no re-runs and no best-of selection.
Run
full-dex-20260810-221947. Started 2026-08-10 22:19:47 UTC, main pass finished 2026-08-11 09:20:36 UTC.
Tasks
All 103, one attempt each (k=1). 84 build tasks and 19 fix tasks; 3 easy, 47 medium, 45 hard, 8 very hard.
Model
anthropic/claude-sonnet-5 at reasoning_effort: xhigh
Agent
claude-code 2.1.226, median 49 steps per trial
Dex
v1.6.3 at commit 1554d248, exmergo-dex-core 1.6.3, warm cache in the image
Image
ghcr.io/snowflake-labs/data-eng-bench-base:1.0.0, dex arm
11.01 h wall clock at n=2 concurrency, 9.4 trials per hour, 22 core-hours total
Headroom
Longest agent execution 2,026 s against a 4,000 s ceiling. Nothing was truncated.
Reliability
Zero errored trials and zero retries. One trial hit a host-side network failure during agent setup, 38 s in and before any tokens were spent, and was re-run to completion on a provably identical configuration.
Reading the numbers
This is a k=1 run: one attempt per task. The 57.3% carries a Wilson 95% interval of 47.6% to 66.4%, roughly plus or minus 9.4 points, and a single task is 0.97 points. Any comparison narrower than about ten points is noise at this sample size.
The assertion-level interval is about two and a half times tighter than the binary one on the same run, 7.9 points against 19.4. Where a single figure is needed, the task-weighted 90.6% is the more sensitive instrument.
The benchmark's own difficulty labels barely separate the outcomes. Medium tasks passed at 60% and hard at 53%, a seven-point spread inside a plus or minus nine-point interval. The failure taxonomy above predicts this model's misses far better than the label does.
Build and fix tasks scored 58.3% (49 of 84) and 52.6% (10 of 19). Snowflake characterise the suite as 84 build and 19 fix but do not publish the split, so this is a first. At n=19 the fix-task interval runs from 31.7% to 72.7% and carries almost no information.
Cost per trial was essentially flat between passes ($2.64) and failures ($2.87), so the failures are not agents burning budget while flailing.
Every trial's assertion tally, cost, step count and dex call count is in the table at the foot of this page, and the raw verifier output is committed alongside the harness configuration.
Reading this run
The arithmetic here is easy for anyone to check, so everything that qualifies the number sits on this page rather than in a footnote.
This is a k=1 run: one attempt per task, below the leaderboard's k=3 minimum. It sizes a submission; it is not one.
There is no baseline arm in this run, so no delta is attributable to Dex. Nothing here measures what the same loop would have scored without it, and the usage gradient above is observational.
Snowflake's published rows are k=3 and ours is k=1. Both estimate the same quantity, but ours is the noisier estimate on roughly a third of the sample.
We publish no Pass^3 figure. A k=1 run cannot produce one, and no reanalysis will conjure it.
Our arm modifies the base image. It adds uv, a warm dex-core cache, the dex CLI on PATH, and an /app/CLAUDE.md, reproduced verbatim below.
The model ran at reasoning_effort: xhigh. Snowflake do not state their setting.
Different day, different hardware, different operator, and no shared control arm. This is a cross-report comparison, not a controlled experiment.
Task digests match snowflake-labs/data-eng-bench@v1.0. Nothing under tasks/ was modified.
Cost is reported at list pricing ($3 and $15 per MTok, $3.75 cache write, $0.30 cache read), which reconstructs the harness figure of $282.12 exactly. The same token ledger at the introductory pricing running through 2026-08-31 comes to $187.95.
Snowflake's published cost and token columns do not reconcile with each other, so we make no cost comparison against the board.
The CLAUDE.md this arm added
94 of the 103 tickets already instruct the agent to run echo $DB_TYPE, and 100 of 103 already instruct it to create a profiles.yml with profile name retail_dw_master. Steps 1, 2 and 5 therefore restate what the tickets already say, and the marginal prompt content is step 3.
# Project setup sequence
Every task in this project follows the same order. Do not skip a step.
1. `echo $DB_TYPE` to determine the backend.
2. Write `profiles.yml` for that backend, profile name `retail_dw_master`.
3. **`dex explore profile <table>` for every source table the ticket names.**
This returns real columns, types, null rates, grain and verified join keys.
4. Write or edit the model SQL.
5. `dbt run -s <model>` to verify it builds.
Profiling tells you what the data *is*. It does not tell you what the ticket
wants. Where the two seem to disagree, the ticket and the existing model are the
specification: implement the change that was asked for, and leave the surrounding
logic alone.
## dex commands
dex explore profile <table> columns, types, null rates, grain, PII
dex explore inventory --rank what exists, ranked by relevance
dex explore relationships --verify join keys, verified against the data
dex explore query "<SELECT ...>" guarded ad-hoc probe
The connector and database path default from `$DB_TYPE` and `$DUCKDB_PATH`, so
no flags are needed. Output is JSON on stdout. It is read-only and writes only a
`.dex/` cache, which dbt ignores.
All 103 trials
Every trial in the run, with its assertion tally, cost, step count and dex call count. Failures also carry their taxonomy bucket, how many assertions stood between the trial and a pass, and the first assertion that failed.
103 of 103 trials shown
ResultTaskLevelAssertionsCostStepsdex
dbt-abc-classification
med49 / 49$3.496820
dbt-basket-composition-analysis
hard85 / 85$3.453713
dbt-calculate-running-balance
med27 / 27$1.914740
dbt-campaign-performance
med1 / 1$1.24303
dbt-cart-abandonment-recovery
med1 / 1$2.485224
dbt-consolidate
med1 / 1$0.95408
dbt-coupon-effectiveness
med60 / 60$3.13516
dbt-customer-churn-cohorts
v.hard3 / 3$2.625517
dbt-customer-cohort-retention
med45 / 45$3.505816
dbt-customer-cross-sell-insights
hard18 / 18$4.66805
dbt-customer-geographic
easy9 / 9$1.334723
dbt-customer-lifecycle-journey
med1 / 1$1.70456
dbt-customer-lifetime-value
hard72 / 72$3.195138
dbt-customer-order-analytics
med13 / 13$1.945519
dbt-customer-risk-scoring
hard11 / 11$1.42370
dbt-daily-order-summary
easy13 / 13$0.59210
dbt-dq-macro-enforcement
hard26 / 26$3.03445
dbt-email-campaign-tracker
med1 / 1$1.755419
dbt-fix-cac-payback-waterfall
med14 / 14$2.33648
dbt-fix-customer-churn
med8 / 8$1.23328
dbt-fix-customer-snapshot-and-build-dimension
hard20 / 20$1.79456
dbt-fix-daily-cohorts
med11 / 11$1.613615
dbt-fix-daily-revenue
med13 / 13$1.914811
dbt-fix-email-attribution
med10 / 10$3.066619
dbt-fix-inventory-model
med1 / 1$2.614711
dbt-fix-paid-search-attribution
medn/a$2.51558
dbt-fix-refund-reconciliation
med1 / 1$0.68200
dbt-fix-timezone-sales
med10 / 10$2.475911
dbt-fraud-detection-model
hard13 / 13$3.177332
dbt-gl-reconciliation
v.hard26 / 26$1.83452
dbt-harbor-product-affinity
hard25 / 25$3.315629
dbt-harbor-warehouse-capacity
hard42 / 42$3.407320
dbt-hr-analytics
hard32 / 32$4.359510
dbt-inventory-stockout-risk
med18 / 18$3.31509
dbt-loyalty-points-analysis
hard11 / 11$1.574418
dbt-order-fulfillment-analytics
hard64 / 64$5.409920
dbt-order-fulfillment-metrics
hard19 / 19$2.04532
dbt-order-reconciliation
hard47 / 47$2.075616
dbt-price-elasticity
hard13 / 13$1.57485
dbt-product-affinity
hard63 / 63$3.987318
dbt-product-category-analytics
hard11 / 11$1.464325
dbt-product-sales-velocity
hard60 / 60$2.844422
dbt-receivables-aging-buckets
hard4 / 4$3.26647
dbt-retail-task-01-channel-revenue
easy10 / 10$1.734412
dbt-retail-task-04-market-roi
med10 / 10$1.854928
dbt-rfm-customer-tiering
v.hard5 / 5$4.83741
dbt-session-attribution
v.hard1 / 1$1.96366
dbt-test-orders-filter
med15 / 15$2.354211
dbt-three-way-matching
hard52 / 52$2.424622
dbt-warehouse-fulfillment-analytics
med1 / 1$1.84418
dbt-web-session-analytics
med22 / 22$2.39490
dbt-weekly-sales-growth
med29 / 29$1.91512
deferred-revenue-recognition
hard20 / 20$1.91415
late-arriving-orders-reconciliation
hard19 / 19$3.11530
payment-risk-scoring
med83 / 83$2.895718
pos-operations
hard69 / 69$8.2012526
shipping-fulfillment-quality-scoring
med125 / 125$2.024515
web-session-quality-scoring
med69 / 69$2.21399
workforce-analytics
hard134 / 134$8.1110552
dbt-product-performance-metrics
Ashort by 4. Staging model stg_order_lines__products does not exist
med29 / 33$3.36564
dbt-channel-attribution-analysis
Ashort by 12. Staging model stg_orders__channels does not exist
med37 / 49$3.635816
dbt-incremental-late-arriving-sales
Ashort by 30. Missing columns in order_version_history: valid_from, valid_to, version_num, order_id, channel_id and others
hard4 / 34$4.30814
dbt-payment-analytics
Ashort by 11. component_2_no_inf_nan_values: Catalog Error: Table with name rpt_payment_analytics does not exist
med1 / 12$0.551710
dbt-inventory-analysis
Ashort by 6. Missing required columns: inventory_value, max_single_item_value
med0 / 6$1.744616
dbt-inventory-turnover-analysis
Adbt run failed: could not find profile named 'dbt_project'. No tests ran.
medn/a$4.88760
dbt-fix-division-by-zero
Bshort by 1. component_3_row_count: Row count mismatch: 390 vs 382
med14 / 15$2.08416
dbt-customer-cltv-forecasting
Bshort by 1. Total current_lifetime_revenue 7,935,436.41 does not match source 599,569.28
hard4 / 5$1.33352
dbt-fix-customer-ltv
Bshort by 3. Calculation mismatch for customer 007f4385
med10 / 13$2.89612
dbt-fix-product-metrics
Bshort by 4. Revenue mismatch: report=643,615.84, actual=596,282.96 (tolerance 0.01)
med7 / 11$1.673910
dbt-customer-ltv-fix
Bshort by 9. Total LTV 5,229,842.66 differs from expected 621,893.43
hard14 / 23$4.27752
dbt-fix-category-revenue
Bshort by 6. Revenue mismatch: report=722,313.24, actual=719,362.46, diff=2,950.78
Eshort by 1. Campaign bc816a68 channels should have equal attribution
hard25 / 26$3.20632
cohort-retention-matrix
Eshort by 1. early_churned_count mismatches
hard23 / 24$3.12583
dbt-customer-retention-risk
Eshort by 1. component_16_above_type_avg_frequency: 30 mismatches
hard17 / 18$4.01700
dbt-payment-fraud-analysis
Eshort by 1. pct_of_total_orders should sum to 1.0, got 98.18
hard17 / 18$1.714021
dbt-carrier-delivery-performance
Eshort by 1. component_14_efficiency_index_formula: efficiency index formula errors
med15 / 16$1.39299
dbt-harbor-marketing-mix-model
Eshort by 3. Should identify at least one over-saturated channel
hard38 / 41$6.841013
dbt-campaign-roi-analysis
Eshort by 1. component_8_cpa_percentile_inverted: campaign 3632acf1 with low CPA 999.00 has a low percentile
med11 / 12$5.008136
dbt-product-return-analysis
Eshort by 2. component_11_moderate_risk_tier_logic: moderate risk product 37d567d9 should be high_risk
med11 / 13$3.246425
dbt-product-return-rate-analysis
Eshort by 4. Return rate logic violated in 10 rows
med13 / 17$5.509214
dbt-fix-marketing-attribution
Eshort by 4
med9 / 13$2.11395
dbt-fulfillment-sla
Fshort by 1. NULL found in column shipping_method_name
med76 / 77$3.29449
dbt-fix-inventory-balance
Fshort by 2. Found 124 products with more than 5% revenue mismatch
med17 / 19$1.74446
dbt-supplier-payment-optimization
Fshort by 1. NULL values found in early_payment_discount_pct
v.hard7 / 8$2.133416
dbt-supplier-scorecard
Fshort by 5. Cost score mismatch for SUP0008: variance=0, expected 66.67, got 0
v.hard14 / 19$1.97577
marketing-campaigns-harbor
Gshort by 1. first_redemption_date mismatch for 007f4385: 2025-11-26 against 2025-11-26 00:32:51
hard218 / 219$3.02780
dbt-fix-multi-touch-attribution
Gshort by 1. component_3_schema: conversions must be INTEGER type
hard9 / 10$2.63444
fifo-inventory-cogs
Hshort by 11. FIFO VIOLATED: 110 allocations do not match expected overlap
hard39 / 50$5.307314
dbt-multicurrency-lifo
Hshort by 1. sale_cogs validation errors
v.hard0 / 1$5.158125
Letters mark the failure taxonomy above. “Short by” is the number of assertions between the trial and a clean pass. Assertion counts read n/a where a verifier emitted no parseable summary or no tests ran. Raw verifier output and harness config
Ready to turn maintenance into an automated habit?
Install the open-source toolkit today.
npx skills add exmergo/dex
Codex, Gemini CLI, Cursor, and others, via AGENTS.md.
