Exmergo logoExmergo
All benchmarks

Published by Snowflake

data-eng-bench

Data Engineering Agent Benchmark

Snowflake's 103-task data-engineering benchmark, scored pass/fail by whether the dbt project's own pytest suite runs clean.

103 tasks1 published rundata-eng-bench on GitHub

What data-eng-bench measures

data-eng-bench drops an agent into a 2,356-model dbt project and hands it a ticket: build a new mart, extend a staging layer, or fix a model that returns the wrong numbers. Each of the 103 tasks is graded by running the task's own pytest verifier afterwards, and the reward is binary and total. A task scores only when the suite exits clean, with zero failures and zero skips. The verifiers are written as many independent assertions rather than one check, so underneath the pass/fail there is a much finer record of how close a near-miss actually was.

Latest result ·

57.3%59 / 103 tasks resolved with Claude Sonnet 5

Fifty-nine of 103 tasks scored a clean pass, with zero errored trials and zero retries. Underneath that binary reward, 2,699 of 2,850 individual verifier assertions passed, and 19 of the 44 failed tasks were blocked by exactly one assertion. The binary number is what a leaderboard records; the assertion record is where the remaining work actually is.

Snowflake published their own data-eng-bench numbers on 6 August 2026, five days before this run. Their Pass@1 is the comparable measure, and on it this run lands at the top of the Sonnet 5 group by 0.7 points, a margin with no statistical content at this sample size.

Read the full 11 August 2026 run→

Run history

Every data-eng-bench run Exmergo has published, newest first. Each run keeps its own permanent page, so earlier results stay citable.

103 tasks
57.3%59 / 103 resolved

Dex + Claude Sonnet 5 resolves 59 of 103 tasks (57.3%) on Snowflake's data-eng-bench at k=1, the highest published figure for any Sonnet 5 configuration, and passes 94.7% of the verifiers' 2,850 individual assertions.

Best run: Claude Sonnet 5 with dex

Read the run→

data-eng-bench questions

What is data-eng-bench?
data-eng-bench is Snowflake's Data Engineering Agent Benchmark: a 103-task suite that drops an agent into a 2,356-model dbt project with a ticket and asks it to build a new mart, extend a staging layer, or fix a model returning the wrong numbers. Each task is graded by running the task's own pytest verifier afterwards, and the reward is binary: a task scores only when the suite exits clean, with zero failures and zero skips.
Who publishes data-eng-bench?
Snowflake publishes and maintains data-eng-bench as an open benchmark at github.com/snowflake-labs/data-eng-bench, and published their own results on it on 6 August 2026. It is a third-party suite rather than one Exmergo authored, which is what makes results on it comparable across tools.
How does Dex perform on data-eng-bench?
In Exmergo's 11 August 2026 run, Dex with Claude Sonnet 5 resolved 59 of 103 tasks (57.3%), the highest published figure for any Sonnet 5 configuration and statistically indistinguishable from the two Snowflake published at 56.6%. The 95% confidence interval is 47.6% to 66.4%. Dex was invoked on 98% of trials, and 2,699 of the 2,850 individual verifier assertions passed.
Why is the run measured at k=1, and what does that mean for the result?
k=1 means one attempt per task across all 103 tasks, with no re-runs and no best-of selection. Snowflake's published figures are k=3, so theirs is the same measurement on three times the sample and ours is the noisier estimate. A k=1 run cannot produce a Pass^3 figure, so Exmergo publishes none. The run also has no baseline arm, so no improvement is attributed to Dex: the run measures what the combination scores, not what Dex adds.
Does the assertion-level score mean the tasks nearly passed?
It means the work was close, not that the points are recoverable. Across the 44 failed tasks, 87.6% of assertions still passed and 19 tasks were blocked by exactly one assertion, but a near-miss only converts on a re-run if its cause is random, and most of these look systematic. The assertion rate describes where the remaining work is rather than a pass rate in waiting. Where a single figure is wanted, the task-weighted 90.6% is the one to quote, because pooling overweights the one task carrying 219 assertions.
Are these results reproducible?
The run page lists the run id, the model and its reasoning effort, the agent version, the Dex commit, the base image, and the dataset digest, and it reproduces verbatim the CLAUDE.md the Dex arm added to the image. Task digests match snowflake-labs/data-eng-bench@v1.0 with nothing under tasks/ modified, and the raw per-trial verifier output and harness configuration are committed under benchmarks/data-eng-bench in the Dex repository.

Results measured by Exmergo on Snowflake’s data-eng-bench.

Exmergo logo
Exmergo