Power BI DAXBench v2 tests AI-generated DAX
DAXBench v2 checks whether AI-generated DAX works across prepared Power BI models, with 94 tasks covering five kinds of work.
Key insights
- DAXBench v2 expands from 30 tasks to 94 and covers five task types.
- Tasks include writing and fixing measures, writing UDFs and calculated columns, and optimizing measures.
- A task fails if any tested result is wrong, including at a subtotal or grand total.
- The benchmark uses one-shot responses: the model does not iterate after checking its own result.
- The benchmark keeps its task set private to reduce the chance that tasks enter model training data.
DAXBench v2 evaluates AI-generated DAX by running it against prepared Power BI models and checking whether the results are correct. It matters to Power BI developers and analysts comparing models for DAX work, but its results are not a guarantee for a different semantic model. SQLBI demonstrates how the benchmark was designed and discusses its results in a YouTube session with DAXBench creator Maxim Anatsko.
What it does
DAXBench is a public benchmark for testing how AI models handle DAX, the language used to define calculations in Power BI semantic models. Version 2 broadens the test set from 30 tasks in version 1 to 94. The tasks run against prepared models rather than treating a plausible-looking formula as proof that the answer is correct.
The benchmark has five task types: write a measure, fix a measure that returns a wrong result, write a user-defined function (UDF), write a calculated column, and fix or optimize a measure that works but is suboptimal. Measures carried over from version 1, while UDFs, calculated columns and repair or optimization tasks add new areas of coverage.
Its key distinction is executable validation. The benchmark checks results across the output, including detail rows, subtotals and grand totals at different granularities. A single failed probe means the task fails; code quality and other criteria matter only after the result passes. This gives the score a specific meaning: success on these defined tests, not proof that a formula is the only valid solution.
Who it is for and what you need
DAXBench is relevant to Power BI developers, data analysts and teams assessing AI for DAX work. It can help readers understand what a benchmark score measures: whether a model can produce a correct first response on a set of prepared tasks. It does not test an interactive workflow in which a model receives feedback and tries again.
The supplied material does not state any Microsoft license, Power BI plan, tenant setting or product prerequisite for using the benchmark. It describes DAXBench as a public leaderboard, but does not establish a Microsoft rollout stage or regional availability. DAXBench is a third-party benchmark, not a Microsoft feature.
The session discusses differences among model scores and cautions against overreading small gaps near the top. The speakers describe leading results as surprisingly high, while emphasizing that exact rank differences may not represent a meaningful difference in practical use. A benchmark score is therefore one comparison point, not a complete model-selection decision.
How it works
The benchmark gives a model a DAX task and evaluates its first answer. It does not let the model verify the output, receive a failed test result and iterate to a solution. That choice isolates initial DAX ability, but also means the test differs from workflows where AI tools can run repeated checks.
- The benchmark presents one of its private DAX tasks to a model.
- The model returns a DAX expression or other requested solution, depending on the task type.
- The expression is evaluated against a prepared Power BI model, with results checked at relevant granularities.
- If any required check fails, the task fails. If the result passes, additional criteria can assess code quality, practices or performance.
Keeping tasks private is deliberate, according to the discussion: publishing them could allow them to appear in future model training data, weakening the test. The speakers also explain why the benchmark does not treat a reference formula as the sole acceptable answer. It checks correctness first, then considers other qualities.
For Power BI teams, the practical takeaway is to distinguish drafting from validation. A model may help create DAX, but the benchmark’s strict pass condition reflects the need to test the result at totals and other relevant levels—not just inspect whether the formula looks reasonable. The discussion notes that real workflows can include iteration and extensive testing, which this one-shot evaluation intentionally excludes.
Limitations and open questions
DAXBench v2 measures performance on a defined set of prepared tasks; it does not establish that a model will be correct on every organization’s semantic model or business rules. The transcript specifically raises complications such as bidirectional filters and many-to-many relationships, which can affect results. The benchmark’s one-shot design also does not measure how well a model improves after receiving test feedback.
The supplied material does not provide enough verified detail to report model names, a complete current ranking, licensing, benchmark setup instructions or a way to run the tests locally. It also does not establish how results transfer to a specific production model. Teams should treat the leaderboard as comparative evidence and validate generated calculations in the model and business context where they will be used.
The session does not describe a setting to enable or disable in Power BI, nor a change to Microsoft product availability. It is a discussion of an external evaluation method. Its most useful contribution for practitioners is the distinction between producing DAX and demonstrating correct results under explicit tests.
Sources
Frequently asked questions
What does DAXBench v2 test?
It tests AI models on 94 DAX tasks using prepared Power BI models. Tasks cover writing and fixing measures, writing UDFs and calculated columns, and measure repair or optimization.
How does DAXBench decide whether a DAX answer is correct?
It evaluates the answer against the prepared model and checks results across relevant output granularities. A failed check means the task fails, even if other results are correct.
Does DAXBench v2 test AI models that can iterate?
No. The described evaluation is one-shot and does not allow the model to verify results and revise its answer. The session distinguishes this from iterative real-world workflows.
Does a high DAXBench score guarantee correct DAX in my Power BI model?
No. The score applies to the benchmark’s prepared tasks. The supplied material does not establish results for a particular organization’s model or business rules.