Power BI DAX LLM Benchmark v2 expands its tests

Key insights

• DAXBench v2 expands from 30 tasks to 94 and evaluates 77 models.
• The benchmark tests creating, correcting and optimizing DAX, including calculated columns and tables.
• Solutions are executed in prepared Power BI models and checked for results and structural requirements.
• A task passes only when all required checks are met; a nearly correct result is not a partial pass.
• The benchmark is not an official Microsoft benchmark, according to the available research notes.

DAXBench v2 expands the tests used to evaluate whether large language models can generate, correct and optimize DAX that works in Power BI models. It matters to Power BI developers and BI teams assessing AI assistance for DAX work, but benchmark results do not guarantee that a model will solve a particular organization’s problems. SQLBI’s short post points readers to the benchmark, while the available research notes describe its expanded task set and evaluation approach.

What's new

DAX is the formula language used for measures, calculated columns and calculated tables in Power BI. DAXBench v2 tests more than whether a model can produce a plausible-looking formula: it evaluates generated code by running it against prepared Power BI data models. That distinction matters because code can be syntactically valid and still return incorrect results under the model’s relationships or filter context.

The research notes describe an expansion from 30 tasks to 94, with 77 models evaluated. The task types now include creating measures as well as correcting and optimizing existing DAX. The set also covers calculated columns, calculated tables and user-defined functions, along with scenarios involving many-to-many and inactive relationships, semi-additive measures, customer analysis and window functions.

These examples target concepts that make DAX work different from simply writing a formula in a general-purpose language. Filter context affects how a measure is calculated, while table relationships determine how filters and values move through a model. A model may produce an answer that reads well but handles totals, relationships or context incorrectly.

Who it is for and what you need

DAXBench is most relevant to Power BI developers, BI teams and people comparing LLMs for DAX-related assistance. It can offer a common set of tasks for evaluating models, but the benchmark’s results should be treated as test results rather than proof that a model is ready to write production DAX without review.

The available material does not state that DAXBench is a Microsoft product or an official Microsoft benchmark. The research notes describe it as developed by Maxim Anatsko in collaboration with SQLBI and say it uses Power BI data models to run and evaluate generated solutions. The SQLBI post itself directs readers to DAXBench.com.

The source material does not specify a required Power BI license, a setup procedure for running the benchmark, or any tenant or regional availability. It also does not establish that the benchmark is integrated into Power BI, Copilot or Microsoft Fabric. Teams should not infer product availability or a supported Microsoft workflow from benchmark results alone.

How it works

The evaluation is designed to check working DAX against the expected behavior of a model, rather than score a response solely by comparing its text with a reference formula. The research notes describe an all-or-nothing pass: a task counts as correct only if all required checks and structural conditions are satisfied. Complexity also affects task weighting, and solutions may receive a quality bonus for efficiency and clear structure.

1. A task describes a DAX requirement tied to a prepared data model.
2. An LLM generates or modifies DAX for that task.
3. The proposed code is executed against the model, so the evaluation can inspect its results as well as its structure.
4. The benchmark checks whether the required conditions are met; a task that misses a required check does not count as a full pass.

The method considers syntax, calculated results and structural requirements. It also considers qualities such as efficiency, readability and task complexity. A partial count of passed checks can help diagnose how close a response came, but it does not turn an incomplete solution into a passed task.

For a Power BI team, the practical takeaway is to look beyond a fluent explanation or a formula that appears reasonable. DAX behavior depends on the specific model and evaluation context. Even a strong benchmark score cannot verify an organization’s own business definitions, unusual relationships or incomplete requirements unless those are represented in the task and model being tested.

Limitations and open questions

The benchmark is a controlled comparison, not a substitute for testing generated formulas in the semantic model and business context where they will be used. The research notes caution that benchmark performance does not guarantee reliable results for organization-specific models, unusual business rules or underspecified requests. Human review remains relevant when a DAX result affects reporting or decisions.

The available material does not provide the benchmark’s full task definitions, the complete model configurations or details about how to reproduce each score. It also does not explain how frequently the model list or tasks are updated. Those details matter when comparing results over time or deciding how closely a benchmark task matches an organization’s own DAX workload.

SQLBI’s post description asks which LLMs are ready to write DAX and points to DAXBench. The research notes give additional benchmark details, but the supplied source list does not include a direct DAXBench page or an official Microsoft reference. Accordingly, this article does not present model rankings as verified Microsoft guidance or as a recommendation to select a particular LLM.

Sources

• SQLBI’s DAX LLM Benchmark v2.0 post
• SQLBI website

Frequently asked questions

What does DAXBench v2 test?

It evaluates LLMs on DAX tasks such as creating, correcting and optimizing code, including measures, calculated columns and calculated tables. The research notes describe execution against prepared Power BI models.

Does DAXBench v2 only check whether generated DAX is valid?

No. The described method runs solutions against models and checks results and structural requirements as well as syntax. A task counts as passed only when all required checks are met.

Is DAXBench an official Microsoft benchmark?

The available research notes say it is not an official Microsoft benchmark and describe it as developed by Maxim Anatsko in collaboration with SQLBI.

Does a high DAXBench score guarantee good results in my Power BI model?

No. The notes caution that benchmark results do not guarantee performance on organization-specific models, unusual business rules or incomplete requirements.