UiPath Maestro Flow: Test flows with evaluations

Key insights

• An evaluation can target a whole Flow or a single node; this example tests the whole Flow.
• LLM Judge Trajectory evaluates the run's path, not the agent's actual email classification text.
• Simulating the Approve Slack Post node avoids pausing tests for a human Action Center approval.
• Expected behavior should describe observable routing and process steps the evaluator can see.
• A run can show three passed data points while its evaluation scores still reveal a behavior problem.

UiPath Maestro Flow evaluations let you rerun test cases against an entire process and check whether changes preserve its expected behavior. They matter to developers who build or maintain Flows with agent decisions, branching, and human approval steps. Anders Jensen [MVP] demonstrates the setup using an email-to-Slack Flow and tests how it behaves after changes.

What it does

An evaluation dataset pairs input examples with expected behavior and an evaluator. Running it executes the selected Flow or node against its data points and scores the results. In the demonstrated example, incoming Gmail messages go to an Email Summary Agent, then through a human approval step before a category determines the route to Slack.

The Flow has Finance, Tech, and Other cases. A dataset with examples for each case gives the developer a repeatable check to run after editing the process, rather than sending new emails and checking Slack manually. The example uses the LLM Judge Trajectory evaluator, which assesses the run as a whole based on what the process did and the route it took.

That evaluator has a meaningful boundary: it can assess the path through the Flow, but the demonstration notes that its trace does not expose the email content or the agent’s classification text. The expectation “classified as Finance” therefore asks the judge to assess something it cannot see. Describing the observable route and the Slack action gives it criteria it can check.

Who it is for and what you need

This method is for people building or testing UiPath Maestro Flows, particularly when a Flow includes agent output, branches, and approval. The example starts with an existing email-to-Slack Flow and uses its Email Received trigger as the entry point for a whole-Flow evaluation. The transcript does not state required licenses, account plans, region, or rollout status.

The example also uses Claude Code to change and repair the Flow. That is the demonstrator’s chosen way to edit it, not a requirement for creating an evaluation dataset. The transcript shows that data points can be added manually or through a guided wizard or captured from a debug run; this particular test creates generated email inputs for its three scenarios.

One practical requirement is that the dataset’s expected behavior must describe evidence available to the chosen evaluator. If the goal is to check a stable output field such as a category, the demonstration identifies Exact Match as an evaluator that could be added for that purpose. The setup shown uses only LLM Judge Trajectory, so it focuses on the route rather than directly verifying the category value.

How it works

1. In the Flow, open Evaluators and review the available options. The demonstration distinguishes Exact Match for a stable field, LLM Judge for the meaning of an output, and LLM Judge Trajectory for assessing an entire run.

2. Open Datasets, choose Add, then Add dataset. Name the dataset and select the whole Flow as the evaluation target. Choose the Email Received trigger as the entry point and add LLM Judge Trajectory as the evaluator.

3. Add a simulation for the Approve Slack Post user step. The example selects generated output and instructs it to “Approve every time.” This lets test runs pass the human approval point without waiting for someone in Action Center. Simulations are useful for steps that should not perform real work during a test.

4. Add three data points for Tech, Finance, and Other. The example uses Generate with AI to create a realistic email for each category, then supplies expected behavior for each one. Run the evaluation with Run evaluation and Save & Run.

5. Read the evaluator’s score as well as the run status. “Three passed” means the three runs completed; it does not mean the judge confirmed every expected behavior. Refine the expectation to describe visible routing—for example, whether the Flow avoids the Other branch and reaches Send Message to Channel.

6. After changing the Flow, rerun the same dataset. In the example, a new priority field is added while the category and summary are meant to remain unchanged. Deleting the approval connection then lowers the evaluation score because the Slack nodes are no longer reached; restoring the connection allows another test run.

Limitations and open questions

Evaluation scores depend on whether the judge can observe the behavior named in the expected-behavior text. A trajectory evaluator is not, by itself, proof that an agent returned the correct category or wrote an accurate summary. To test a stable category field directly, the demonstration points to adding an Exact Match evaluator; the transcript does not detail the configuration for that additional check.

The example’s three generated emails are a small set of cases, not evidence that every possible input or branch has been tested. Its results also show why status and score should not be confused: runs can complete while the judge reports a low score. The demonstration does not establish licensing, broad availability, or a release stage, so check those details separately before planning a rollout.

The core benefit is reusability: once the dataset and simulation are set up, the same scenarios can be rerun after edits to look for regressions. The material does not compare this method with a specific previous UiPath testing feature; it contrasts rerunning evaluations with manually sending emails and checking Slack.

Sources

• Anders Jensen [MVP] demonstration of UiPath Maestro Flow evaluations

Frequently asked questions

Can UiPath Maestro Flow evaluations test an entire Flow?

Yes. The demonstrated dataset targets the whole Flow and starts at its Email Received trigger. The setup also offers single-node evaluation.

How do you test a Flow that includes human approval?

Add a simulation for the approval step. In the example, the Approve Slack Post step is set to generate “Approve every time,” so the test does not wait for a human in Action Center.

Does LLM Judge Trajectory verify an agent's category?

Not in the demonstrated setup. It evaluates the visible route through the Flow, not the agent's email content or classification text; the example suggests Exact Match for a stable field such as category.

What does “passed” mean in an evaluation run?

It means the data point's run completed. Review the evaluator score too, because completed runs can still have low scores when the Flow does not follow the expected path.