AI-Generated UI Tests in Business Central, or How to Check That "Passed" Actually Means Something

The test passed. The official bc-replay runner reports passed, and the Page Scripting pane says "Script completed without errors". But the record isn't in the database.

---

The problem and the usual approaches

These days, more and more processes are being automated with AI agents. It's becoming an essential part of how our work is changing. AI agents already write code, search for and check information, analyze an existing codebase, make plans, and draw conclusions from everything they've gathered. So a natural question comes up. How can we trust the result?

Teams want to be sure their features work. That has always been true, and for good reason. The product's reputation depends on it. It matters even more in business systems, where the customers are businesses themselves and a mistake can be expensive. And when it comes to checking features, the obvious answer is testing.

With AI agents, a developer can cover a new feature in detail with automated tests written in code. Manual QA can click through the feature and see how it works. Yes, it's slow, and you need to know exactly what was added (I describe how to make this part much better in another article). But this way we see how the system works with our own eyes. Automation QA gives you a visible result too.

Let's try to find an approach that doesn't need the statistics and metrics used to evaluate ML models, doesn't require you to know code syntax to read the tests, and doesn't make you check the feature by hand or write the same steps in code for an automated check.

Closer to the point

To trust a result, it's best to see it in a way that's simple and clear to everyone who works with the system. That means the UI.

Business Central already has a tool for replaying clicks. It's called Page Scripting. I record my clicks, the pane saves them to a YAML file, and from here on I'll call that file a plan. In CI, bc-replay runs the same plan. In the end, we get a ready-made answer.

In my approach, an AI agent writes these scenarios from start to finish. I'll cover exactly how it does that in a separate article. Here the question is whether you can take such a test's result at face value, even when official Microsoft tools run it.

I decided to check how reliable this approach is. During the experiments I saw, with my own eyes and on the same screen, both the green status "Script completed without errors" and a red BC error.

The Page Scripting pane. On the right, "Script completed without errors" and green check marks on every step. On the left, the red banner "The length of the string is 15, but it must be less than or equal to 10 characters", and BC is also asking whether to discard the unsaved changes.

Let's look at the problem cases I found and try to pick methods that can protect a test setup from false passed results. From here on, green means the test passed and red means it failed, the same colors you see in reports and CI. If you want more background on the topic, I've put it in the notes.

Three cases where the test passed but the step didn't run

I did the study on BC 28 and BC 29, on-prem and in a SaaS sandbox. I ran the plans with the official npx replay runner from the @microsoft/bc-replay package, which I'll just call the runner from here on, and with the Play button in the Page Scripting pane. Judging by the BC client code, both play the plan with the same player, so they give the same result. Fields with the AUT prefix belong to my test extension.

The first thing I had to add was a deterministic check that a new record really appeared in the database. On-prem, a separate check script with SQL access to the database did it. On SaaS, requests to the standard API did the same job. From here on I call this the independent data check, because it looks straight at the data, separately from the session that created it. I'll describe it in detail after the three cases.

Server validation banner

When the BC server rejects a value you entered, for example one that's too long for the field, it doesn't open a dialog and doesn't stop you from working with the page. Instead, a red bar with the error text appears above the form. I call it the server validation banner. A person sees it and fixes the value. A script just moves on.

The plan creates a new Customer and enters 15 characters into a Text[10] field. BC shows the red banner "The length of the string is 15, but it must be less than or equal to 10 characters" and saves the record without that value. The runner reports passed every time, with no warning at all. The player itself, if you read its result directly, records no error (error: null) and counts every step as replayed. You can see the banner in the screen recording the runner makes automatically for every run, but the field in the database is empty. The record was created, and Name was saved. Only the value that was too long never reached the database, and the "Not saved" status in the screenshot below refers to that value. The same plan with 9 characters saves the value. In the pane, the banner is on screen too, but what people read is the status, "Script completed without errors". Below is the shortened runner output for the first on-prem run.

# npx replay console
  1 passed

# results.xml (JUnit)
<testsuites … tests="1" failures="0" skipped="0" errors="0" …>

# replay-log.yml, the step with the 15 characters, no error in it
  - type: input
    target:
      - page: Customer Card
      - field: AUT Loyalty Code
    value: ARTLEN15XXXXXXX

The same plan and the same player as in npx replay, but without the final step that closes the page, because that step opens a dialog on top of the banner. The runner doesn't open the Page Scripting pane, so there's no pane on the right. The player reported no error and counted every step, even though BC shows the length banner and the status "Not saved", and the value from the Loyalty Code field never reached the database.

A filter with the value true

The player silently fails to apply a filter on a Boolean field with the value true. The list stays unfiltered, and the filter step is marked as done. In my case, the true came from the AI agent that built the plan. In my test environment, a plan filtered the Vendor List this way. The runner gave passed, and the last frame of the recording shows all six vendors.

  # the agent's plan, passed, but the filter isn't applied
  - type: input
    target:
      - page: Vendor List
      - scope: filter
        field: AUT Preferred
    value: true

A filter with the value true in the pane. The Preferred filter is added with no value, the list shows all vendors, and the pane marks the filter step green.

For the same filter the recorder writes value: 2, which is the option number in the list of values (0 empty, 1 No, 2 Yes). The pane again says "Script completed without errors", but now only one vendor is left in the list, as it should be. value: "Yes" works too. And if you change just the 2 to true in the recorder's output, the filter silently stops working again, so for a Boolean filter, write the option number, the way the recorder does.

  # the recorder's output, the filter works
  - type: input
    target:
      - page: Vendor List
        runtimeRef: b8r
      - scope: filter
        field: AUT Preferred
    value: 2            # 0 empty, 1 No, 2 Yes

A step the player doesn't understand

The player checks page and field names against the open page and fails honestly when they're wrong, for example "Field '…' was not found". But it doesn't check how the step itself is written, and it silently counts a step it doesn't understand as done. The easiest way to show this is with the type: key. Every step in a plan starts with it, and it tells the player what to do. For example, type: input means enter a value into a field. If you write step: instead of type:, the player doesn't run the step, but it doesn't fail either. The plan below creates an Item and fills in two fields. The step for the second field has the key step: instead of type:.

  - type: input
    target:
      - page: Item Card
      - field: Description
    value: =Parameters.'Description'   # value from the plan's parameters

  - step: input            # should be type:
    target:
      - page: Item Card
      - field: AUT Shelf No.
    value: =Parameters.'ShelfNo'

The Page Scripting recorder never writes it this way, and there's no official schema that would reject an unknown key. So this case and the previous one show up exactly when an agent writes the plan or someone edits it by hand. For generated plans, checking passed becomes a required part of the approach.

If the plan doesn't check the skipped step, the runner gives passed every time. The player treats the step: step as replayed, even though nothing happens in the client. The runner fails a run only when the player returned an error or replayed no steps at all, and warnings don't affect the result. So the run is green, and in the variant from the snippet above the Item exists, but the field is empty.

When every step is written this way, as in the screenshot below, no Item appears in the database. The Page Scripting pane shows this plainly, because every such step gets a green check mark. If the plan has a validate on the skipped field itself, the run turns red.

Every step of the plan is written with the key step. The pane says "Script completed without errors" and gives each step a green check mark, but the Items list filtered by the description from the plan is empty, so the Item was never created.

The player is just as quiet when it skips a step with a typo in the type (type: imput) or in the value key (vaule: instead of value:). In both cases the runner gave passed, the pane showed green check marks, and the field stayed empty. The same happens with check steps. More on that in the negative control section.

Methods that protect against a false passed

All three cases lead to one conclusion. A test result has to be confirmed by a check that doesn't depend on the session that produced it.

Independent data check

The independent data check reads the result straight from the database or through the API, not through the browser session that created it. It knows nothing about what the screen or the runner showed. Every plan writes a unique run tag into the record, for example into the customer's name, so the check looks for that exact record.

But the check itself can be wrong too. There are two basic mistakes worth guarding against from the start, and doing so barely complicates the setup.

• The check looks in the wrong place. A query that returns "0 rows" may be looking at the wrong database or company, and then "no record" proves nothing. The fix is to count known data in the company before searching for the tag. This is the positive control, and that's what it's called in the code below. If it finds nothing, the check itself is broken, and we don't trust the run.
• A mistake in the query itself. The positive control confirms the table is the right one, but it doesn't catch a mistake in the query's filter. What protects against that is a control plan, one where the data is guaranteed to be saved. For step: it's the same plan with type:, and for the banner it's the plan with 9 characters. In both cases, the check found the record with its value.

The result is the diagram below, and the check's verdict only means something at its third step.

The independent data check. If either of the first two steps fails, there's no conclusion about the run.

Back to how the check reads the data. The check itself is a script written in advance that can only read the data, and the agent can only run that script. How that access is set up depends on how your Business Central is deployed. As examples, here are the two approaches I used for the demo, SQL for on-prem and the API for a SaaS sandbox.

On-prem through SQL

In a container, the check script runs a SQL query straight against the database. In the snippet below, the first query is the positive control. It counts all the records in the table for the right company. The second query looks for the record by its run tag.

-- 1. positive control: the check can see the right company's table
SELECT COUNT(*) FROM [<company>$Customer$<base-app-id>];
-- 0 → the check is broken (wrong database / company / prefix), don't trust the run

-- 2. result check (length banner)
SELECT c.[No_], c.[Name], x.[AUT Loyalty Code$<ext-app-id>]
FROM [<company>$Customer$<base-app-id>] c
LEFT JOIN [<company>$Customer$<base-app-id>$ext] x ON x.[No_] = c.[No_]
WHERE c.[Name] = @RunTag;
-- after npx replay (passed): 1 row, AUT Loyalty Code empty

Run the check twice, once before the run and once after. Before the run, there must be no record with the tag yet, and the positive control shows how many records the table holds. After the run, the count must have grown by exactly the number of records the plan creates, and a search by tag must find exactly that record, with the value the plan entered. If there's no record with the tag, or the field is empty, but the runner gave passed, that passed is false. And if a record with the tag turns up before the run, the tag isn't unique and there's no conclusion.

SaaS through the API

For the cloud version, which Microsoft calls Business Central online, the documentation on extracting data says that "For Business Central online, the only supported option for reading data is using APIs". So on SaaS, the data check becomes a GET request to the standard API v2.0 on behalf of the test user. The first request in the snippet below is the positive control. It checks that the company returns at least one record. The second looks up the record by its run tag.

# 1. positive control: the company MUST return at least one record, otherwise the check is broken
GET https://api.businesscentral.dynamics.com/v2.0/<tenant>/<env>/api/v2.0/companies(<id>)/customers?$top=1&$select=number
Authorization: Bearer <test user's delegated token>

###
# 2. result check (length banner)
GET https://api.businesscentral.dynamics.com/v2.0/<tenant>/<env>/api/v2.0/companies(<id>)/customers?$filter=displayName eq '<run tag>'&$select=number,displayName,addressLine2
Authorization: Bearer <test user's delegated token>
# → 1 row, "addressLine2": ""   (runner: passed)

The proof here, as on-prem, is a missing record or an empty field after the positive control has passed. The check gets its token through its own app registration in Entra ID with delegated permissions, so it sees exactly what the test user sees.

The pane says "Script completed without errors" and gives all six steps a green check mark, while above the form there's the red banner "The length of the string is 60, but it must be less than or equal to 50 characters" and the status "Not saved". On top of the page, BC also shows the dialog "A modal dialog is open in another window". After this run, the API returns a new Customer with an empty addressLine2.

Pre-run plan check

By pre-run plan check I mean an automated check of the YAML plan's text without running it, the way a linter checks code. It sits in front of the runner and the pane, because neither of them checks the plan. This check stops an unknown key, a typo in the step type, typos in value: and operation:, and the value true in a filter. But it knows nothing about value length, so only the data check catches the banner. There's a minimal snippet at the end of the article.

A negative control for every validate

Not every change can be checked in the database. A lot of functionality doesn't change data at all. It only shows the data differently, like a filter on a list. For that, Page Scripting has validate, a check step you put in the plan alongside the actions. My measurements showed that validate works when it checks exactly the thing that broke. A check on the skipped field turned the step: run red ("Was expecting 'A1-01' but got ''"), and a check on the first row after the filter with the value true gave "Was expecting '20000' but got '10000'".

The same skipped step, but the plan checks the Shelf No. field with validate. The field is empty, and the pane turns red on exactly that check.

After the filter with the value true, the validate on the first row expects vendor 20000 but sees 10000. Here the pane honestly says "Script failed".

But a validate that has never failed is still a hypothesis. My measurements showed that the player silently skips a check with a typo in the type (type: validat) or in the operation: key, just as it skips actions. If you deliberately break the expected value in such a check, the run is still green, so the check might as well not be in the plan.

A check with a typo in the operation key and a deliberately wrong expected value, A1-02. The pane says "Script completed without errors", even though the last step has a red icon and the field holds A1-01.

So every such step needs a negative control. That's what I call a copy of the plan with the expected value broken on purpose. A run of that copy must fail on exactly that step and nowhere earlier. If it's green, the check doesn't work. This is the counterpart of the positive control in the data check. There we make sure the check sees known data. Here we make sure validate can fail.

  - type: validate
    target:
      - page: Item Card
      - field: AUT Shelf No.
    operation: "="
    value: =Parameters.'ShelfNo'   # A1-01
    # negative control: value: A1-02 → the run must fail on exactly this step

In my test environment I made a negative control for each of the two validate steps in the control plan (the one where the data is saved correctly) and ran each of them with the runner twice. All four runs failed on exactly the broken step, and every step before it passed. The error was the one you'd expect, "Was expecting 'A1-02' but got 'A1-01'". Meanwhile, the data check found a new Item with the correct value every time, and the same plan without the break came up green. So the red came from the check itself, not from the data. The player stops at the first red validate, so one negative control tests exactly one check.

The negative control in the pane. The expected value is deliberately changed to A1-02, the field holds the correct A1-01, and the last check step is the one that fails, while every step before it is green.

This technique has a limit. A negative control proves that the check runs and can fail, but it doesn't show whether the value depends on the behavior the requirement describes. So it's a cheap minimum. Mutating data or AL code is the stronger tool. dubcrab describes a similar technique, inverting the assertion, in Your green tests are lying, and the same author covers data mutation on his blog. LethAL does AL code mutation for BC. More on it in the notes.

How to apply this to your tests

The scripts from my test environment are tied to my setup, but the principle carries over to any Page Scripting run.

1. Check the result directly with a database query and a positive control, so the check fails when it can't see a known record.
2. Make a deliberately red run for every validate. Break the expected value, and that exact step must fail.
3. Keep the video of the green run too, next to the data check result.
4. Check the plan before running it, so an unknown key or the value true in a filter never reaches the runner or the pane.

A minimal plan check for item 4 can look like this.

// plan = the YAML plan, parsed with any parser (e.g. js-yaml)
const KNOWN = new Set(['navigate', 'invoke', 'input', 'filter', 'validate', 'close-page' /* + the types you actually use */]);
function checkPlan(plan) {
  const errs = [];
  plan.steps.forEach((s, i) => {
    if (s.type === undefined) errs.push(`step ${i + 1}: no 'type:' (keys: ${Object.keys(s).join(', ')})`);
    else if (!KNOWN.has(s.type)) errs.push(`step ${i + 1}: unknown type '${s.type}'`);
    if (s.type === 'input' && !('value' in s)) errs.push(`step ${i + 1}: input without 'value:'`);
    if (s.type === 'validate' && !('operation' in s)) errs.push(`step ${i + 1}: validate without 'operation:'`);
    for (const t of s.target ?? [])
      if (t.scope === 'filter' && ['true', 'false'].includes(String(s.value)))
        errs.push(`step ${i + 1}: the player will silently fail to apply filter '${t.field}' with the value ${s.value}; for a Boolean write Yes / No or the option number (2 = Yes)`);
  });
  return errs; // non-empty → don't start the runner
}

And write the task description you give to an agent or a person so that there's something to prove it with. It helps when the task describes

• who acts, which data and preconditions, and where it starts and ends;
• the observable outcome, meaning a document or a ledger entry. "The field was saved" isn't enough for this;
• for each requirement, whether you check the UI or the data, which data check with a positive control covers it, and where the negative controls are;
• what is out of scope.

If you want to practice and check this yourself, you could, for example, use the ready-made scenarios from the how-to articles on MS Learn and the test names in microsoft/BCApps.

What's next

A green status from the runner or the pane proves only that the player reached the end of the plan without an error. The independent data check with a positive control proves that the record really is in the database. The negative control proves that validate really checks something. The pre-run plan check proves that the plan reaches the player intact.

These methods work today on plain Page Scripting. The limit is that its player itself doesn't see BC's banners and dialogs, it doesn't check how a step is written, and you can't put a data check or a BC error check between the steps of a plan. More generally, I'd like to see something closer to how a person does the steps, as if they were walking through them in the browser themselves, with every stage and nuance. So let's try bringing in a browser automation tool. I'll share what came of it in the next article.

If you've run Page Scripting and seen green where there was no data, I'd like to hear how it went for you.

---

Notes. What Microsoft and the community already have

• AL tests don't see the UI. TestPage runs inside BC, without a browser, so real dialogs are out of its reach. The author of LethAL, a tool that deliberately breaks AL code, writes that "Tests run without a GUI, so code behind a Confirm or a dialog may be unmeasurable" (LethAL README).
• Page Scripting is meant not only for developers. Microsoft writes that "Customers, consultants, and testers can create repeatable user acceptance tests … without writing AL code" (roadmap entry RM573384).
• Page Scripting in CI. bc-replay runs the same plan in CI (MS Learn), and AL-Go for GitHub supports it starting with v6.1. For a green run, AL-Go publishes no details. "If the page scripting tests succeed - the details are not published" (AL-Go release notes).
• A plan without the recorder. Erik Hougaard built a plan from Page Metadata back in 2024 (hougaard.com), Andy Wingate generates AI variants of a recorded script (D365BC-vibe-page-scripting), and in rf9000/CreateScriptsForVideos an agent writes the plan.
• The player's silent modes. FBakkensen described them in his bc-replay format reference, for example "no error, the row silently never inserts" (format reference). Checking data outside the UI and deliberately failing a check are both known techniques on their own. My part here is only to put them together for the BC UI and measure them on the runner, the pane, and two BC versions.
• The blind spot of a UI scenario. FBakkensen writes that "bc-replay re-reads the bound Rec exactly as a TestPage does, so its oracle is blind to that fault class" (al-page-script skill). In other words, if a UI scenario re-reads the same record that TestPage does, it will miss the same bug that TestPage misses. That's one more reason to check the result outside the session that created it.
• The same thing is noticed outside BC too. ACCELQ, a test tool vendor, writes about Dynamics 365 that "A passing UI test only proves the interface behaved correctly" (ACCELQ article).

Comments

[Auten]

Disclosure first: I'm on the Auten team (we build a computer-use MCP server that records a task once and replays it), so "the replay said done but the thing didn't happen" is the failure we worry about most. This is the clearest write-up of it I've seen.

The banner case is the nasty one, because nothing in the plan is wrong. The player did exactly what it was told, and the app quietly refused one value. Your independent data check is the right ground truth. One cheaper layer that catches it earlier: after each input step, read the field back from the page and compare it to the value you sent. An empty or truncated field, or a red banner in the page text, fails the step right there instead of at the end of the run.

For the step: / imput / vaule cases, since an agent writes the plans: did you consider linting each plan against the keys the recorder actually emits before it ever reaches the runner? An unknown key is then a red build, not a silently skipped step.

And a question: in your runs, did the auto-recorded video ever show something the data check missed, or did the data check catch everything once the positive control was in place?

[Serhii Korniichuk]

Thanks, glad it was useful.

Read back after each input is a good point. That's the direction I'm taking in the next article, where a browser automation tool checks the page after every step, banner included.

On linting, yes. The article has a "Pre-run plan check" section and a minimal snippet at the end that stops an unknown key, a typo in the step type, and the value true in a filter before the runner sees the plan.

On the video, the data check covered every case where data should change. The filter case is different, because a filter doesn't change anything in the database. There the last frame of the runner's recording showed all six vendors, so you could see the filter didn't work. What catches it in the run is a validate step that checks the first row after the filter. I also give that step a negative control, so I know it can actually fail.

Curious which systems you use Auten with. Do you work with Business Central or other ERPs, or did you find the article through the general topic of replay testing?

[Auten]

Honest answer: no Business Central or other ERP work on our side. I found your article by looking for exactly this topic, replay that reports "passed" when it shouldn't, because that is the failure we care most about.

The filter case is the useful part for us. Nothing changes in the data, so a data check can't see it, and only a check on what the screen shows afterwards can. Giving that validate step a negative control so you know it can fail is a nice touch; a check that never fails is easy to ship by accident.

Looking forward to the browser follow-up, especially how you decide which banner text counts as a failure versus an info message.