Every Release Tested by AI Before It Ships

Overview

This client builds and runs its own line-of-business applications, with heavy integration between them. Every release needed a large manual testing effort. Testers had to work out what had changed, which features that could affect, and what to retest before anything went live.

Veriland put AI agents into the release process. Before each release, agents now produce the version differences, show which user stories each change might affect, and run the regression tests. Then they test the affected journeys through the user interface, the way a tester would. Between releases, agents also check every bug raised in Azure DevOps, and confirm each fix in the running system before the ticket is closed. The client's team reviews the results and decides.

The Challenge

AI coding tools made writing code faster, but writing code was never the slow part of delivery. The queue forms where work is checked, and release testing is usually the longest part of that queue.

Manual release testing had two weak points here. It was slow, because each release needed people to retest large parts of the applications. It was also guesswork. Nobody could say with confidence which user stories a set of changes touched, so testers either retested everything or risked missing something.

What We Did

We built the release testing process around six stages.

1. Compare versions. Agents work out what changed between the current and the next release, across code and configuration.
2. Map story impact. Agents map each change to the user stories it might affect, so nobody has to guess what to retest.
3. Generate tests. Agents create or update the tests those stories need, including regression tests across integrations.
4. Run the suite. The regression suite runs before every release, and failures come back with the likely cause.
5. Test the UI. Agents open the applications and work through the affected user journeys as a user would: clicking, entering data and moving between screens. They check what appears at each step, including data that arrives through integrations, and record screenshots.
6. Release decision. The client's team reviews the results and the story impact report, and decides whether the release goes.

Bug Verification From Azure DevOps

A bug marked resolved is not always fixed in the system, so agents check every one.

1. Read the ticket. Agents read the bug in Azure DevOps: the steps to reproduce, the expected behaviour, attachments and linked user stories.
2. Find the bug. Agents follow those steps in the system to confirm the bug exists, and record what they see.
3. Trace the cause. Agents link the failure to the change, configuration or data behind it, and add their findings to the ticket.
4. Verify the fix. Once a fix is deployed, agents repeat the original steps in the system to check the bug is really resolved.
5. Close or reopen. The work item gets the steps, screenshots and result. The client's team closes it, or reopens it with the evidence attached.

We keep the detail of the agent design and tooling for conversations under NDA.

Where People Stay in Control

• A person decides every release. Agents supply the evidence.
• The story impact report shows exactly which changes were tested and why.
• Failures are reported with the change and the story they relate to.
• Every UI test run comes with screenshots and a step log, so the team can see exactly what the agent saw.
• A bug is only reported as fixed once an agent has repeated the original steps and seen the correct result. People close the ticket.
• Test results feed into the client's change approval process.

The Results

• Every release is tested by AI before it goes live.
• 15+ testing roles are no longer needed.
• Per-story impact. Each release comes with a report of the user stories its changes might affect.
• Tested through the UI. The affected user journeys are tested on screen before every release, not only through code-level tests.
• Every bug fix verified. Each Azure DevOps bug is reproduced and its fix checked in the running system before the ticket closes.

Could This Work for You?

The approach fits any team that ships its own applications on a regular release cycle. It also applies to Dynamics 365 F&O, where our FinOps Workbench reads the AOT so agents can see what each update changes. Agentic Software Delivery explains how we run it.