A mid-sized insurance carrier was migrating its decades-old policy administration system to a modern, cloud-based platform. The migration put thousands of business rules, hundreds of workflows, and millions of customer records at risk. Manual testing could not keep pace. By adopting BotGauge, the team cut its regression cycle from 14 days to 2 days, expanded test coverage by 4x, and completed the migration three months ahead of schedule.
About the Company
The carrier sells auto, home, and small-business insurance across multiple regions. Its policy administration system had grown over more than twenty years. It handled quoting, underwriting, policy issuance, endorsements, renewals, billing, and claims intake.
Like many established insurers, the company ran on a patchwork of old and new technology:
- A core system built on legacy technology that few engineers fully understood
- Several newer web portals for agents and customers
- Third-party integrations for credit checks, payment processing, and document generation
- Region-specific rules that changed every year as regulations evolved
The business decided to modernize. The goal was a cloud-native platform that would let the company launch new products faster and give agents and customers a better digital experience.
The Challenge
A migration with almost no margin for error
In insurance, a software bug is rarely just a bug. A rating error can mean a customer is overcharged or the carrier underprices risk. A broken renewal workflow can lapse thousands of policies. A faulty document template can create regulatory exposure.
The QA lead described the situation this way: "We were not testing screens. We were testing promises we had made to customers."
The numbers behind the pain
| Problem | Impact |
|---|---|
| Full regression cycle | 14 days of manual effort |
| Business rules to validate | 3,000+ across products and regions |
| Test scenarios documented | ~900, with an estimated 2,500+ needed |
| QA team size | 9 testers, plus borrowed business analysts |
| Release cadence | Once per quarter, to avoid repeating the regression cycle |
| Production defects per release | 20 to 30, mostly in edge cases |
Why the old approach was failing
1. Coverage was thin where it mattered most.
The team could realistically execute only a fraction of the scenarios required. Testers prioritized common paths such as a standard auto policy for a standard driver. Edge cases, like a multi-vehicle policy with a mid-term endorsement and a payment plan change, were rarely tested.
2. Knowledge lived in people's heads.
Several critical business rules were documented nowhere except in the memory of senior analysts. When one of them went on leave, test quality visibly dropped.
3. Previous automation attempts had collapsed.
Two years earlier, the company had invested in a script-based automation framework. It required specialized engineers, and the scripts broke whenever the interface changed. Within eighteen months, most of the automated suite was abandoned because maintaining it cost more than running the tests by hand.
4. The migration multiplied the workload.
Every module moved to the new platform had to be tested twice: once to confirm the new system behaved like the old one, and again to confirm it still behaved correctly after each change. The testing backlog was growing faster than the team could clear it.
The Search for a Different Approach
Leadership set three requirements for any testing solution:
- Business analysts must be able to contribute. The people who understood the insurance rules were not programmers.
- Tests must survive change. A migration changes screens, labels, and flows constantly.
- Results must be auditable. Regulators can ask for evidence that critical workflows were validated.
After evaluating several options, the team chose BotGauge. Three things stood out during the pilot:
- Test scenarios could be written in plain English, which meant analysts could author them directly
- Self-healing behavior kept tests running when interface elements moved or were renamed
- Every execution produced clear, traceable results suitable for audit review
The Solution: A Phased Rollout
The team deliberately avoided a "big bang" automation project. Instead, they rolled out BotGauge in four phases.
Phase 1: Prove the value (Weeks 1 to 3)
The team picked one high-risk workflow, mid-term policy endorsements, and automated it end to end. Analysts described each scenario in natural language, for example:
Scenario: Add a second vehicle to an active auto policy mid-term
Given an active auto policy with one vehicle and monthly billing
When the agent adds a second vehicle effective 15 days from today
Then the premium is recalculated on a prorated basis
And the billing schedule reflects the updated installment amount
And an endorsement document is generated and attached to the policy
Within three weeks, the team had 60 endorsement scenarios running automatically, including edge cases the manual team had never had time to cover. The pilot also exposed 11 defects in the new platform that had gone unnoticed.
Phase 2: Capture tribal knowledge (Weeks 4 to 8)
Senior analysts ran working sessions in which they talked through business rules while the QA team converted them into BotGauge scenarios. This did something unexpected: it became the first complete, executable documentation of how the company's rules worked.
"We finally had a living record of our own business logic. New hires could read the scenarios and understand the product in days instead of months." (QA Lead)
Phase 3: Scale across products and regions (Weeks 9 to 20)
With the pattern proven, the team expanded to quoting, underwriting, renewals, and billing. Region-specific rules were handled by parameterizing scenarios, so one scenario could run against many regional rule sets instead of being rewritten each time.
By the end of this phase:
- 2,600+ scenarios were automated
- The full regression suite ran overnight
- Analysts, not just QA engineers, were authoring and reviewing tests
Phase 4: Embed in the delivery pipeline (Weeks 21 to 26)
The team connected BotGauge to its CI/CD pipeline so a subset of critical tests ran on every build, and the full suite ran nightly. Developers now got feedback within hours instead of weeks.
Implementation Timeline
| Phase | Duration | Key Outcome |
|---|---|---|
| Pilot (endorsements) | 3 weeks | 60 scenarios, 11 defects found |
| Knowledge capture | 5 weeks | Business rules documented as executable tests |
| Scale-out | 12 weeks | 2,600+ scenarios automated |
| Pipeline integration | 6 weeks | Tests run on every build and nightly |
The Results
Before and after
| Metric | Before BotGauge | After BotGauge |
|---|---|---|
| Full regression cycle | 14 days | 2 days |
| Automated scenarios | ~0 (abandoned suite) | 2,600+ |
| Business rules covered | ~35% | ~90% |
| Release cadence | Quarterly | Monthly |
| Production defects per release | 20 to 30 | 5 to 8 |
| Audit evidence preparation | 3 weeks | 3 days |
| Migration completion | Projected 15 months | 12 months |
What the business noticed
- The migration finished three months early. Faster feedback removed the testing bottleneck that had been the biggest schedule risk.
- Production incidents fell by roughly 70%. Fewer post-release fixes meant fewer escalations, fewer customer complaints, and fewer emergency weekends.
- Audit readiness improved sharply. When regulators asked for proof that renewal and billing workflows had been validated, the team produced traceable execution records in days instead of weeks.
- Testers moved up the value chain. Instead of repeating the same manual steps, the QA team focused on exploratory testing, risk analysis, and designing smarter scenarios.
What Went Wrong Along the Way
No honest case study is complete without the bumps.
Early scenarios were too vague. In the first few weeks, some plain-English scenarios were written loosely, which produced ambiguous results. The team introduced a short review checklist: every scenario needed a clear starting state, one action, and measurable outcomes.
Test data became a bottleneck. Insurance scenarios need realistic policies, drivers, and properties. The team built a managed library of test data profiles, which reduced setup time and improved consistency.
Not everyone was convinced at first. A few senior testers worried automation would replace them. Leadership addressed this directly: the goal was to eliminate repetitive work, not roles. Within six months, those same testers were leading scenario design reviews.
Lessons Learned
- Start with your riskiest workflow, not your easiest. A visible early win on a painful problem builds credibility fast.
- Let domain experts write tests. When the people who understand the rules can express them directly, quality and coverage rise together.
- Treat tests as documentation. Executable scenarios stay current in a way that written specs never do.
- Design for change. In a migration, the interface will shift weekly. Self-healing is the difference between a suite that lasts and one that gets abandoned.
- Invest in test data early. Reliable data is as important as reliable scripts.
- Measure what leadership cares about. Regression time, production defects, and audit readiness told the story more clearly than raw test counts.
Key Takeaways
- Legacy modernization is as much a testing challenge as an engineering one
- Natural-language test creation lets business experts contribute directly
- Self-healing automation keeps suites alive through constant change
- Faster feedback turns testing from a bottleneck into an accelerator
Ready to Modernize with Confidence?
If your team is facing a migration, a growing backlog, or a regression cycle that slows every release, see how BotGauge can help you test faster and ship with confidence.
Top comments (0)