DEV Community

Sandro
Sandro

Posted on

How Bug Hunter Evolved From a Test Runner Into a Self-Hosted Quality Intelligence Platform

#sysops #testing #automation

A few months ago, I wrote about Bug Hunter as a self-hosted testing tool for web applications and APIs.

At that point, the project could already:

  • discover application routes,
  • execute Playwright browser tests,
  • run direct API scenarios,
  • import OpenAPI definitions,
  • collect runtime evidence,
  • detect deterministic findings,
  • reproduce supported failures,
  • and generate reports.

Since then, the project has changed significantly.

The biggest change is not one individual feature.

It is the direction of the product itself.

Universal Bug Hunter is no longer something I think of as just a test runner.

It is becoming a self-hosted Quality Intelligence platform.

The current goal is:

Discover application behavior, generate and execute tests, detect regressions, explain failures, reduce test maintenance, and track product quality over time.

And importantly, the core workflow still works without an LLM.

AI is optional assistance, not the execution engine.

The workflow is now much broader

The current operator flow looks more like this:

Analyze application
→ build Application Map
→ create deterministic Test Plan
→ optionally generate AI suggestions
→ run tests
→ collect evidence
→ detect Quality Issues
→ qualify results
→ compare regressions
→ track reliability, coverage and performance
Enter fullscreen mode Exit fullscreen mode

That is a very different product from the early version I shared.

The focus has moved from:

"Run a scenario and detect a failure"

toward:

"What changed in product quality, why did it change, and what should the operator investigate next?"

Reliability became a first-class concept

One of the first major improvements was reliability tracking.

A failed test is not automatically treated as a confirmed product bug.

The conceptual flow now looks more like this:

detected signal
→ Needs review
→ expected
   ignored
   observation
   quality_issue
→ regression evaluation where applicable
Enter fullscreen mode Exit fullscreen mode

This distinction matters.

If a test produces seven signals, that does not automatically mean the product has seven bugs.

Some signals may be browser noise.

Some may be known behavior.

Some may be expected.

Some may become a confirmed quality_issue only after review.

That model makes the platform much more useful for real engineering workflows.

Self-healing and likely-cause analysis

Test maintenance is another problem I wanted to address.

UI automation can become expensive when selectors or page structures change frequently.

Universal Bug Hunter now includes a self-healing layer that can help identify alternative targets when an existing interaction no longer matches the application.

The important rule is that healing is not silent magic.

The system preserves evidence and keeps the behavior explainable.

The same principle applies to failure analysis.

Instead of simply showing:

Test failed
Enter fullscreen mode Exit fullscreen mode

the platform can correlate evidence and surface a likely cause when there is enough information to support it.

For example:

Browser action
→ network request
→ HTTP 500
→ UI failed to update
→ likely cause: backend request failure
Enter fullscreen mode Exit fullscreen mode

I deliberately avoid labeling every console or network event as a root cause.

Evidence and interpretation remain separate.

Visual regression testing

Visual testing is now part of the platform as well.

The system can work with baseline screenshots and compare them with current execution results.

The workflow includes:

baseline
→ current screenshot
→ visual comparison
→ threshold evaluation
→ regression result
Enter fullscreen mode Exit fullscreen mode

This is useful for catching changes that functional assertions may miss.

Examples include:

  • layout shifts,
  • missing elements,
  • unexpected responsive changes,
  • visual regressions after frontend changes.

The goal is not to replace visual review.

The goal is to make visual changes measurable and reproducible.

Accessibility testing

Accessibility testing was added as another quality signal.

The platform can detect WCAG-related issues and associate them with the same project/run context as functional, visual and API testing.

This is important to me because I do not want Universal Bug Hunter to become a collection of disconnected testing tools.

Accessibility findings should live in the same quality model as other product evidence.

Project Knowledge

Another large change was adding Project Knowledge.

Applications often cannot be understood only from routes and DOM elements.

Business requirements matter.

Product documentation matters.

Rules matter.

Universal Bug Hunter can now ingest project documentation and retrieve relevant context for AI-assisted exploration.

But there is an important security boundary:

Project documentation is context, not permission.

A document cannot:

  • expand allowed routes,
  • enable destructive actions,
  • override safety policy,
  • introduce credentials,
  • or bypass human approval.

The system treats retrieved project knowledge as untrusted reference material.

That distinction is especially important when AI is involved.

API Intelligence moved beyond basic smoke tests

The earlier version supported simple API smoke scenarios.

That area has expanded significantly.

The platform now has a stronger API testing model around:

  • OpenAPI contracts,
  • request and response validation,
  • parameters,
  • authentication profiles,
  • extracted variables,
  • request chaining,
  • assertions,
  • negative cases,
  • and boundary testing.

The intention is to keep browser and API testing connected to the same product quality model rather than building a separate API-testing product inside the project.

Test Impact Analysis and CI/CD integration

Running every test for every code change does not always make sense.

Universal Bug Hunter now supports Test Impact Analysis.

Conceptually:

Git change
→ changed files
→ affected application areas
→ affected tests
→ recommended test set
Enter fullscreen mode Exit fullscreen mode

A critical design rule here is uncertainty handling.

If the platform cannot confidently determine which tests are affected, it must not silently select zero tests.

Instead, it falls back to a defined policy such as:

smoke
critical
all
Enter fullscreen mode Exit fullscreen mode

smoke is the default fallback, but the policy can also be configured to use critical or all.

The important rule is that uncertainty must not silently produce an empty test set.

This makes the system usable in CI without creating false confidence.

Quality Intelligence Dashboard

The project now tracks quality across runs instead of looking only at one execution.

The Quality area includes concepts such as:

Quality Issues
Regressions
Reliability
Coverage
Performance
Enter fullscreen mode Exit fullscreen mode

The dashboard intentionally avoids vanity metrics.

If the system cannot calculate a trustworthy percentage, I prefer showing counts instead of inventing a percentage.

Similarly:

Regression not evaluated
Enter fullscreen mode Exit fullscreen mode

is represented as:

—
Enter fullscreen mode Exit fullscreen mode

not:

0
Enter fullscreen mode Exit fullscreen mode

Those details sound small, but they matter when people start making engineering decisions from dashboards.

Performance Intelligence

Performance testing has also been integrated into the same workflow.

The platform can execute performance measurements and compare them against approved baselines.

The important part is that performance semantics remain separate from functional execution.

A run can be:

functionally successful
Enter fullscreen mode Exit fullscreen mode

while performance regression status is:

failed
Enter fullscreen mode Exit fullscreen mode

or:

not evaluated
Enter fullscreen mode Exit fullscreen mode

These are not the same outcome.

The platform keeps:

execution status
performance verdict
release gate
Enter fullscreen mode Exit fullscreen mode

as separate concepts.

Another safety rule is that performance measurement does not re-run destructive browser workflows.

Repeated API measurements are constrained to safe, read-only request patterns.

The enterprise layer

One of the largest recent changes was Phase 35: the team and enterprise platform.

Universal Bug Hunter now has an organization-based tenant model.

The structure is roughly:

User
→ Organization Membership
→ Team / Project Grants
→ Capability-based permissions
Enter fullscreen mode Exit fullscreen mode

A user can belong to multiple organizations.

Teams belong to one organization.

A normal member does not automatically receive access to every project.

Authorization is evaluated through a central access-control service.

I intentionally avoided spreading checks such as:

if (user.role === "admin")
Enter fullscreen mode Exit fullscreen mode

through route handlers.

Roles are labels over capabilities.

The permission engine remains the authority.

Persistent collaborative triage

Quality triage is now shared and persistent.

When one authorized team member classifies a detected signal, that decision is stored as platform state.

The supported classifications are:

expected
ignored
observation
quality_issue
Enter fullscreen mode Exit fullscreen mode

The decision includes revision tracking so concurrent edits can produce a conflict instead of silently overwriting another user's work.

Confirmed quality_issue classifications can then appear in the Quality Issues view without changing the original run result.

That separation is intentional.

Platform audit log

Enterprise operations also need accountability.

The platform now has an append-only application-level audit log for governance events such as:

  • membership changes,
  • teams,
  • project grants,
  • invitations,
  • triage,
  • service accounts,
  • API tokens,
  • OIDC configuration,
  • and other administrative operations.

The audit log is tenant-scoped and redacted.

I also try to be precise about what it is not.

It is not currently a cryptographically immutable WORM audit system.

Service accounts and API tokens

CI/CD workflows should not depend on human accounts.

Universal Bug Hunter now supports service accounts with scoped Bearer tokens.

The security model is:

service-account permissions
∩
token scopes
=
effective token authority
Enter fullscreen mode Exit fullscreen mode

A token can reduce authority.

It cannot increase it.

Raw tokens are shown once.

Only a hash is stored.

Revocation and expiry are checked from the database.

An invalid Bearer token also does not silently fall back to a valid browser cookie.

Machine identity and human identity remain separate.

OIDC SSO

OIDC support was added as an enterprise SSO extension point.

The flow includes:

authorization code
PKCE S256
state
nonce
JWKS signature validation
issuer validation
local durable session
Enter fullscreen mode Exit fullscreen mode

One of the most important rules is:

Identity provider claims authenticate the user. They do not grant Bug Hunter permissions.

If an IdP sends something like:

{
  "roles": ["admin"]
}
Enter fullscreen mode Exit fullscreen mode

that does not make the user an administrator inside Universal Bug Hunter.

Authorization remains local and capability-based.

Account linking is also based on trusted OIDC identity:

issuer + subject
Enter fullscreen mode Exit fullscreen mode

not simply matching an email address.

Production migration safety

After building the enterprise control plane, the next question became:

Can this system actually be operated safely in production?

That started Phase 36.

Database upgrades are now treated as an operational contract.

The platform schema is currently v9.

The migration system includes:

preflight
→ backup acknowledgement
→ migration lock
→ durable migration marker
→ stepwise transaction
→ schema version update
→ startup
Enter fullscreen mode Exit fullscreen mode

A production database newer than the binary is rejected.

No downgrade is attempted.

A failed migration cannot falsely advance the schema version.

Only one migrator is allowed at a time.

If a migration is interrupted, the next startup fails closed instead of pretending everything is healthy.

Production backup and restore

The latest production-hardening work has focused on backup, restore and disaster recovery.

This implementation is currently under review before being merged into the main branch.

The first production model is intentionally conservative:

Backup and restore are offline maintenance operations.

I chose this instead of pretending the current SQLite architecture supports online distributed snapshots.

A backup includes:

platform database
project data
run artifacts
manifest
checksums
Enter fullscreen mode Exit fullscreen mode

SQLite is copied using the SQLite backup API rather than copying a live WAL database file.

Included backup files are covered by SHA-256 integrity metadata, together with a deterministic artifact-tree digest.

The backup format is versioned independently from the platform schema.

Restore is also a security operation

A restore is not just copying old data back.

It can also restore old credential state.

Imagine this sequence:

T0: backup
T1: API token revoked
T2: restore T0
Enter fullscreen mode Exit fullscreen mode

Without additional protection, the old token could become valid again.

Universal Bug Hunter therefore applies a post-restore credential safety policy.

By default:

human sessions      → invalidated
OIDC transactions   → invalidated
pending invitations → revoked
API tokens          → invalidated
Enter fullscreen mode Exit fullscreen mode

API tokens can only be preserved through an explicit high-risk operator override.

Human sessions are always invalidated.

Crash-safe restore boundary

Database and artifact directories cannot be atomically replaced together across all environments.

Instead of pretending otherwise, restore uses a durable state marker.

The flow looks roughly like:

validate backup
→ verify integrity
→ stage database
→ stage artifacts
→ apply credential safety
→ write restore marker
→ rotate database
→ rotate projects
→ finalize
→ clear marker
Enter fullscreen mode Exit fullscreen mode

If the process crashes after the database is swapped but before project artifacts are swapped, the marker remains.

On the next startup:

restore_interrupted
Enter fullscreen mode Exit fullscreen mode

blocks normal API startup.

The server does not listen until an operator resolves the recovery state.

That is much safer than starting with a database from one recovery point and artifacts from another.

Integrity is not authenticity

This was another important distinction.

The backup uses SHA-256 checksums.

That gives integrity checking against corruption or mismatched files.

It does not mean the backup is cryptographically authenticated.

If an attacker can modify:

payload
+
manifest
+
checksums
Enter fullscreen mode Exit fullscreen mode

they can potentially produce a different internally consistent backup.

Signed backups are not part of the current format.

I prefer documenting that limitation rather than implying stronger security than the system actually provides.

Backups are still sensitive data

Another thing I wanted to make explicit is that a backup is not equivalent to a secret-free archive.

Plaintext provider secrets and environment secret values are excluded, but the platform database can still contain security-sensitive material such as:

password hashes
token hashes or digests
identity metadata
session metadata
encrypted application credential material
customer and project data
Enter fullscreen mode Exit fullscreen mode

That means backup files should still be protected with restrictive filesystem permissions and treated as sensitive operational data.

The current implementation uses restrictive permissions where the platform supports them.

Current production topology

Another thing I wanted to make explicit is the supported deployment topology.

Today the honest supported model is:

1 API writer instance
+ optional stateless web instances
+ SQLite platform database
+ shared project/artifact storage
+ reverse proxy / TLS
Enter fullscreen mode Exit fullscreen mode

The project does not currently claim:

active/active API HA
multi-writer SQLite
distributed job execution
Enter fullscreen mode Exit fullscreen mode

Those may be future directions, but they are not things I want to pretend already work.

Where the project is now

Universal Bug Hunter has moved a long way from the version I originally shared.

The platform now covers several layers:

Application discovery
Test planning
Browser testing
API testing
Visual testing
Accessibility
Quality triage
Regression tracking
Reliability
Performance
CI impact analysis
Project knowledge
Teams and RBAC
Audit
Service accounts
OIDC
Migration safety
Backup and restore
Enter fullscreen mode Exit fullscreen mode

Some of the latest backup and restore work is still going through final review before merge, but the broader architecture is already clear.

The design principle has stayed surprisingly consistent:

Automation should reduce uncertainty, not hide it.

If the system is unsure which tests are affected, it should say so.

If a regression was not evaluated, it should not show zero.

If a backup is not authenticated, it should not claim authenticity.

If the architecture is single-writer, it should not advertise high availability.

If AI generates something, that does not mean the system should trust or execute it automatically.

What comes next

The current production-hardening roadmap continues with:

production configuration validation
health and readiness hardening
operational logging and observability
reference production deployment
multi-instance safety review
operator recovery drills
Enter fullscreen mode Exit fullscreen mode

The platform already has basic health and readiness endpoints.

The next step is to make their production semantics more explicit and useful for operators and orchestrators.

A canonical production deployment path is also still planned.

Local deployment already exists, but I want one clearly documented production reference stack instead of treating development deployment as production guidance.

There are also features I am intentionally not rushing into yet:

SCIM
SAML
billing/licensing
multi-IdP
PostgreSQL migration
full active/active HA
Enter fullscreen mode Exit fullscreen mode

I would rather finish the operational foundation properly before adding more enterprise surface area.

Final thought

Building this project has changed how I think about testing infrastructure.

The difficult part is rarely:

Can Playwright click this button?

The more interesting questions are:

Can I trust this result?

Can another engineer reproduce it?

What evidence supports the conclusion?

What happens when two people change the same state?

What happens when the database upgrade crashes?

What happens when a backup restores old credentials?

What happens when the system itself is uncertain?

Those questions are what pushed Universal Bug Hunter from a collection of automated tests toward a broader Quality Intelligence platform.

The project is still evolving, but the direction is much clearer now.

If you work in QA, DevOps, platform engineering or test infrastructure, I would be interested in feedback on one question in particular:

What would a self-hosted quality platform need to replace the collection of disconnected testing dashboards your team uses today?

Top comments (0)