For the first few years I worked with this application, quality could feel like a permanent firefight.
A customer would find something. We would stop, triage it, reproduce it, trace it, get it fixed, verify the change, and move on. Then another issue would surface somewhere else. Sometimes fixing one problem exposed another. We were making progress one defect at a time, but the larger condition of the system was not really changing.
That was the part that bothered me.
The problem was not that we had a giant backlog sitting in front of us. We did not even know where all of the defects were yet.
We had a discovery problem.
Years of behavior had accumulated inside the application, and many of the defects costing the company time were still hidden inside normal workflows. They were waiting for the right person to click the right thing, enter the right data, or hit the right condition.
That meant fixing the next bug was necessary, but it was not enough.
At some point I realized the little moves were never going to change the trajectory. I needed a broad stroke. One mechanism that could reach across the application, expose what we could not yet see, and help us work through years of accumulated defects at an accelerated pace.
I needed an engine for discovery.
The system was already telling us where it hurt
Once I started looking at the problem that way, one part of the answer was sitting in plain sight.
I could test the application deeply, but I could never use it exactly the way the entire company used it.
Customer support moved through one set of workflows. Engineers touched another. Operations had their own routines. Other employees reached parts of the product I might never naturally encounter during a planned test session.
Every one of those people was already exercising the system every day.
They were generating quality signal without calling it quality signal.
The problem was that most of those observations died where they happened. Someone would see something strange, think somebody should probably know about it, and then continue with the work in front of them.
So the design question became much more interesting than, "How do I find more bugs?"
How do you turn normal product use into meaningful defect discovery without asking the whole company to become testers?
Designing the engine
The answer I came up with was an internal bug bounty program.
At a high level, the idea was simple. If someone encountered a legitimate defect while doing their normal work, they could submit it through a lightweight process. I would review the report, filter duplicates and noise, and move valid findings into the defect workflow.
But the idea only worked if people actually participated.
That was the first real design constraint.
People were already busy. Finding a bug did not mean they were going to stop what they were doing, document it properly, and send it to us. Reporting the problem cost attention, and attention is expensive inside a working company.
So I had to think about the incentive almost as carefully as I thought about the defect process.
I wanted the reward to be meaningful enough that someone would pause and capture what they found, but I did not want to create a second administrative problem just to solve the first one. Anyone who has ever tried to move a small incentive through normal HR and payroll channels knows how quickly a simple idea can collect red tape.
We found a cleaner path through an existing recognition mechanism. The bounty used a tiered structure so contributions could be recognized according to their value without turning the program into something complicated.
That detail mattered because the system had to feel easy from the participant's side.
Find something real. Capture it. Submit it. Let the quality process take it from there.
There were rules and guardrails behind that simplicity. They protected the signal from duplicates, noise, and submissions that were outside the purpose of the program. I do not need to unpack every one of those mechanics here, but they mattered. The program had to invite participation without letting participation overwhelm the process.
What made the design powerful was that nobody had to change how they worked.
A support representative did not need to pretend to be a tester. An engineer did not need a special exploratory session. People simply used the application the way they already did, and now there was a reason and a path to surface the problems they encountered.
The peer-to-peer recognition element became a useful cultural layer on top of that. It was not the solution itself. It was an extra way to make good contributions visible and let quality participation feel shared instead of hidden.
The core mechanism was still the feedback loop.
As the program moved from idea into execution, the scope became real very quickly.
As the scope became clear, I brought in my colleague Ryan Smith. I knew Ryan's diligence and reliability were something I could count on.
When better visibility looked like worse quality
Then the submissions started coming. A lot of them.
For a moment, the numbers could tell the wrong story if you looked at them without context. The defect count went up sharply.
But we were not suddenly creating more bugs. We were finally exposing bugs that had been there all along.
That is a very different thing.
Better visibility can make an unhealthy system look worse before it makes the system better. The graph was not showing quality collapsing. It was showing the discovery engine gaining traction.
Finding more defects, however, was only useful if the loop closed behind them.
Discover. Validate. Resolve. Verify. Repeat.
Again and again.
That cycle became the broad stroke I had been looking for. Instead of waiting for defects to emerge one customer call at a time, we had created a mechanism that let many people expose them through real use while we continuously moved them toward resolution.
The goal was to create an engine that could work through years of accumulated defects at an accelerated pace. And the first graph shows exactly what that looked like.
Submissions rose aggressively as hidden problems surfaced. Then, as those problems were worked through, the curve began to bend.
Then the curve turned
Eventually, submissions started trending down. That was the moment the graph changed meaning again.
The program was still active. The incentive still existed. People could still report problems. There were simply fewer obvious defects left for them to find.
That was encouraging, but I cared about a harder question: was any of this changing the experience outside our own quality process?
So I looked at customer-reported issues. That trend was falling too.
This is where the technical story became a business story.
The second graph belongs beside the first because the two tell different parts of the same system. One shows what happened when we aggressively increased internal discovery. The other shows what happened to customer-reported issues as that work moved through resolution.
I am careful with what those graphs can prove. I would not claim that every reduction in customer-reported issues came from one program. Real systems have too many variables for that.
But the relationship is difficult to ignore.
Internal discovery surged. We worked through the defects. Internal submissions then declined. Customer-reported issues declined afterward as well.
We did not have to rely on the feeling that the application was getting cleaner. We could point to evidence that its direction had changed.
The quality problem was a business problem
Before this program, triage could feel like whack-a-mole.
An issue would arrive and pull people away from whatever they were doing. Support had to capture it. Testing had to reproduce it. Engineering had to investigate it. Product might need context. Someone had to verify the fix. Then another problem could appear and start the cycle again.
A defect is rarely only a defect.
It is time.
It is interruption.
It is a customer conversation that should not have been necessary. It is an engineer tracing yesterday's problem instead of building tomorrow's capability.
That is why I do not think the most important result of the program was the number of bugs we found. The point was to reduce the cost those bugs were distributing across the company.
As the defect population came under control, repeated triage eased. The constant firefighting eased with it. Engineers and other teams had more room to do the work they were actually there to do.
Today, that old rhythm is largely gone. Issues still happen, but they are controlled in a way they were not during those earlier years. I rarely see the kind of repeated escalation that used to make quality feel permanently reactive.
The answer was not hundreds of clever fixes performed in isolation. It was one broad stroke that changed how the entire system revealed its problems.
From an architectural perspective, that is the part I find most interesting. We did not only improve defect resolution. We improved the quality signal itself.
Problems became easier to surface, easier to evaluate, easier to route, and harder to lose. Once the signal improved, the rest of the system could respond better.
Why the mechanism lasted
Two years later, the program is still running. That matters more to me than whether the first quarter looked impressive.
The application still has defects from time to time. Of course it does. The point was never perfection.
What changed was the baseline.
Finding meaningful defects now usually requires looking considerably harder than it did before. The steady stream of obvious accumulated problems is no longer the defining condition of the application.
And the mechanism did not disappear when the initial push was over.
A quality initiative that works only while everyone is excited about it is a campaign. What I wanted was something that could survive the campaign.
That is where Ryan's steady follow-through matters. He continues to help keep the engine running, carrying the repetitive, less visible work that still has to be done well as the program matures.
The lasting value came from reducing friction in the quality feedback loop. People had a reason to participate. They had a simple path to contribute. Their observations moved into an actual process. And the organization could see those contributions turning into real fixes.
That is how the program became part of the culture rather than another temporary quality push.
This was not a one-off fix. It was a mechanism that changed organizational behavior around quality and produced lasting improvement in an area that had challenged us for years.
Changing direction
Older systems can be overwhelming when years of hidden defects have turned normal quality work into constant reaction.
Sometimes replacement is the right answer. Sometimes an architecture really has reached a point where incremental improvement is no longer enough.
But age alone does not decide a system's future.
Sometimes the breakthrough is not another fix.
Sometimes it is recognizing that you are solving the problem at the wrong level.
I could have kept attacking defects one at a time. We would have continued making progress. But I do not think we would have changed the direction of the system.
The larger lesson I carried forward was to ask a different question: is there one mechanism that can change how an entire class of problems is discovered, understood, and resolved?
That is part of being a student of the craft. You learn when to solve the problem directly, and when the better move is to redesign the conditions around the problem.
Most of the time we ride the wave, but every once in a while we get the chance to build something that can ride it for us.
We did not replace the application.
We changed its direction in one quarter.
And years later, that trajectory is still carrying us forward.
Top comments (0)