Confident Refactoring When Money Is on the Line

Safely Refactoring Critical Payment Systems

In the high-stakes world of financial technology, code modifications carry extraordinary risks that extend far beyond typical software development concerns. When dealing with payment systems, even minor errors can translate into substantial financial losses and irreversible damage to customer trust. This reality creates a paradox where development teams often avoid essential refactoring, allowing critical codebases to deteriorate over time.

The Deterioration of Payment Infrastructure

Having extensively worked with legacy payment processing systems, I've witnessed firsthand how codebases can become tangled webs of complexity when developers approach them with excessive caution. This phenomenon mirrors the broken window theory in urban decay—when code appears neglected, it invites further neglect. In payment systems, we might call these "spooky windows"—sections of code so intimidating that teams develop an almost superstitious avoidance of them. The natural consequence is that new features get awkwardly shoehorned into existing structures rather than properly integrated, creating technical debt that compounds with each iteration.

A Case Study: Payment Calculation Logic

Consider a recent project where I needed to enhance functionality affecting customer charges. The core calculation logic resided in a single module that had become nearly incomprehensible through years of incremental changes. New features had been implemented as additional conditionals that rarely aligned with the original design intent.

The situation was further complicated by:

• Complete absence of unit tests to document expected behavior
• Integration tests scattered throughout the test suite with significant coverage gaps
• Comments revealing fundamental uncertainty about the code's purpose, such as "not sure why we do this" and "how can this be zero?"

These comments appeared in code responsible for determining customer charges—a deeply unsettling situation for any developer.

A Strategic Approach to Safe Refactoring

When faced with such challenges, traditional testing strategies often fall short. In this case, the module's complexity and interconnectedness made it impractical to simply add tests as a safety net. Instead, I employed Verify Branch by Abstraction—a technique developed by Steve Smith for introducing new code while minimizing failure risk.

This strategy involves creating a toggle that delegates between the original implementation and a new candidate version, with both paths running simultaneously. The results are compared, and any discrepancies trigger alerts while ensuring the original behavior remains in effect for end users.

Implementation in Production

The approach began by isolating the existing calculation logic in a dedicated file (old_calculator.rb) while developing the refactored version in a separate file (candidate_calculator.rb). The entry point was modified to run both implementations when enabled:

def call(input)
  original = OriginalCalculator.call(input)

  if Feature.enabled?("calculator_refactor")
    candidate = CandidateCalculator.call(input)
    compare(original, candidate)
  end
  rescue => e
    log_error(e)
  ensure
    original
end

def compare(original, candidate)
  # compares the attributes of both objects and logs
  # an error if they differ
end

This implementation provided several safety mechanisms:

1. Feature flag control to enable/disable the comparison
2. Comprehensive error handling
3. Assurance that the original implementation always takes precedence

Observing and Addressing Discrepancies

With both code paths running in production, discrepancies quickly emerged, revealing two issues:

1. A misunderstanding of how a particular discount was applied
2. An edge case scenario that hadn't been anticipated

After implementing fixes, the failure rate dropped to zero. Additional regression tests were added, and previously undocumented functionality was properly recorded. Despite these improvements, I maintained the dual implementation for two weeks before switching entirely to the new version. During this extended monitoring period, another obscure bug surfaced, reinforcing the value of thorough testing in production environments. The final transition to the refactored code occurred only after a full month of dual operation.

Alternative Tools and Approaches

While the Verify Branch by Abstraction pattern can be implemented manually as shown, several specialized tools exist to streamline this process. The Ruby gem "scientist" provides a robust framework for exactly this purpose, with an elegant API for conducting controlled experiments in production. I've successfully used it to migrate an Elasticsearch cluster from version 1.X to 6.X without service interruption.

For teams working in other programming languages, numerous alternatives to scientist exist across different ecosystems. These tools typically offer standardized approaches to:

• Running parallel code paths
• Comparing results
• Managing feature flags
• Analyzing discrepancies

Conclusion

Refactoring critical payment systems doesn't have to be a high-stakes gamble. By implementing careful verification strategies and leveraging production data for validation, development teams can modernize essential infrastructure without risking customer experience or financial integrity. The key lies in embracing gradual, controlled change rather than attempting risky big-bang approaches that have historically given refactoring its dangerous reputation.

In the end, patience and thorough verification prove more valuable than speed when working with mission-critical payment code. The technical debt accumulated through years of avoidance can be systematically addressed—provided teams have the right tools and methodologies to do so safely.