By Stefano Dalla Palma · Development Tooling Engineer, Adyen
In a large-scale engineering organization such as Adyen, where hundreds of developers produce hundreds of merge requests per day across thousands of modules, continuously evolving code to meet the latest standards is essential for long-term platform health and performance. However, executing routine updates manually, such as adopting new framework patterns, keeping up with API enhancements, or fine-tuning static analysis rules can introduce friction that takes developers away from building core features.
When I joined Adyen’s internal developer platform team, I wanted to understand whether any of that friction could be automated away. Around that time, I came across an episode of Software Engineering Radio (link) where the creator of OpenRewrite made an argument that stuck with me: “We are asking framework authors to start taking responsibility for providing recipes when they make breaking changes.” I loved that framing. The hassle of keeping up shouldn’t fall entirely on the consumers of a library, but on the people who changed the contract in the first place.
While we couldn’t change how external maintainers worked, we could adopt that mindset internally. I saw an opportunity: if a tool could absorb the friction of a breaking change, we could handle the cleanup on behalf of our developers. But my enthusiasm wasn’t going to convince anyone. The engineers I pitched to wanted to see it work before they’d trust it, and I don’t blame them.
The only way to earn trust was to do the work: pick a real use case, drive it to completion, and let the results speak. One team had a ticket to migrate an internal Jackson 2 wrapper to Jackson 3, a tricky refactoring with deep caller dependencies across hundreds of modules, exactly the kind of high-effort, lower-urgency work that teams understandably prioritize behind customer-facing delivery. I asked if I could take a shot at it with OpenRewrite. Two months later, including ramping up on the problem, designing the recipe, and running it, the migration was done.
> Recipe (OpenRewrite): “A single, stand-alone code transformation that can be linked together with other recipes to accomplish a larger goal such as a framework migration.”
The tool worked, but integrating OpenRewrite smoothly into our build system required careful calibration to ensure it enhanced developer workflows while maintaining standard CI pipeline predictability. Our early prototypes provided valuable technical insights, showing us how easily automated suggestions could create review fatigue if not properly focused. After optimizing our orchestration strategy to prioritize targeted, high-value improvements, the system hit its stride. A year later, an engineer I’d never spoken to, shared a screenshot of one of our automated MRs and wrote: “Not sure who configured this kind of automated OpenRewrite MRs, but I have to say it is one of the coolest things I’ve seen recently.” That’s when I knew the system was working, because someone with zero context found it genuinely useful.
The results at a glance (TL;DR)
Before diving into how we built the orchestration system, it helps to look at what happened when we rolled it out at scale. In just our first two months, the system quietly produced over 4,000 automated MRs across the codebase, with a steady 70% merge rate, a median review turnaround under two hours, and a workload that spread naturally across more than 400 unique reviewers, so no single engineer was ever overloaded.
The macro-impact on our codebase was immediate. Beyond the initial Jackson migration, a single custom recipe automated a complex internal logging migration, instantly saving us hundreds of hours of manual refactoring overhead. We also hit 100% full compliance across every single module in scope for core cleanups like EqualsAvoidsNull, part of our groundwork for adopting stricter nullability checks across the codebase.
In that regard, our AnnotateNullableParameters recipe successfully achieved full compliance across roughly 80% of our 5,000+ modules, turning our upcoming NullAway adoption into a much shorter conversation. Finally, by launching a migration of our internal text utilities to standard, SonarQube-compliant equivalents, we’ve set ourselves up to permanently cut down NPE-related false-positive Sonar noise going forward.
Those results depended entirely on keeping the workflow frictionless. Here’s how we got there.
The calibration problem
When my team and I first integrated recipes into Adyen’s CI pipeline with the goal of automatically suggesting patches to developers’ MRs, we ran into a calibration problem: too few active recipes meant zero value; too many turned suggestions into noise. That noise had a particular shape specific to OpenRewrite. Because the tool processes Java files as complete units, a recipe rewrites everything it matches in a file; not just what the developer touched.
My first cut at this got it wrong, and the problem was a usability one. A developer modified a single line in a test class? My first version of the CI bot would fire an active ‘JUnit to AssertJ’ recipe and rewrite every non-compliant assertion in that file, surfacing changes developers hadn’t made and didn’t ask for. They responded predictably: they ignored most of the patches it generated. The lesson was simple in hindsight: you can’t cleanly enforce a standard on new code until the existing baseline already meets it.
Cleaning up before enforcing
To win back developer trust, we stopped enforcing recipes at MR time and moved the work into a background orchestration. Instead of introducing changes during the MR review itself, we deployed a system agent to clean the codebase quietly before activating any recipe in CI.
The agent runs iteratively in the background using Podman on a dedicated VM. It produces small MRs touching fewer than five files on average. These are fast to review, easy to approve, and unlikely to conflict with active feature work. We tracked compliance in a dedicated Spring Boot service connected to a PostgresSQL database that marks each code module as compliant once it passes a given recipe with no patch generated, at which point the recipe is activated for that module directly. If the run can’t complete due to timeouts, memory limits, missing build configuration, and the like, the recipe is disabled for that module and the failure is logged for future audit. If a patch is generated, an MR is opened. The agent keeps watching from there: if the MR is merged, the recipe is enabled; if it’s closed without merging, it’s disabled, with the closed MR taken as a signal that the recipe may need another iteration to handle an edge case the reviewer caught.
Figure 1: The two-phase system. Background containers clean modules one recipe at a time (left); the CI job queries only the recipes a module is already compliant with and post patch suggestions (right). The OpenRewrite Orchestrator and its compliance database sit at the center.
That compliance database became the brain of our CI strategy. When a developer opens an MR today, the CI job queries the compliance service to see which recipes the module is already compliant with. Only those active recipes run. Because the module is already at a clean baseline, any patch the recipe now generates is attributable to the developer’s new code; no ambiguity, no false blame.
Figure 2: When the CI bot flags an issue, the fix is one command away. No manual edits, no review-time ambush.
The case for staying small
We kept our automated MRs deliberately narrow by running one recipe per module, even when it would have been tempting to push entire modules to full compliance in a single sweep. Figure 3 shows what that looks like in practice, and why we stopped trying. There are a few reasons this was the right call for us.
Figure 3: The breaking point of sweeping refactors. A real example of a large multi-file change that proved too unwieldy to review as one unit, leading the developer to split it into smaller MRs.
Edge cases hide in the noise. Recipes are code, and like any code they have edge cases. The first version of the AnnotateNullableParameters recipe we used, for example, could correctly infer nullability from our custom utility Text.hasText(str): since the utility acts as a semantic null check, flagging str as @Nullable is correct. But applying the same logic to Text.hasText(str.getSomething()) would also flag str as nullable, even though it’s dereferenced before the check, resulting in a false positive that silently alters the method’s contract.
In a small MR, a reviewer has the bandwidth to catch this. In a sweeping MR, the same nuance gets buried in noise and either ships with incorrect assumptions or derails the entire review.
Legacy code carries dormant violations. There’s another risk in large-scale changes. Some modules contain long-standing classes that predate tools like Checkstyle, written before today’s automated style checks existed. The moment a recipe touches one of these files, dormant pre-commit checks fire and flag every long-standing violation: a utility class with a public constructor, a class that should be final, and so on. Whether that legacy code should be modernized or left alone is a genuine decision, but not one we want forced into an automated MR. With a small MR, a pipeline failure on a dormant violation costs us a couple of files and a quick decision. With a sweeping refactor, the same latent check can block hundreds of legitimate changes until someone finds the time to untangle a problem they didn’t sign up for.
Some changes quietly alter behavior. A third reason is that some recipes don’t just clean code, but they may quietly change behavior, and only the code owners know whether that change is acceptable. Our migration from an internal Text utility to its Apache Commons equivalent is a good example: the two libraries handle a couple of edge cases differently, and for most callers the difference is invisible, but for a handful of modules that adopted the internal version specifically for that behavior, it isn’t. A small MR makes that decision tractable: rollback is one revert, and the call gets made by the person who’s part of the team that knows why the original code looked the way it did. In a sweeping migration, the same decision either gets buried or gets made by someone who shouldn’t be making it.
Figure 4: A small MR gives reviewers room to resolve genuine ambiguity. Here, two reviewers work through whether an edge-case difference between two utilities matters — a decision that turns on domain knowledge, not the recipe.
One counterintuitive risk of staying small I observed: the recurring shape of recipe-generated diffs can create false familiarity. After approving a dozen near-identical MRs, reviewers may start to pattern-match rather than scrutinize. The Text migration above is exactly the kind of change where that matters: the diff looks routine, but the decision underneath isn’t. While running the first version of the AnnotateNullableParameter recipe, I saw a handful of legitimately faulty annotations slip through in otherwise clean batches. Our mitigation was a lightweight meta-review layer: while a recipe was still new to our context, a designated person walked through approved MRs before merging to double-check the changes. Once we were confident in the recipe’s behavior, we relaxed that layer.
One practical note: “small” was the right default for the null-safety recipes we prioritized, where one wrong change ripples through a lot of downstream code. For something like AssertJBestPractices, for example, a broader sweep is probably what you want.
When trust becomes engagement
The review side surprised me. I expected a cautious warm-up period; instead, people were challenging the recipe’s changes from day one.
For instance, one reviewer questioned an annotation on a method parameter, arguing that all current callers pass non-null, so @Nullable felt unnecessary. Rather than accepting the suggestion, she pushed a follow-up commit swapping @Nullable for @NonNull (Figure 5). That’s the system working as intended. The recipe surfaced a contract decision that was previously implicit; the reviewer made it explicit using her domain knowledge. Also, doing so, she ensured the recipe correctly leaves that parameter alone next time it runs on the same file.
Figure 5: A reviewer overrides the suggested @Nullable with @NonNull, making the contract explicit and preventing future suggestions on the same parameter.
One human-side footnote: some reviewers encountering our bot for the first time assumed it was an AI agent. We saw MRs sit for days because a reviewer asked a question and waited for a reply that was never coming. The quick fix is a templated response pointing back to a human owner. The broader observation is that the “leave a comment and someone will respond” expectation is now so deeply embedded in how engineers interact with tooling that a deterministic bot reads, at first glance, like a conversational one.
A few things I’d recommend
Moving from a single successful script to an automation engine that processes thousands of modules completely changed how I think about platform tooling. If you are looking to build something similar, these are the operational design choices I recommend making on day one:
- Auto-merge on approval. I noticed a frustrating pattern: developers would enthusiastically approve an automated migration or cleanup, and then… the MR would just sit there forever. Reviewers treat automated code fixes like a drive-thru; if they approve it, they want it integrated immediately. We eliminated the manual step entirely.
- Mute secondary tooling. Overlapping automation erodes a bot’s credibility fast. When OpenRewrite MRs triggered secondary AI reviewers or strict SonarQube rules on unrelated lines in the same file, reviews stalled. We decided to disable auxiliary tooling with specific labels and tackle those findings separately.
- Auto-resolve informational threads. If your platform requires all threads to be resolved before merging, a simple webhook listener is non-negotiable. Reviewers were incredibly quick to approve automated code, but they consistently left system-generated comment threads open, treating resolution as someone else’s job. We set up a notes webhook to auto-resolve them as soon as they were posted.
- Force yourself to start small, even when you don’t have to. It is incredibly tempting to run a recipe across an entire repository to hit a metric, but large codebases are packed with custom utilities and build quirks that open-source recipes were never written to expect. A micro-MR is the cheapest, safest telemetry you can ask for. Besides, the fixes tend to make good open-source contributions.
What’s next
We have laid the groundwork for automated code modernization at Adyen, but scaling it across the entire corporate infrastructure is where things get fascinating. CI pipelines already carry heavy loads, they run security scans, tests, and database sanity checks, and code modernization naturally sits lower on that priority stack. Running every recipe against every commit diff on top of that doesn’t hold indefinitely, but our architecture softens the problem: because recipes are only enabled per module once that module is compliant, CI overhead grows gradually alongside adoption rather than hitting the system all at once. One direction I want to explore is predicting which recipes are worth running on a given change based on the diff itself, similar in spirit to how we already do test selection [link]. You might wonder why we don’t just shift left and block these violations at the source using lightweight local checks, but OpenRewrite’s execution overhead makes local enforcement too slow to be practical at our scale, where developers already run a number of pre-commit hooks locally.
But the harder questions aren’t purely technical; they are deeply behavioral and cultural. Lately I’ve noticed a subtle dynamic worth designing around: when a developer defers a suggestion to keep a feature moving, the module’s compliance state can drift, so a later change to those files may surface a suggestion unrelated to that engineer’s work — the very situation the system is meant to avoid. I don’t have a clean answer yet, but it tells me the next phase of this work is as much about incentive design and engineering culture as it is about tooling.
A few colleagues also raised the question of accountability in a high-stakes engineering environment like ours. Every automated change is reviewed and approved by a human before it merges, but when a bot opens an MR, who actually owns the resulting commit? Is it the engineer who triggered the run, the team that authored the recipe, or the system itself? Each option shapes reviewer behavior and operational scalability differently. I don’t think there is a universal answer; it likely depends on the scope of the change and the maturity of the recipe. But it’s a question worth asking, because anyone building a similar system will run into it, sooner than they expect.
Acknowledgments
Huge thanks to the Build Optimization and Automated Testing (BOAT) team for driving this work, to the CI/CD team for their instrumental support and enablement, and to the Corporate Communications and Developer Relations teams for making this post possible.
Top comments (0)