DEV Community: Brad Chase

November 25 XRPL Bug: Current Status and What’s Next

Brad Chase — Wed, 27 Nov 2024 19:29:08 +0000

On 13:39 UTC on November 25 2024, the XRP Ledger experienced an issue in which several nodes across the network crashed and restarted at similar times. During periods of instability, the XRPL network consensus model favors safety over progress, and as a result, the network did not process any transactions for approximately 10 minutes as it recovered. There was no loss of funds (just momentary pause of new transactions). The network resumed normal behavior and forward progress at 13:49 UTC on the same day.

Thanks to the community’s quick response once the fix was introduced (1:30 UTC November 26), 33 of 35 validators on the default UNL upgraded to rippled 2.3.0 by publish time of this post, along with nearly half of known servers. While the core of the network has upgraded, unpatched nodes remain potentially susceptible, and we encourage all users to update their infrastructure to rippled 2.3.0 immediately.

Last week, at the same time as completing the testing of rippled 2.3.0, the RippleX team identified the bug and included a fix in an internal release candidate. To minimize risks, we decided not to isolate the issue pre-release or create a standalone patch, as this could have enabled reverse engineering and exploitation of the bug. When a similar issue appeared on mainnet, we validated the fix and worked directly with the community and other UNL operators to coordinate upgrades and secure the network.

The issue stems from a bug introduced more than 6 months ago. In some circumstances, the caching layer in rippled can return an inconsistent result type which can cause a server to crash. Although this bug went undetected during testing of this refactor, there is no prior evidence of exploitation. To minimize potential risks for unpatched users, we are still withholding specific technical details until the majority of servers are upgraded.

So what's next? To give remaining node operators sufficient time to upgrade, we will wait to share more technical details until December 12. Once those details are available, the bug will become easier to identify, potentially increasing the risk for nodes that remain unpatched. Again, we strongly recommend upgrading to 2.3.0 as soon as possible.

This issue is a reminder of the complexities that come with innovation and introducing new features to a decentralized system like the XRPL. There’s always room to improve, and we appreciate the close collaboration among the community to make the network stronger and safer.

The XRPL depends on the global community of validators, developers, and contributors to keep it running. By learning and continuing to iterate, together, we can make the network more secure and reliable for everyone.

XRP Ledger AMM Bug Fix Now Integrated: A Detailed Analysis

Brad Chase — Fri, 12 Apr 2024 19:49:02 +0000

The bug fix for the XRP Ledger’s Automated Market Maker (AMM feature) has been successfully integrated into the XRPL Mainnet as of April 11. This marks the resolution of an issue discovered on March 23 regarding an unexpected number of LPTokens in an AMM pool. Thanks to the collaboration with XRPL validators and community participants, we were able to work together to identify, diagnose and resolve the issue swiftly.

In my previous blog post, I provided an overview of the issue, immediate response, and next steps. As promised, this follow-up post will dive deeper into the details of the bug, its impact, the lessons learned, and the ongoing work to prevent similar issues from occurring in the future.

Background - Payment Engine and Liquidity

A major benefit of the XRPL is how features are composable, which enables flexibility and efficiency in payment processing. Through the XRPL’s payment engine, the AMM and built-in decentralized exchange (DEX) can both interact to provide liquidity for payments. Some payments, such as transferring XRP from one account to another, are straightforward. Others are more complex. For example, consider a scenario where a user wants to send USD and have the destination account receive EUR. In this case, the payment engine would use the USD to EUR order book to convert the funds. The process of specifying how currencies are meant to be converted is called a “payment path.”

The payment engine can use one or more paths to complete a payment. In the example, in addition to the USD to EUR order book, the sender could also specify a second path that uses an order book to convert from USD to XRP; then a second order book from XRP to EUR. Both payment paths may be used in the same payment, and the payment engine will always consume liquidity from the path that offers the most cost-effective option for the sender.

AMMs provide a new way to convert between assets that complements the DEX’s order books. Unlike offers, which have fixed exchange rates, the AMM exchange rate varies continuously as liquidity is consumed from the AMM pool according to a constant product formula (see XLS-30 for details). The payment engine needs a way to compare the discrete liquidity from an offer with the continuous liquidity from an AMM to provide the most cost-efficient path for payments.

To address this challenge, the payment engine creates synthetic offers backed by the AMM. From the payment engine's point of view, these synthetic offers act the same way as users' offers. In particular, consuming liquidity from these synthetic offers does not change the exchange rate. The size of synthetic offers depends on the specific payment paths. When there is only a single payment path, the engine can use the constant-product formula to infer the right-sized exchange rate and offer size. In the multi-path case, that size depends on the liquidity from other paths, so the payment engine iteratively generates synthetic offers of increasing size to utilize liquidity from the AMM pool.

The Bug

The bug occurred when a multi-path payment interacted with an AMM synthetic offer, and the synthetic offer required to fulfill the payment exceeded the AMM pool size. In this case, the payment engine attempted to use the single-path strategy to back-out the maximum liquidity available to swap against the pool. Although this condition was properly detected, there was an error on how the payment engine resized the synthetic offer to consume liquidity only up to the AMM size. Instead of using the actual swap rate, which represents the cost of fully swapping one side of an AMM pool, the new synthetic offer’s exchange rate was incorrectly set to the spot exchange rate of the AMM (that is, the rate it takes to execute an infinitesimally sized order). Consequently, an AMM operation that should have been expensive became relatively cheap, effectively violating the constant-product invariant of the pool. In this state, another user could deposit funds into the drained side of the pool, generating a large number of LP tokens and taking ownership of the pool. Now that the fix amendment is active, the team has opened a PR with a unit test demonstrating this behavior.

Impact

At the time of the bug report, there was only a single impacted AMM pool holding USD.rvYA and USD.rhub. Prior to the bug, the pool had $273 of USD.rhub and $3 of USD.rvYA. After a subsequent payment that caused the constant product rule to be violated, the pool had nearly $0 of USD.rhub and $16 of USD.rvYA. The payment was instructed to swap out $272.455 of USD.rhub by spending up to $50 of USD.rvYA, but was fully executed by spending only $13 of USD.rvYA. This left the pool in an unbalanced state, where a subsequent deposit in the pool enabled a single depositor to own nearly all the pool’s LP tokens. Since that time, a large AMMBid transaction burned enough LP tokens to restore the constant product invariant of the pool.

During the time after the release of the bug fix but before the amendment became enabled, there appears to only be one other AMM pool impacted by this bug, this time holding EUR.rhub and SeagullCash.rNHe. For this pool, an OfferCreate transaction, which automatically leverages auto-bridging to execute against available liquidity sources, was able to withdraw the entire €6.103 from the pool in exchange for 1 billion SeagullCash.rNHe. Due to the bug that fully drained one side, the pool is in a stuck state where assets cannot be added to or removed from the pool.

Timeline & Response

As a brief recap of events:

Bug Discovery: Early morning March 23, the Ripple dev team received a report from Tequ, a member of the XRPL community, regarding an unexpected number of LP Tokens in an AMM pool.
Issue Diagnosis: By the evening of March 23, the Ripple dev team, alongside community members Tequ and Orchestra, swiftly investigated the issue to identify the root cause and assessed the impact on AMM pools.
Fix Development and Testing: Once the bug was reproduced, the team quickly had a fix prepared, but worked through March 24 to add an additional invariant check to enforce the properties of AMM pools during transaction processing. This check took extended time to develop in order to validate the numerical sensitivity of the invariant across a broad range of liquidity scenarios.
Community Communication: We promptly communicated the issue to the XRPL community through X (formerly Twitter) and direct outreach to community members including AMM UIs, ensuring transparency and collaboration throughout the process.
Hotfix Release: On March 26, a hotfix PR was opened. It was merged on the morning of March 27.
Amendment Voting: The amendment for the bug fix was supported by XRPL validators and reached 80% consensus on March 28 and activated on April 11, showcasing the community's commitment to the XRPL’s stability and security.

Lessons Learned and Fortifying the Future

The initial focus was on fixing the bug and mitigating the impact. Since releasing the fix, the team has spent significant time reflecting on how to prevent a similar issue from occurring in the future, and on how the community can best respond and manage incidents like this.

Invariants are key. Invariant checking is a safety feature of the XRP Ledger. It consists of a set of checks, separate from normal transaction processing, that guarantee that certain invariants hold true across all transactions and helps eliminate any adverse edge conditions or error case mishandling. The AMM feature has an obvious invariant from the constant product formula -- the number of issued LP tokens should always be less than or equal to the product of the square root of the assets in the pool. Although the team had built an invariant check during development, it was inadvertently removed before release. This was a huge miss.

The team is taking several corrective actions – first, the missing invariant check was restored as part of the fix. Second, the team is also working to add additional invariants for other AMM transactions, and will initiate a workstream to review existing XRPL features and propose additional invariants that would make those features more robust. Going forward, the team will also propose updating the XLS and PR process with an explicit section on invariants, as well as a section to address any numerical sensitivity decisions (which was a proximate issue for this AMM bug).
Composability comes with complexity. As mentioned above, a major benefit of the XRPL is the way features like the DEX, AMM and payments compose and work together; there aren’t any other protocols that seamlessly integrate these types of features. But this also creates a large set of interactions and edge cases to test. Although the XRPL has a strong track record of performance and stability due to the existing software development and testing processes, there are always ways to improve. The team is finalizing a PR that significantly increases unit test coverage of the AMM feature, with a goal of 100% coverage for any new features added to the Ledger going forward. Leveraging the added invariant checks, the team will also start a workstream to build out context-aware fuzz testing to thoroughly stress the protocol in a controlled manner. We are excited to learn more about community proposals for a Canary Network, and are also iterating on the existing Bug Bounty program to ensure it has the right visibility and support for extensive testing in the test and dev networks in addition to the main network.
It takes a village. A huge thank you to the community members who responsibly disclosed the bug, the projects who quickly responded to limit further deposits into AMMs, the devs who worked around the clock on the fix, validator operators who voted in favor of the amendment fix, and the broader community for the support and patience during this process. These last few weeks should serve as a starting point for a playbook for future incidents, a foundation that the XRPL community can collectively develop and improve.

A major challenge in responding quickly was the 2 week window for approving amendments. The two week period is meant to give sufficient time for node operators to upgrade to the version of rippled that supports that amendment, for validator operators to review the changes to decide their vote, and also change their vote based on feedback from other operators and the community. For new features, this is an appropriate mechanism for the community to decide the best path forward for the ledger. For urgent bug fixes, 2 weeks is an eternity. The team will work on XLS proposals for alternative ways to deploy fixes, such as fast-track amendments that have an accelerated voting process. We look forward to hearing similar ideas and innovations from the community on ways to respond quickly when necessary.

As we move forward, Ripple will continue to monitor the AMM feature closely alongside the XRPL community to ensure its stability and reliability. As with any new functionality, we encourage the community to test and educate themselves on how it works. Together, we can work towards building a stronger, more resilient XRP Ledger that empowers innovation and drives the future of on-chain finance.

Update on the XRP Ledger AMM Bug and the Path Forward

Brad Chase — Wed, 27 Mar 2024 16:14:11 +0000

Last Friday, after over two years in the making, XLS-30, the Automated Market Maker (AMM) amendment, went live on XRP Ledger mainnet. The native feature was designed to provide on-chain liquidity and trading capabilities for DeFi developers and users.

Despite the multiple rounds of rigorous testing and audits leading up to the mainnet launch, on Saturday, Tequ and Orchestra Finance, members of the XRPL community, discovered a bug affecting the AMM functionality. Over the past few days, the bug prompted swift action from developers and the RippleX team to ensure the security and efficiency of the XRPL AMM feature. Following thorough testing, the AMM bug fix was released Tuesday evening, and is now available under the 2.1.1 release of rippled.

In this update, we want to provide transparency regarding the recent events surrounding the AMM bug, discuss the work done to address it, and outline the path forward.

What happened
Saturday morning, March 23, Ripple received a report of an inconsistency in an AMM liquidity pool. Alongside community members Orchestra Finance and Tequ, analysis was conducted on the corresponding transactions and an error was identified in how the payment engine interacts with order books and AMM pools. In rare cases, complex payment path scenarios draw an incorrect amount of liquidity from an AMM pool.

This issue was not related or caused by the single-sided deposit feature, which is an intended functionality of the AMM, as it offers a more streamlined user experience, and could result in price impacts when pools have less liquidity. We acknowledge the concerns raised by some users regarding the challenges associated with single-sided deposits. While the feature is functioning as designed, it’s important to address these concerns. As such, we and others will explore and propose potential future improvements for the community to consider to enhance the overall user experience.

Given that this AMM bug fix requires an amendment voting process to correct, the issue remains present on the network in the interim period. Although this appears to be a rare occurrence, out of an abundance of caution, it’s best for users to limit their AMM usage until an amendment addressing the bug is passed by the validators. To prevent exploitation, the in-depth details about the issue are being withheld at this time but AMM UIs have been advised to close deposits. However, after the fix has been activated post amendment voting process, more information will be shared.

It would be remiss not to mention the quick response from many community participants – including Sologenic, Anodos Finance, Orchestra Finance, Dex Finance Pro, XPMarket, and others – in acting quickly to close deposit functionality, as well as many individuals who were vital in spreading the word on the issue and fix. The team will continue to stay in close contact with them as monitoring is still in place.

How the team responded
Once the bug was identified, two parallel workstreams kicked off; the first to assess the scope and impact on AMM pools, and the second to reproduce and fix the bug. Regarding scope and impact, the bug has not affected additional pools since the initial report. There is ongoing monitoring in place to detect if the issue were to re-occur.

The bug was successfully recreated in a unit test environment, facilitating the development of a fix alongside a new invariant check to enforce properties of AMM pools during transaction processing. Since the payment engine rules reside in the protocol layer, implementing the fix necessitates an amendment voting process which is reviewed by UNL validators at minimum of two weeks, requiring over 80% consensus. Extensive internal reviews, performance testing, and quality assurance measures have been conducted on the code changes. The code is viewable on GitHub, but excludes the unit tests to minimize the risk of retriggering the bug on the network. These tests will be incorporated into a subsequent release once the amendment activates.

What is next
The top priorities these last few days have been to (1) mitigate the impact of the bug on the network for XRPL users, and (2) prepare and thoroughly test a fix – thus intentionally limiting the amount of details shared about the bug to minimize the risk of additional impacts while the fix is being released and voted on by the network. All validators are encouraged to commence the voting process to quickly address this issue. Note that the default vote in the source code has been set to “yes” given the importance of the fix, though as always, the amendment must hold over 80% support for two weeks to activate, so there is time for operators to review and change their support as they see fit.

Post-activation, more specific details of the bug, the fix, as well as additional follow-on actions will be shared via a root cause analysis of the incident. A key part of this will be addressing how Ripple and the larger XRPL community can work more closely together to build, test, deliver and improve the XRPL for all its users.

Lastly, a huge thank you to the community members who responsibly disclosed the bug, the projects who quickly responded to limit further deposits into AMMs, the devs who worked around the clock on the fix, and the broader community for the support and patience during this process. It’s been great to see the community involvement in initiating discussions on ways to improve the XRPL and we are on the front lines with you.