Scaling Ory Hydra to ~2bn monthly OAuth2 flows on a single PostgreSQL DB

OryDev — Fri, 04 Aug 2023 10:26:40 +0000

Ory Hydra is the most popular open-source OAuth2 and OpenID Connect server that provides secure authentication and authorization for applications. One of the key challenges in building a scalable and performant OAuth2 server is managing the persistence layer, which involves storing and retrieving data from a database.

Motivation

Ory was approached by a popular service provider to optimize performance in their auth system at high load. They were currently using Auth0 and often struggled to cope with the huge influx of authorization grants during peak times (over 600logins/sec). Looking for another solution, they started to evaluate Ory Hydra, investigating if it can handle this amount of grants. After reaching out to the Ory team, we started to assess Ory Hydra's performance capabilities and investigate ways to improve the overall performance to make Ory Hydra faster and more scalable than ever before. The key to success was to re-engineer parts of Hydra's persistence layer to reduce write traffic to the database, moving to a transient OAuth2 flow.

Moving to a transient OAuth2 flow

One of the core parts of this work was moving a large chunk of the transient OAuth2 flow state, which is exchanged between the three parties involved in an OAuth2 flow, from the server to the client. Instead of persisting transient state to the database, the state is now passed between the parties as either AEAD-encoded cookies or AEAD-encoded query parameters in redirect URLs. AEAD stands for authenticated encryption with associated data, which means that the data is confidential and also can't be tampered with without knowing a secret (symmetric) key.

The flow is then only persisted in the database once when the final consent is given.

This change has several benefits. First, it reduces the amount of data that needs to be stored in the database, which in turn reduces write traffic. Second, it eliminates the need for multiple indices on the flow table that were previously used during the exchange.

The OAuth2 authorization code grant flow in detail

The relevant part of the OAuth2 flow that we wanted to optimize is an exchange between the client (acting on behalf of a user), Hydra (Ory's OAuth2 Authorization Server), and the login and consent screens. When a client requests an authorization code through the Authorization Code Grant, the user will be redirected first to the login UI to authenticate and then to the consent UI to grant access to the user's data (such as the email address or profile information).

Below is a sequence diagram of the exchange. Observe that each UI gets a CHALLENGE as part of the URL parameters (steps 3 and 12) and then uses this CHALLENGE as a parameter to retrieve more information (steps 4 and 13). Finally, both UIs either accept or reject the user request, usually based on user interaction with the UI (from steps 6 to 8 and 15 to 17). This API contract keeps Ory Hydra headless and decoupled from custom UIs.

Optimization: Passing the AEAD-encoded flow in the URL parameters
To reduce database access we now pass as the LOGIN_CHALLENGE, LOGIN_VERIFIER, CONSENT_CHALLENGE, and CONSENT_VERIFIER an AEAD-encoded flow. This way, we rely on the parties involved in the OAuth2 flow to pass the relevant state along.

Before:

The login and consent challenges and verifiers are random UUIDs stored in the database.
After:

The login and consent challenges and verifiers are the AEAD-encoded flow.

Before:

Accepting or rejecting a request from the UI involves a database lookup for the specific challenge.
After:

Accepting or rejecting a request from the UI involves decrypting the flow in the challenge and generating an updated flow as part of the verifier.

Implementation

Since Ory Hydra is open source, you can review code changes in the Ory GitHub repositories. This is the relevant commit: https://github.com/ory/hydra/commit/f29fe3af97fb72061f2d6d7a2fc454cea5e870e9.

Here is where we encode the flow in the specific challenges and verifiers:

// ToLoginChallenge converts the flow into a login challenge.
func (f *Flow) ToLoginChallenge(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsLoginChallenge)
}

// ToLoginVerifier converts the flow into a login verifier.
func (f *Flow) ToLoginVerifier(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsLoginVerifier)
}

// ToConsentChallenge converts the flow into a consent challenge.
func (f *Flow) ToConsentChallenge(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsConsentChallenge)
}

// ToConsentVerifier converts the flow into a consent verifier.
func (f *Flow) ToConsentVerifier(ctx context.Context, cipherProvider CipherProvider) (string, error) {
    return flowctx.Encode(ctx, cipherProvider.FlowCipher(), f, flowctx.AsConsentVerifier)
}

In the persister (our database repository) we then decode the flow contained in the challenge. For example, here's the code for handling a consent challenge:

func (p *Persister) GetFlowByConsentChallenge(ctx context.Context, challenge string) (*flow.Flow, error) {
    ctx, span := p.r.Tracer(ctx).Tracer().Start(ctx, "persistence.sql.GetFlowByConsentChallenge")
    defer span.End()

    // challenge contains the flow.
    f, err := flowctx.Decode[flow.Flow](ctx, p.r.FlowCipher(), challenge, flowctx.AsConsentChallenge)
    if err != nil {
        return nil, errorsx.WithStack(x.ErrNotFound)
    }
    if f.NID != p.NetworkID(ctx) {
        return nil, errorsx.WithStack(x.ErrNotFound)
    }
    if f.RequestedAt.Add(p.config.ConsentRequestMaxAge(ctx)).Before(time.Now()) {
        return nil, errorsx.WithStack(fosite.ErrRequestUnauthorized.WithHint("The consent request has expired, please try again."))
    }

    return f, nil
}

Let's look at the impact of the changes when compared to the code without optimizations:

The flows are now much faster and talk less to the database.

Improved indices lead to further performance improvements
By introducing a new index on the hydra_oauth2_flow table, we were able to increase throughput and decrease CPU usage on PostgreSQL. The screenshot below shows the execution of the benchmarks without the improved indices where CPU usage spikes to 100%, and with improved indices, where CPU usage stays below 10%.

With the newly added indices, CPU usage (green bars) is removed, which reduces the likelihood of BufferLocks and related issues:

Results

The code and database changes reduced the total roundtrips to the database by 4-5x (depending on the amount of caching done) and reduced database writes by about 50%.

Benchmarks

Benchmarking the new implementation on Microsoft Azure with the following specifications:

Ory can perform up to 1090 logins per second at the peak and 800 logins / second consistently in the above configuration. This is possible by making the flow stateless and optimizing indices in frequently used queries.

Conclusion

The performance optimization work done by the Ory team has resulted in a significant improvement in Hydra's performance and scalability. By reducing write traffic to the database and improving the codebase and dependencies, Hydra is now faster and more responsive than ever before. By improving the indices, Hydra now scales much more efficiently with the number of instances.

In the future, we will continue to optimize Ory's software to handle even more traffic. We believe that it's possible to get 5x more throughput on a single PostgreSQL node with data model optimizations.

If you're building an OAuth2 server, we highly recommend giving Ory's fully certified OpenID Connect and OAuth2 implementations a try: Ory OAuth2 – our fully managed service running on the global Ory Network, based on open source Ory Hydra – already utilizes the optimizations described in this article and setting it up only takes few minutes!

Boosting Performance and Scalability with Ory Network Edge Sessions

OryDev — Tue, 18 Jul 2023 11:12:08 +0000

In today's digital world, where users demand fast and responsive applications, optimizing performance and scalability is paramount. This also includes a fast and scalable delivery of personal information and session data.

One effective technique that can enhance the performance of an application and by that also the user experience is the use of Ory Network Edge Sessions. By enabling Edge Sessions, Ory Network takes advantage of Cloudflare's global network infrastructure, utilizing smart caching strategies to validate sessions at a node which is milliseconds away from virtually every Internet user, thereby reducing latency.

Benefits of using Ory Edge Sessions

Ultra Low Latency: In the realm of identity management and access control, every millisecond matters. Edge Sessions provide critical user session information at the node with the closest geographical proximity to the requesting client, rather than repeatedly fetching session data from a datacenter that may be hundreds or even thousands of miles away. This approach drastically reduces response times, enabling swift authentication processes, fostering a frictionless user experience.

When leveraging Edge Sessions with Ory Network, you can expect a P95 latency of ~60ms and a P99 latency of ~70ms across the globe. (*1)

Scalability for Growing Demands

As businesses expand and user bases grow, scalability becomes a paramount concern. Our Ory Network and Ory Edge Sessions have been designed with scalability in mind, allowing organizations to effortlessly handle increasing traffic and concurrent user sessions. By distributing session information across multiple nodes, Ory Networkgu ensure optimal performance and eliminate bottlenecks. This supports the evolving needs of organizations to thrive and expand.

Fortifying Security Measures

Security is at the core of our products and edge sessions play a vital role in strengthening data protection. By smart caching session data in a secure and encrypted manner, we minimize the need for frequent database or server access, reducing the attack surface and potential vulnerabilities. Additionally, Edge Sessions are equipped with robust expiration and invalidation mechanisms, ensuring that session data is automatically cleared after a specified time or updated when a user updates their profile. This proactive approach to managing sessions enhances security by preventing unauthorized access and mitigating the risk of session hijacking.

Optimized Resource Utilization

Edge Sessions not only enhance performance and security but also optimize resource utilization within Ory Network. With session data stored in an edge node, the load on servers and databases is significantly reduced. This efficient resource allocation translates to improved overall system performance, enabling Ory Network to handle larger user volumes without sacrificing speed or compromising user experiences. Organizations can rely on Ory Network to deliver exceptional performance consistently, even during peak usage periods.

Conclusion

With Ory Network, we prioritize delivering a comprehensive identity management and access control solution that excels in performance, scalability, and security. By leveraging the power of Edge Sessions, we enhance the user experience by providing the lowest possible latency and seamless scalability. Trust in our commitment to delivering excellence as you leverage Ory Network to streamline user management and access control with optimal efficiency and peace of mind.

Get in touch with the Ory team to learn more about how our solution can empower your organization.

The Edge Sessions feature is a available on all Ory Network paid plans. You can find more information on how to facilitate Edge Sessions in the Ory Network here.

(*1) Results can vary depending on the geographical proximity to the closest node

DEV Community: OryDev

Scaling Ory Hydra to ~2bn monthly OAuth2 flows on a single PostgreSQL DB

Motivation

Moving to a transient OAuth2 flow

The OAuth2 authorization code grant flow in detail

Implementation

Results

Benchmarks

Conclusion

Boosting Performance and Scalability with Ory Network Edge Sessions

Benefits of using Ory Edge Sessions

Scalability for Growing Demands

Fortifying Security Measures

Optimized Resource Utilization

Conclusion