DEV Community: SWCode

Mastering Data Consistency: A Deep Dive into the Transactional Outbox Pattern

SWCode — Tue, 23 Jan 2024 07:30:41 +0000

Unlock the potential of modern distributed systems with our latest exploration: The Transactional Outbox Pattern.
Discover how this pattern enhances data consistency, reliability, and system resilience in microservices architectures.

Introduction: Navigating the Complexities of Modern Software Development

In the realm of software development, managing data consistency and integrity is a cornerstone of reliable systems.
The complexity escalates when we transition from a single, well-understood system to an interconnected web of services and components.

The simplicity and robustness of SQL databases have long been a foundation in software development.
These databases offer ACID (Atomicity, Consistency, Isolation, Durability) properties, ensuring transactions are processed reliably.
In this controlled environment, developers can perform business logic within transactions confidently, knowing that any failure will trigger a rollback, preserving data integrity and preventing inconsistent states.
This traditional model has proven effective and straightforward when operations are confined within the bounds of a single database.

However, modern software architectures often require interactions beyond a single database.
The introduction of third-party systems and external services adds layers of complexity when business logic involves modifying data across distributed systems,
such as a SQL database and a event bus like RabbitMQ.
In these distributed architectures, the ACID properties, which are the backbone of transactional integrity within a single database,
become challenging to enforce across multiple systems.

One of the core issues is the uncertainty of atomicity.
For instance, consider a scenario where a transaction involves writing an event to a RabbitMQ and also updating a record in a database.
There's a possibility that the event write to RabbitMQ succeeds, but the outer database transaction fails.
This results in a scenario where an event indicating a business action is published,
but the corresponding database state does not reflect this action because the transaction was not completed successfully.
The system thus ends up in an inconsistent state, with the event bus reflecting a change that the database does not.

When integrating an event bus or any other external service within an application in a distributed system, there are two naive approaches one might consider.
Each approach has its own set of challenges and potential pitfalls:

1. Execute Business Logic Within the Transaction and Publish Events After Committing the Transaction

In this approach, the application first executes all the required business logic within a database transaction.
After this transaction is successfully committed, it then publishes the relevant events to the event bus.

Challenges:

Risk of Event Loss: If the application crashes or encounters an issue between the transaction commit and the event publication, the event may never get published. This leads to a situation where the database state has changed, but the corresponding event informing other services of this change is lost.
Inconsistency in Distributed Systems: Other services relying on these events for triggering their own processes will not be aware of the changes, leading to data inconsistency across the system.
Complex Recovery Mechanisms: Implementing a mechanism to check for such failures and recover from them can be complex and error-prone.

This approach can be suitable for situations where the events being published are not critical to the core business logic or data integrity of the application.
An example of this might be events used for monitoring or logging purposes.
In these cases, the loss of some events due to system failures or crashes might be considered an acceptable risk.

2. Execute Business Logic and Publish Events Within the Transaction

In this approach, the application attempts to execute business logic and publish events to the event bus, all within the same database transaction.

Challenges:

Event Publication Preceding Transaction Commit: When events are published to the event bus as part of the database transaction, there's a risk that the transaction might not successfully commit even after the event has been sent out. This situation can occur due to various reasons, such as issues in completing subsequent operations within the same transaction or a late-occurring failure in the database.
Inconsistent State Risks: If the transaction fails to commit after the event is published, it leads to a critical inconsistency. The event bus carries an event that suggests a change or action that, in reality, the database did not finalize. This discrepancy can lead to serious data integrity issues, as other services responding to the event may act on data that doesn't accurately reflect the current state of the system.
Complex Error Handling: The application must be prepared to handle these scenarios, where the event is out in the world, but the corresponding transaction failed. This requires sophisticated error handling and potentially complex rollback mechanisms, not just for the database changes, but also for any actions taken by services that consumed the premature event.

In the past, one common approach to address such issues was through distributed transactions, like the two-phase commit (2PC) protocol.
2PC was designed to ensure atomicity across multiple database systems,
attempting to maintain data consistency by coordinating a commit or rollback across all involved systems.
However, this approach comes with significant drawbacks, particularly in distributed and microservice-based architectures:

Performance Overhead: The 2PC protocol is known for being resource-intensive and can significantly impact system performance, particularly in high-throughput environments.
Complexity and Scalability Issues: Managing and coordinating distributed transactions across multiple systems adds a layer of complexity and can become a bottleneck in scalable systems.
Lack of Support in Modern Ecosystems: Many modern message buses and RESTful APIs do not support the 2PC protocol, making it challenging to implement in contemporary microservices architectures.

The Real-World Implications of Naive Approaches

From a theoretical standpoint, the challenges with these naive approaches in integrating an event bus within an application may seem obvious.
However, drawing from my professional experience across various projects, I have observed these issues manifest repeatedly in real-world scenarios.
Whether it's dealing with distributed systems employing message buses, interacting with external systems through REST APIs, or managing multiple databases, a common error persists: a lack of clarity among developers regarding the consistency semantics of their applications.

In practice, this oversight leads to perplexing and hard-to-diagnose errors.
Developers often underestimate the complexity introduced by distributed environments and might not fully consider the implications of their chosen implementation strategy.
As a result, applications can exhibit erratic behavior, where the root cause is deeply embedded in the flawed interaction between different system components.
These issues can be especially challenging to debug and resolve, given the distributed nature of these systems and the often subtle interdependencies between their components.

The crux of the problem lies in the assumption that techniques and patterns used in monolithic or single-database applications can be directly applied to distributed systems.
This misconception overlooks the fundamental differences in how distributed systems handle data consistency and transaction management.
The consequences of this oversight are not just theoretical inconsistencies but tangible bugs and system failures that can have a significant impact on the reliability and integrity of the application.

This challenge highlights the inherent difficulty in ensuring atomicity and consistency when dealing with internal transactions and external communication within the same operational scope.
It underlines the necessity for a pattern like the Transactional Outbox, which offers a way to decouple these concerns,
ensuring that events are only published after the transaction’s successful commitment, thereby maintaining the integrity and consistency of the overall system.

Understanding the Transactional Outbox

The Transactional Outbox is a design pattern used in microservices architectures to ensure consistent and reliable message publishing in distributed systems.
At its core, the pattern solves a fundamental problem: how to update a database and publish messages/events to a message bus or event-driven system and keep the state consistent.

How It Works

Local Transaction with Outbox Table: Instead of directly publishing an event to the message bus, the application writes the event into a special table within the same database as part of the business logic transaction. This table is commonly referred to as the outbox.
Atomic Commit: The key to this pattern is the atomic commit. The changes to the business data and the insertion of the event into the outbox are done in the same database transaction. This ensures that either both are committed or neither, maintaining the atomic nature of operations.
Reliable Event Publishing: After the transaction commits, a separate process (which could be a service, a scheduled job, or a database trigger) reliably reads events from the outbox and publishes them to the message bus. This process ensures that each event is eventually published without being lost.
Event Deletion/Mark as Processed: Once the event is successfully published to the message bus, it is either deleted from the outbox or marked as processed, preventing it from being published again.

By using the transactional outbox, the application ensures that the state in the database and the published events are always consistent.
Moreover, the pattern exhibits a remarkable resilience to system failures. If the application crashes after committing the transaction but before the event is published, the separate process ensures that the event will still be published later.
Another significant advantage of the Transactional Outbox Pattern is the decoupling it offers. By separating the concerns of database transactions and event publishing, the pattern allows each to operate independently. This separation not only simplifies the architecture but also enhances scalability and maintainability.
It allows the system to evolve more flexibly, adapting to changes and scaling components independently without disrupting the fundamental operations of transaction management and event distribution.

Additionally, an often overlooked yet significant aspect of the Transactional Outbox Pattern is its efficiency,
particularly regarding the speed of writing to the outbox.
Writing an event to the outbox involves merely appending a row to a table, which is one of the fastest operations in database management. This efficiency means that the pattern does not impose a notable performance penalty.

Challenges and Solutions

Implementing the Transactional Outbox Pattern, while beneficial, brings its own set of challenges that need careful consideration and handling.

At least once delivery

When implementing the Transactional Outbox Pattern in a distributed system, it's important to understand that this pattern inherently guarantees at-least-once-delivery semantics.
At-least-once-delivery is a delivery approach where the system ensures that a message is definitely delivered to its destination at least once.
Since the delivery mechanism is designed to ensure that messages are not lost, it errs on the side of sending duplicates rather than risking a message not being delivered at all.
As a result, the receiving services or components must be equipped to identify and appropriately handle these duplicate messages.
This involves implementing logic to detect if a message has already been processed and ensuring that processing it again does not adversely affect the system's state or lead to incorrect operations.

A common approach to handle duplicate messages is implementing idempotency at the technical level.
This involves recording the IDs of consumed events and checking against this record before processing new events.
If an incoming event's ID matches one that has already been processed, it is ignored, thus preventing duplicate processing.

A more sophisticated approach is to design idempotency based on business rules.
Instead of just tracking event IDs, this approach involves designing the business logic and data models in such a way that repeating the same operation does not have adverse effects.
For example, applying the same update twice would result in the same state as applying it once. This method requires a deeper understanding of the business context but can lead to a more robust solution.

Consider the process of stock reservation in a warehouse.
In this scenario, every time an item is reserved for a shipping order, the system communicates not just the quantity reserved, but also the remaining quantity in stock.
When an article is reserved, the system sends an update that includes both the number of pieces reserved and the current stock level.
This comprehensive update provides a complete picture of the stock status after each transaction.
With the detailed stock information at hand, a replenishment component in the system can easily apply business rules to monitor stock levels.
For instance, if the system receives multiple updates about the same reservation due to a duplicate message, it can discern that the stock level hasn't further decreased since the quantity and remaining stock are consistently reported.
By structuring the update messages in this way, applying the same stock reservation update more than once does not lead to incorrect stock calculations.
The system remains aware of the actual stock level, regardless of duplicate messages, ensuring that replenishment decisions are based on accurate and current information.

Eventual Consistency

Eventual consistency refers to a model where the state of the system is not immediately synchronized across all its components following a change.
Instead, these changes propagate through the system over time, resulting in a lag during which different services might have different views of the system’s state.
Eventually, however, all services reach a consistent state, hence the term "eventual consistency."
This concept contrasts with "strong consistency," where a system ensures immediate consistency across all nodes after any operation.

In the realm of distributed systems, particularly those transitioning from traditional SQL database architectures, the concept of eventual consistency often poses a significant mental shift for developers.
Accustomed to the strong consistency guarantees of SQL databases, where the recent write operation is immediately visible to all subsequent read operations, developers may find the notion of eventual consistency unsettling.
This model, where data is not immediately consistent across all services of the system but becomes consistent over time, can seem counterintuitive and raise concerns about data integrity and application behavior.

In discussing the eventual consistency inherent in the Transactional Outbox Pattern, it's important to clarify a common misconception:
the introduction of the Transactional Outbox Pattern does not, in itself, bring about eventual consistency in a system.
Rather, if your system already incorporates an event bus, it is by design operating under the principles of eventual consistency.

The nature of an event bus in a distributed architecture inherently implies that there's a delay between when an event is generated and when it is received and processed by other parts of the system.
When an event is published to the bus, it doesn't instantly synchronize the state across all services and components.
Instead, these components update their state as and when they process the event, leading to a period during which different parts of the system may have slightly different views of the overall state.

However, a closer examination of real-world processes reveals that eventual consistency is not only natural but often the norm. Consider the example of ordering a meal in a restaurant:

Ordering Process: When you place an order with the waiter, there's no immediate check with the kitchen to confirm if every ingredient is available. The waiter takes orders from multiple tables, and only later are these orders processed by the kitchen.
Handling Unavailability: If an ingredient is unavailable, the waiter informs you later, and you're given the option to choose an alternative. This delay in information and the need to adapt is a classic example of eventual consistency at play.

The same principle applies to many everyday business processes:

Hotel Bookings: When you book a hotel room online, there's often a delay before the booking is confirmed. During high-demand periods, you might even receive a notification later that the room is no longer available, prompting you to find an alternative.
Online Shopping: Shopping on platforms like Amazon involves a period where the order is processed, and only later do you get confirmation. In some cases, you might be notified of out-of-stock items or shipping delays.
Flight Bookings: Booking a flight doesn’t guarantee a seat until various checks are completed, and occasionally, overbookings lead to last-minute changes.

Just as these real-world scenarios adapt and manage eventual consistency, applications in distributed systems must be designed to handle it gracefully.
It's essential to recognize that eventual consistency is not a flaw but a characteristic of distributed systems that, when properly managed, can lead to greater scalability and resilience.
Applications should be designed to expect and handle delays in data synchronization and to manage the exceptions that arise when things don't go as expected.

Developers can draw lessons from these real-world analogies to better understand and implement eventual consistency in their systems.
By acknowledging that not all processes require or benefit from strong consistency, and by designing systems that can handle inconsistencies and adapt as needed, developers can build more robust, flexible, and scalable applications.
This mindset shift is crucial in successfully navigating the complexities of modern distributed systems.

Hands-On Guide: Implementing the Transactional Outbox Pattern with Spring Boot and RabbitMQ

We will demonstrate how to implement the Transactional Outbox Pattern in a Spring Boot application, integrated with RabbitMQ for message handling.
For the transactional outbox mechanism, we'll utilize the Transaction Outbox library,
a well-curated Java library that offers seamless integration with Spring Boot and other frameworks like Quarkus.

The sample project provides an REST endpoint for creating user profiles.
Upon creating a user, it publishes a UserCreatedEvent to RabbitMQ, demonstrating the Transactional Outbox Pattern's capabilities.

1. Cloning the Repository

Our example project repository is available at SWCode Samples on GitHub.
This repository provides a "plug and play" setup with a docker-compose file that includes a PostgreSQL database and a RabbitMQ container, offering a hassle-free way to get the environment up and running.

git clone https://github.com/sw-code/swcode-samples.git
cd swcode-samples/transactional-outbox

Running Application

Starting the Application: Use Docker to start the PostgreSQL and RabbitMQ services docker-comose up, and then run the Spring Boot application.
Creating Users: To test user creation and the subsequent event publishing, you can use the /users endpoint. This will trigger the flow from the user creation in the UserService to the publishing and handling of the UserCreatedEvent.
Observing Event Handling: As you create users through the endpoint, keep an eye on the logs of the RabbitMqConsumer. You should see the UserCreatedEvent payloads being logged, indicating that the events are successfully being consumed from RabbitMQ.

For convenience, the repository includes a sample-requests.http file,
which provides easy access to all the endpoints if you are using IntelliJ IDEA.
This file simplifies the process of making requests to the application for testing purposes.

Explore the Code

The UserController controller provides an endpoint to create users.

@PostMapping("/users")
fun createUser(@RequestBody request: UserCreateRequestDto): User {
    return userService.createUser(request.name)
}

When a user is created, a UserCreatedEvent is published.
This is done using the OutboxPublisher to persist the event in the outbox first, and then forward it to RabbitMQ.

class UserService(
    private val userRepository: UserRepository, 
    private val eventPublisher: EventPublisher) {

    fun createUser(name: String): User {
        return userRepository.save(User(UUID.randomUUID(), name)).also {
            eventPublisher.publish(UserCreatedEvent(it.id, it.name))
        }
    }
}

class OutboxPublisher(private val outbox: TransactionOutbox): EventPublisher {
    override fun publish(event: DomainEvent) {
        outbox.schedule(RabbitMqSender::class.java).send(event)
    }
}

The schedule method leads to the persisting of the task, which involves sending the event to RabbitMQ.
This task is stored in the outbox.
Then, it schedules the execution of this task for after the current transaction has been committed.
This ensures that the event is only sent after the transaction is successfully completed.

In the context of handling potential failures during event sending after a transaction, the TransactionalOutboxScheduler class provides a robust fallback mechanism.
This class periodically executes a run method that continuously calls transactionOutbox.flush().
The flush method in TransactionOutbox is responsible for finding and retrying events that haven't been successfully executed after a certain amount of time.
It specifically targets events that were initially set up to be executed but haven't been completed as expected and have not exceeded a predefined number of retry attempts.
This loop continues to invoke flush as long as it returns true, indicating there are events to be processed. Once flush returns false, signifying that there are no more events requiring attention, the loop terminates. This approach ensures that all pending events are addressed promptly.

class TransactionalOutboxScheduler(private val transactionOutbox: TransactionOutbox) {
    @Scheduled(fixedDelay = 2000)
    fun run() {
        while (transactionOutbox.flush()) {
            // NOP
        }
    }
}

To simulate errors in event sending, utilize the /failing-events endpoint.
This endpoint generates a FailingEvent and attempts to send it to RabbitMQ.
The RabbitMqSender is programmed to mimic an error by intentionally failing once for each instance of FailingEvent.

Play around and have fun exploring the outbox pattern!

Conclusion

As we conclude our exploration of the Transactional Outbox Pattern,
it's evident that this pattern is more than just a technical solution—it's
a strategic approach to ensure data consistency and reliability in the increasingly complex landscape of distributed systems and microservices.

The key takeaway from this discussion is the importance of understanding and adapting to the nuances of distributed systems.
The shift from traditional SQL database architectures to distributed models, while challenging, opens up new avenues for building scalable, resilient, and flexible applications.
The Transactional Outbox Pattern is a testament to the innovative solutions that emerge when we rethink how data consistency and reliability should be handled in modern application development.

As you implement this pattern in your projects, remember that the journey towards mastering distributed systems is ongoing.
The Transactional Outbox Pattern is just one of the many tools in your arsenal, equipping you to tackle the challenges and leverage the opportunities presented by distributed architectures.
Stay curious, continue exploring, and most importantly, keep building systems that not only meet today's demands but are also prepared for tomorrow's challenges.

References and Further Reading

Microservices.io - Transactional Outbox Pattern An excellent resource for a concise overview of the pattern and its applications in microservices.
Designing Data-Intensive Applications by Martin Kleppmann: This book offers an in-depth exploration of various patterns and models used in designing modern data-intensive applications, including considerations for consistency and reliability in distributed systems.
Building Microservices by Sam Newman: A comprehensive guide to designing, building, and maintaining microservices-based applications, with insights into communication patterns and data management in distributed environments.
Transaction Outbox Library Dive into the source code and documentation of the Transaction Outbox library to understand its implementation and how it facilitates the Transactional Outbox Pattern.

Google's Zanzibar and Beyond: A Deep Dive into Relation-based Authorization

SWCode — Tue, 09 Jan 2024 07:46:41 +0000

Dive into the world of online permissions and it quickly becomes clear: it's all about the connections you hold. Imagine being at an upscale gala. It's not just about your invitation card that gets you through the door. It's about who you know inside, the conversations you're a part of, and the circles you move in.

In the digital realm, it's similar. Access isn't always about direct permissions; it's about the intricate web of relationships that you're entangled in. Can you view this document? Well, it depends – are you linked to its creator, or part of the team that worked on it?

It's this nuanced dance of relationships that defines relation-based authorization. Less of a straightforward checklist and more of a dynamic network of ties, it offers a richer, deeper layer to deciding "Who gets to do what?".

TL;DR: We explored the foundational concepts of relation-based authorization models, tracing back to the roots in tech giants like Google.
With the power of Ory Keto and Spring Boot, we crafted a hands-on example to bring the theory to life.
Want to dive straight into the action? Check out our detailed hands-on integration: A Hands-on Integration of Ory Keto and Spring Boot.

The Google Zanzibar Paper and its Influence on Rapid Authorization Framework Development

In recent years, a notable event has had a profound influence on the world of authorization frameworks. This event was the release of the Google Zanzibar paper, a highly insightful document that details Google's approach to scalable, efficient authorization.

Google's Zanzibar system is a testament to the prowess of modern technology. According to the paper, Zanzibar handles access management for an astonishing number of resources - we're talking trillions of objects - used by hundreds of millions of users. What makes these figures even more impressive is the wide array of client services that rely on Zanzibar. Renowned platforms like Google Maps, Google Drive, and YouTube, all depend on Zanzibar for their authorization needs.

But Zanzibar doesn't just handle this enormous load, it does so rapidly and reliably. The system boasts a remarkable response time, with 95% of requests being handled within just 10 milliseconds and 99% within 20 milliseconds. On top of this, Zanzibar has proven its reliability, maintaining an exceptional availability rate of 99.999% over the past three years.

However, the impact of the Zanzibar paper extends beyond these impressive technical capabilities. The release of the paper pushed a flurry of development in the authorization realm, resulting in a variety of open-source projects and SaaS offerings inspired by Google's Zanzibar, including Ory Keto, AuthZed, and OpenFGA.

But why has this paper sparked such a wave of development? The answer lies in a combination of factors:

Scalability: As mentioned, Zanzibar's ability to handle authorization for trillions of objects by hundreds of millions of users is awe-inspiring. As more businesses adopt microservices and cloud-native architectures, the need for scalable authorization solutions becomes increasingly pressing. Zanzibar offers a proven blueprint for achieving such scalability.
Flexibility: The Zanzibar model, termed Global, Consistent Authorization System, is not tied to specific business logic or policies. This flexibility makes it broadly applicable across various domains and use cases, inspiring many to leverage it in their contexts.
Efficiency: Efficiency in authorization is crucial. Zanzibar's use of data sharding and re-sharding for optimal use of resources has been an inspiration for many developing authorization frameworks.
Reliability: Google's paper emphasizes Zanzibar's ability to provide consistent authorization decisions, even in the face of network partitions and system failures. This high level of reliability is a key factor in Zanzibar's appeal.
Innovation: Finally, the Zanzibar paper represents a significant leap in the way we think about and implement authorization. Its innovative approach to handling access control at a massive scale has piqued the interest of many in the field, spurring them to explore and adopt similar strategies.

The widespread interest and rapid development sparked by the Zanzibar paper are testaments to its game-changing potential in the realm of scalable authorization.
As we at SWCode embark on our journey to implement a scalable authorization architecture, we too are inspired by the lessons gleaned from Google's Zanzibar.

Unpacking Zanzibar's Relation-Tuple-Based Model

At the core of Google's Zanzibar system is a surprisingly simple yet powerful model: relation tuples. A relation tuple is a small record that describes a relationship, in the form of namespace:object#relation@subject. In this format:

namespace represents the type of the object, such as document or folder.
object refers to a unique identifier for a specific instance of that type, like a specific document ID or folder ID.
relation relation signifies the type of relationship between the object and the subject, such as owner or viewer.
subject can either be a user or another relation tuple, allowing for complex, nested relationships.

For example, a relation tuple might look like this: document:readme#owner@user456, meaning that user456 is the owner of readme in the document namespace.
Expanding on this, the subject in a relation tuple doesn't necessarily need to be a user. It could also be another relation tuple. This introduces the ability to create relationships between objects, as exemplified by document:readme#parent@folder:A, indicating that the folder A is the parent of the document readme.

By combining these simple elements, Zanzibar can describe complex hierarchical relationships and permissions across vast numbers of objects and users. This granularity of control is one of the key reasons for Zanzibar's flexibility and scalability, and it's a fundamental concept that has been adopted by Zanzibar-inspired frameworks mentioned before.

Expanding on this concept, Zanzibar's relationships weave a structured graph connecting objects. Its strength lies in the straightforward approach: for authorization validation, Zanzibar merely verifies the presence of a certain relationship. Using our previous example, it would simply verify if there's an owner connection between the document readme and user456. To dive deeper into this mechanism and gain a clearer understanding, continue to our next chapter where we dissect the Ory Keto framework, a system inspired by Zanzibar.

It's worth noting that this explanation is somewhat simplified for clarity. The original Zanzibar paper is a scientific work and delves into greater theoretical depth, but the above should provide readers with a foundational understanding of the core concept.

Ory Keto

Ory Keto, written in Go, is an open-source authorization system that pivots around the concept of relation tuples, echoing the foundational principles of Google's Zanzibar. Ory Keto is packaged as a standalone service, offering both HTTPS and gRPC APIs for seamless integration across diverse systems. Recognizing the varied deployment needs of organizations, Ory provides Keto both as a Software as a Service (SaaS) offering and a self-hosted solution, ensuring adaptability across different infrastructural requirements.

While conceptualizing Ory Keto, certain guiding principles and assumptions were maintained to ensure its efficacy:

Simplicity in Management: The system was built on the hypothesis that no individual or team would dedicate their full time to configuring an authorization system.
Intuitive Nature: Keto is designed to be self-explanatory, minimizing the learning curve and ensuring that users find the system approachable.
Familiarity: By adhering to familiar paradigms and practices, Keto reduces the onboarding time for developers and system administrators.
Ease of Modification: Any changes or updates to the authorization configurations can be carried out with high confidence, ensuring system stability.
Editorial Support: Thanks to the Ory Permission Language, which is a subset of Typescript, Keto is compatible with numerous editors. This ensures users have the flexibility and familiarity in their choice of tools.

Core Concepts of Ory Keto

Adoption of the Relation Tuple Model: Ory Keto adapts the Zanzibar-inspired relation tuples model. As we’ve previously elaborated, a relation tuple succinctly describes a relationship in a format like namespace:object#relation@subject. Through these tuples, Ory Keto can express and evaluate intricate hierarchical relationships and permissions across a multitude of objects and users, ensuring a granular and fine-tuned access control mechanism.
Ory Permission Language (OPL): OPL, envisioned as a subset of TypeScript, is Ory Keto's method to detail and evaluate policies within its framework. Despite the internal engine of Keto being rooted in Go, OPL offers developers a familiar TypeScript-like syntax to outline intricate authorization policies seamlessly and precisely.

OPL integrates namespaces and rules to aid in permission evaluations by exploring relations, though the actual tracing is managed by Keto internally.
While Google Zanzibar's model was framed scientifically for researchers and experts, Ory Keto uses TypeScript-based OPL, targeting a broader developer audience for a more user-friendly experience.

Ory Permission Language

In Ory Keto, defining namespaces and relations is at the heart of the model. Let's illustrate this with a hands-on example featuring two namespaces: Document and User.

Within the Document namespace, a user is designated as either an owner, a viewer, or an editor of a document through specific relations.

The core of authorization revolves around permissions, such as view or edit, rather than checking relations like 'owner' or 'editor'. The concrete permission is checked against the relationships. Within the Ory Permission Language, permissions are declared as functions inside the permits property of the corresponding namespace.

The code provided outlines a basic permission structure, which Keto subsequently processes to generate relation tuples internally.

class User implements Namespace {}

class Document implements Namespace {
  related: {
    owners: User[]
    editors: User[]
    viewers: User[]
  }

  permits = {
    view: (ctx: Context): boolean =>
      this.related.viewers.includes(ctx.subject) ||
      this.related.editors.includes(ctx.subject) ||
      this.related.owners.includes(ctx.subject)
  }  
}

Revisiting Jane with Ory Keto Relations

In our preceding discussions on Role-Based Access Control (RBAC), we introduced Jane, an auditor working for GlobeCorp. Jane was granted 'readonly' access to GlobeCorp's financial data and held the admin role at EcoLife. This time around, let's reframe Jane’s scenario using Ory Keto’s relation tuple framework.

Here's a quick recap:

GlobeCorp is a global conglomerate
TechNovelties and EcoLife are direct subsidiaries of GlobeCorp
GreenHealth is a specialized vertical of EcoLife

In the OPL, all organizations will be clustered within a singular 'Organization' namespace, with functional dependencies illustrated using the 'parents' relations.

class Organization implements Namespace {
  related: {
    editors: User[]
    viewers: User[]
    parents: Organization[]
  }

  permits = {
    view: (ctx: Context): boolean =>
      this.related.viewers.includes(ctx.subject) ||
      this.related.editors.includes(ctx.subject) ||
      this.related.parents.traverse((p) => p.permits.view(ctx))
    edit: (ctx: Context): boolean =>
      this.related.editors.includes(ctx.subject) ||
      this.related.parents.traverse((p) => p.permits.edit(ctx))      
  }  
}

The diagram effectively visualizes Jane's direct relations and how they cascade down the organization hierarchy using the parents relation in the context of Ory Keto's permission system.

Given this arrangement and the provided OPL:

Jane has a viewer relation to GlobeCorp, allowing her 'readonly' access to its data. This relationship extends hierarchically, meaning Jane can also view data of the companies that are subsidiaries or children of GlobeCorp due to the parents relation, which are TechNovelties and EcoLife.
Jane, due to her editors relation with EcoLife, is endowed with edit access to its data. Leveraging the hierarchical structure, this access naturally extends to GreenHealth, EcoLife's subsidiary.

Creating Relations in Ory Keto via REST API

Defining namespaces, relations, and permission rules forms the configuration phase in Ory Keto.
Once this setup is complete, you can then dynamically create the actual relations to express relationships between various entities.
This is achieved on-demand via REST API calls.

To illustrate, let's define a relation that indicates EcoLife is a child organization of GlobalCorp.
You can do this by making a PUT request as follows:

curl --location --request PUT 'http://localhost:4467/admin/relation-tuples' \
--header 'Content-Type: application/json' \
--data '{
      "namespace": "Organization",
      "object": "EcoLife",
      "relation": "parents",
      "subject_set": {
        "namespace": "Organization",
        "object": "GlobalCorp",
        "relation": ""
      }
    }'

In this example, the relation parents for the object EcoLife points to GlobalCorp, indicating the hierarchical relationship between the two organizations.

Likewise, to establish a relationship between a user and an object, you would follow a similar approach.
For instance, if you want to specify that the user jane has a viewers relation to the organization EcoLife, you would craft a REST call as such:

curl --location --request PUT 'http://localhost:4467/admin/relation-tuples' \
--header 'Content-Type: application/json' \
--data '{
    "namespace": "Organization",
    "object": "GlobalCorp",
    "relation": "viewers",
    "subject_id": "jane"
  }'

In the subsequent hands-on section, we provide a comprehensive configuration setup for Ory Keto, ensuring that you have all the tools and information at your disposal to effectively establish relations using the REST API.

A Hands-on Integration of Ory Keto and Spring Boot

Now that you've had a sip of the theory behind relation-based authorization, it's time to roll up your sleeves and witness it in action. Our destination? A realistic scenario involving the integration of Ory Keto and Spring Boot.

Step 1: Cloning the Repository

To kick things off, we have laid out everything on a silver platter for you.
By cloning our repository, you get access to a fully equipped application for our use case and docker-compose file:

A PostgreSQL database to store your data.
Keycloak, already set up with pre-configured users to handle authentication.
Ory Keto, primed with an OPL definition, ensuring authorization processes are streamlined right from the get-go.

git clone https://github.com/sw-code/swcode-samples.git
cd swcode-samples/authz/authz-keto

Step 2: Explore the Code

Within the authz-keto directory, you'll stumble upon our representation of the Jane and GlobalCorp scenario.
Remember Jane and her relationship with GlobalCorp and its sub-organizations? Yup, that's what we're bringing to life here.

Now, let's dive deep into the heart of our application: the OrganizationController.
Two distinct endpoints are waiting to unveil the story of relation-based authorization in action.

Viewing an Organization:

@GetMapping("/organizations/{organizationId}")
@PreAuthorize("hasPermission(#organizationId, 'Organization', 'view')")
fun getOrganization(@PathVariable organizationId: String): ResponseEntity<OrganizationDto>

The beacon guiding us here is the @PreAuthorize annotation.
Notice the "hasPermission" argument? This expression is the gatekeeper, determining if the user can view the specified Organization.

Updating an Organization:

@PutMapping("/organizations/{organizationId}")
@PreAuthorize("hasPermission(#organizationId, 'Organization', 'edit')")
fun updateOrganization(
    @PathVariable organizationId: String,
    @RequestBody request: OrganizationUpdateDto
): ResponseEntity<OrganizationDto>

Again, the @PreAuthorize annotation stands guard. This time, however, it's on the lookout for those who wish to edit the Organization.

But what brews behind this magical curtain of @PreAuthorize: The KetoPermissionEvaluator.
This component, a standard approach in Spring Security, implements the PermissionEvaluator interface and is instrumental in collaborating with Ory Keto to check permissions.

override fun hasPermission(
    authentication: Authentication,
    targetId: Serializable,
    targetType: String,
    permission: Any
): Boolean {
    return ketoKetoClient.checkPermission(authentication.name, targetType, targetId as String, permission as String)
}

The hasPermission method takes in four arguments:

authentication: This is an object provided by Spring Security, representing the current user's authentication information.
targetId: A Serializable ID, in this context, typically the identifier for a specific entity within the system.
targetType: This string represents the type or category of the entity. In our case, it corresponds to the namespace in Ory Keto.
permission: Represents the specific action or permission we want to check, such as view or edit.

Within the method, a call is made to ketoKetoClient.checkPermission(...), which, in turn, communicates with Ory Keto via REST API.

In essence, this method is asking Ory Keto: "Given this user and this specific entity in the provided namespace, do they have the specified permission?"
The response is a Boolean, indicating either permission granted or denied.

An important detail to note is our choice of using the username directly from the authentication, instead of the typically recommended user ID.
This decision was made purely for illustrative clarity.
While in a production scenario, leveraging UUIDs as the subject would be the standard approach, here we've prioritized a straightforward demonstration to ensure a clear and uncomplicated understanding.

Step 3: Setup & Configuration

To initialize the necessary test data, we utilize the SampleDataInitializer class.
This class orchestrates the creation of our well-known GlobalCorp and its associated organizational hierarchies upon application launch.
Additionally, it ensures Jane is granted the appropriate access.

While user-to-organization relationships are forged directly within this class as part of sample data,
the relationships defining the organizational hierarchies (parents) are dynamically established.
Whenever a new organization is brought to life in our system, an OrganizationCreatedEvent is emitted.
This event is the catalyst, triggering the creation of parents relationships in Ory Keto when necessary.

Step 4: Launch & Play

Before launching the application, make sure you've initiated the necessary infrastructure using the supplied docker-compose file.

Once the application is up and running, test out the endpoints.
Within the repository, there's a Postman collection tailor-made for this hands-on.
This collection includes everything you'll need, from fetching a token from Keycloak for a user to making calls to our endpoints.
Start by trying to retrieve GlobalCorp's data as Jane using GET /organizations/1.
Since Jane holds viewer permissions, this attempt will succeed.
But if you try to update this data? Expect a "Forbidden" response.
Remember, though, that Jane has editing rights for EcoLife.
This means you can update its information using PUT /organizations/3.
The same goes for its sub-organization, GreenHealth, accessible via PUT /organizations/4.

Certainly, you might've spotted endpoints like GET /organizations and POST /organizations that haven't been enveloped in our authorization discussion yet.
These endpoints, unlike the others, don't target a specific entity and thus require a slightly different approach.
However, fret not; the path forward with Keto will be smooth and intuitive.
The upcoming posts, where we'll delve deeper into these aspects and navigate through other exciting challenges ahead.

Conclusion

In this comprehensive exploration, we dove into the fascinating world of relation-based authorization models.
With its roots deeply embedded in Google's Zanzibar, this approach presents a robust paradigm shift in how we perceive and implement permissions in the digital realm.

We unveiled a ready-to-use example coupling Spring Boot and Ory Keto, offering readers a tangible demonstration of these concepts in action.
This elucidated how the various building blocks, when seamlessly integrated, can create an effective and secure authorization system.

As we step further into this domain, our journey is only just beginning.
Looking ahead, there are several compelling challenges to address:
understanding the dynamics of users and roles, managing blocking relations for exception handling, and tackling the demands of searching and filtering with permissions in mind.

The world of authorization is vast and ever-evolving, and we're just scratching the surface.
Stay with us as we delve deeper into its depths in our forthcoming discussions.

Requirements for a scalable Authorization Architecture

SWCode — Wed, 27 Dec 2023 15:37:01 +0000

Our prior exploration, Authorization Basics of our Scaling Authz blog series, provided a primer on the foundational elements of authorization. Yet as organizations evolve, so too must the systems that safeguard their data and resources. This article delves deeper, defining essential criteria for an adaptive authorization system and starting our journey into their detailed exploration.

Requirements

When designing an authorization system that scales, it's crucial to prioritize functional requirements that guarantee both performance and user satisfaction. Here are the foundational prerequisites for creating such a system.

1. Fine-Grained Access Management

Every individual or system component should have tailored access rights. Think of it as ensuring that each employee in a vast company has the right keycard for the specific rooms they need to enter, nothing more and nothing less. Moreover, as businesses evolve and expand, so do their requirements. What starts as a simple access model can soon become a complex web of permissions. Therefore, it's crucial to have an authorization system that not only adapts to these growing needs but is also designed to handle intricate access patterns from the outset. Building flexibility into your system from day one ensures you're not caught off guard when the business asks for a new, nuanced access rule tomorrow.

2. Built with Security Front and Center

Authorization, at its core, stands as a bastion against unauthorized access to critical resources. But to ensure robustness, the underlying framework employed within the authorization system becomes pivotal. Let's examine some of the foundational pillars that ensure the security of such a framework:

Secure Coding Practices: As with any software component, the framework should be developed with best coding practices to avoid common vulnerabilities such as code injection or buffer overflows. Regular security audits and adherence to guidelines like the OWASP Top Ten provide a layer of assurance.
Extensibility without Exposure: A scalable authorization architecture must allow for extensions and integrations, but each integration point poses a potential risk. The framework must support secure APIs and integrations, ensuring that extensibility doesn't equate to exposure.
Strong Encryption Standards: Data, especially concerning user roles, permissions, and policies, is critical. The chosen framework should support robust encryption both at rest and in transit, using industry-approved algorithms and protocols.
Minimal Attack Surface: The framework should be lightweight with no unnecessary components or plugins that could be exploited. This "minimalism" reduces the attack surface, making it tougher for adversaries to find vulnerabilities.
Regular Patching & Updates: Security isn't a one-time affair. As new vulnerabilities are discovered, the framework must be actively maintained and patched, ensuring that known issues are promptly addressed.
Fine-Grained Auditing and Logging: Being able to trace back actions can be a lifesaver when it comes to detecting and investigating security incidents. The framework should offer detailed auditing capabilities, capturing who did what and when.
Role and Policy Lifecycle Management: Over time, roles may become obsolete, or policies might need refining. The framework should offer mechanisms to safely deprecate roles or adjust policies, ensuring that outdated rules don't become security loopholes.

3. Technology Agnosticism

An effective authorization system must be a universal translator, smoothly integrating across diverse technological landscapes. It shouldn't matter whether one department uses Python and another uses Java. Authorization should be a seamless bridge.

4. Low Latency

Every time a user interacts with a system, there's a series of backend operations taking place, many of which are invisible to the user. Among these operations is the crucial step of verifying whether the user has the right to perform the action they're attempting. This verification, the permission check, is a constant companion to every API call, database fetch, or file access. For a seamless user experience, these checks must be lightning fast. Any delay, no matter how slight, amplifies with every interaction and request. The cumulative result is a noticeable increase in response time, which could make the difference between an efficient, enjoyable user experience and a slow, frustrating one. Thus, an authorization system must be designed with latency at the forefront, ensuring that these repeated checks don't become bottlenecks in the user's journey.

5. Cloud-Compatibility

In today's digital landscape, scalability is synonymous with cloud compatibility. Only cloud environments offer the elasticity and on-demand resources necessary to meet the volatile demands of a growing user base. Without a cloud-native design, the architecture would struggle to efficiently scale, both in terms of handling large numbers of users and the vast array of resources they access. It's worth noting: if your architecture doesn't need to thrive in the cloud, you probably don't need a massively scalable authorization system in the first place. Thus, designing for cloud compatibility isn't just about keeping up with technological trends; it's about laying the foundation for the scalable, high-performance system that modern applications demand.

6. Consistent Authorization Decisions:

Consistency in authorization is paramount. Let's consider two contrasting scenarios:

A user has been restricted from accessing a sensitive resource, perhaps due to a change in their role or an observed security risk. If this revocation isn't consistently applied across all instances immediately, there's a potential window where the user can still access this data. Such inconsistencies can lead to unintentional data leaks, unauthorized actions, or breaches.
Conversely, imagine a user who has just been granted access to specific resources, perhaps as a result of a promotion or a new project assignment. If the system doesn't promptly recognize and enforce this updated permission, the user may face unnecessary access denial errors. This not only hampers their workflow but can also be a significant source of frustration, leading to decreased productivity and satisfaction. In both scenarios, any change in access rights should be instantly reflected across the system, ensuring that access controls are accurate and reliable at all times, no matter where or when the access request is made.

7. Simplicity in Maintenance

Just like a well-curated library where one can effortlessly locate and replace books, the components of an authorization system should be designed for ease of modification, replacement, or scaling. This foresight ensures longevity, adaptability, and can lead to cost savings in the long run.

Separation of Concerns: Authorization should be managed within its own dedicated layer, distinct from the business logic. Mixing business logic with authorization checks can lead to increased complexity, making updates or changes more prone to errors. Separating these concerns ensures that modifications to one do not inadvertently affect the other.
Transparent Mechanisms: The authorization mechanism should be transparent and replicable. This ensures that, as the team grows or changes, new developers can quickly understand and work with the system without a steep learning curve.
Flexibility from Business Requirements: While the authorization system must be robust and capable of handling a myriad of scenarios, it should be independent of ever-evolving business requirements. This means that changes in business rules or strategies shouldn't necessitate a reconfiguration of the authorization architecture. The system should be adaptable enough to accommodate these changes without a need for frequent overhauls.

8. Scalability to Match Growth

As organizations evolve, they often witness a surge in user numbers, resources, and the frequency of requests. An authorization system should not only be comparable to a well-trained athlete, capable of sprinting when demands peak, but also maintain its performance and resilience over extended periods.

To delve deeper into what scalability truly means in this context:

User Base Scalability: The system should handle a rapid increase in users seamlessly. A tenfold or even hundredfold surge in user numbers shouldn't cause the system to falter or slow down. It should be designed such that adding more users doesn't create an exponential increase in required resources or system load.
Adaptable to Expanding Business Needs: As a company grows, it often diversifies, bringing forth new business models, services, and use cases. The authorization system should be flexible enough to accommodate this evolution. Whether there's a need for integrating new services, managing more granular permissions, or interfacing with new technologies, the system should be ready.
Service Distribution: Modern businesses, especially tech-driven ones, often adopt microservice architectures or other distributed systems. This fragmentation can pose unique challenges to authorization, as permissions may have to be checked and reconciled across various services and databases. A scalable authorization system should, therefore, be prepared to operate efficiently across such distributed environments, ensuring consistent and timely access decisions.

9. Testability for Peace of Mind

Ensuring the robustness of an authorization system isn't just about design – it's about validation. Testability plays a pivotal role in this:

Continuous Validation: Integrating an authorization system's test suite with an external library or service is not merely a luxury; it's a necessity. With continuous testing, you can promptly catch any regressions or unexpected behavior changes. The immediacy of feedback accelerates the development process, all the while keeping systems in check.
Security Assurance: Security is a non-negotiable aspect of authorization, and a well-crafted test suite aids immensely in this regard. By thoroughly testing the various authorization scenarios, edge cases, and potential vulnerabilities, developers can confidently make changes and enhancements. The testing framework serves as a safety net, catching any potential breaches or inconsistencies before they go live. This setup empowers developers to focus on the primary task at hand, ensuring top-tier security without getting bogged down by the nitty-gritty of each potential risk.
Maintainability & Scalability: As your system grows and evolves, so do its intricacies. A comprehensive test suite ensures that changes, no matter how minor or significant, don't inadvertently disrupt existing functionalities. This not only maintains the health of the system but also scales its resilience. Developers can swiftly integrate new features or scale existing ones, all the while having the assurance that the rest of the system remains unaffected.

Tackling the Essentials

In our quest to establish a solid foundation for scalable authorization, we will embark on a journey through each key requirement. These requirements aren't just academic or theoretical constructs; they represent the core of what it takes to make authorization truly effective and scalable in real-world scenarios. Our exploration begins with arguably one of the most crucial facets: Fine-Grained Access Management.

Fine-Grained Access Management: The Heart of Modern Authorization

When discussing authorization, particularly in complex and evolving systems, the need for fine-grained access management becomes paramount. It's no longer sufficient to rely on broad strokes and generalized roles. As business requirements expand, so too does the need for a more nuanced approach to permissions. Let's delve deeper into the complexities of this and understand the nuances.

The Limitations of Flat Roles

When we talk about flat roles, we're typically referring to systems where users are assigned broad, general roles such as "admin", "editor", or "viewer". These roles, while simple and intuitive, come with their own set of challenges when implemented within intricate organizational setups.

In real-world systems, developers often grapple with the limitations of these generalized roles by encoding information directly into the role name. A prime example would be designations like "editor:organization_a". While this might seem like a handy workaround initially, it quickly compounds the complexity of the system and detracts from the initial simplicity that flat roles promised.

These flat roles are easily understood but come with a host of limitations:

Inadequate Representation of Complex Hierarchies: In most real-world scenarios, resources and their access patterns are not flat. They follow a complex hierarchy, often with nested levels. For instance, consider a multinational corporation with departments, sub-departments, teams, and sub-teams. A flat role might easily allow access to an entire department but struggles when you need to provide access only to a specific sub-team within that department.
Lack of Flexibility: With flat roles, the permissions are typically rigid. If you need to give someone access to a resource that doesn't align neatly with these predefined roles, you either risk over-privileging them (by giving them a broader role) or under-privileging them (by denying access they might legitimately need).
Maintenance Challenges: Over time, as more exceptions and one-off permissions are added to accommodate the inadequacies of flat roles, the system becomes increasingly challenging to manage and maintain. This might lead to inefficient practices like creating a multitude of narrowly defined roles, exacerbating the issue of role explosion.

Role Explosion

As systems aim for more granularity, there's a risk of over-complicating things. When every new business need results in a new role being created, you end up with 'role explosion'. This brings its own set of challenges:

Maintenance Nightmare: With hundreds of roles, managing and updating them becomes a Herculean task. It's easy to lose track of what each role represents.
Decreased Security: The more roles you have, the harder it is to ensure each is correctly configured. This could lead to security gaps.
Cognitive Overload: For those managing these roles, remembering and understanding the function and scope of each becomes a challenge.
Token Bloat: In flat role-based systems, especially those leaning on IDPs, roles are commonly embedded as claims within JWT tokens. Role explosion causes these tokens to become bloated. Consequently, these oversized tokens may exceed the HTTP header size limitations set by some receiving systems, rendering them non-functional due to the token's size. This leads to requests being denied, creating potential operational disruptions.

Hierarchies and Their Importance

Role Hierarchies

A role hierarchy can be visualized as a pyramid or tree structure, where roles inherit permissions from roles beneath them. In a hierarchical role-based system, each role can encompass the permissions of roles "below" it, making the management of permissions more modular and reducing redundancy.

Example: In a corporate setting, we might have:

Employee: Basic access to company resources, such as the intranet or time tracking system.
Manager: Inherits all permissions of the Employee role, but also has added permissions like approving timesheets or access to team performance data.
Director: Inherits all permissions of both Employee and Manager roles. They might also have access to department-wide data, budgeting systems, and strategic planning tools.

In this model, if a basic permission, like accessing the company intranet, needed to be changed, you'd only need to modify it at the Employee level. Both Manager and Director roles would automatically inherit this change, thanks to the hierarchy.

Resource Hierarchies

Just as roles can be hierarchical, so can resources. Resources are often not flat; they're nested within each other, forming a structured hierarchy. Understanding and modeling these hierarchies is essential for fine-grained access control.

Let’s take the concept of resource hierarchies and apply it to a scenario we're all familiar with: the structure of a multi-national corporation.

GlobeCorp is a global conglomerate with its headquarters in New York. This is our “Parent Organization”.

Under GlobeCorp, there are several subsidiaries:

TechNovelties in Silicon Valley
EcoLife in London

Now, EcoLife itself has a specialized vertical focused on organic health products named GreenHealth, based in London as well.

Imagine an employee, let’s call her Jane. Jane is an auditor and is given a 'readonly' access role to the financial data of GlobeCorp.
Due to the hierarchy, this 'readonly' access should automatically extend to both TechNovelties and EcoLife, since they are subsidiaries of GlobeCorp.

However, Jane's role at EcoLife isn't just to audit. She's also responsible for operations. Thus, she's granted an 'admin' access role specifically for EcoLife.
This 'admin' access, following our hierarchical structure, should grant her broader privileges not just for EcoLife, but also for its child organization, GreenHealth.

Through this example, it’s evident how roles and access permissions can cascade through a hierarchical system.
It also underscores the importance of having a flexible authorization system that can discern between broad, inherited permissions and specific, assigned ones.

Exception Handling in Hierarchies

An efficient hierarchy-based system isn't merely about cascading permissions down the chain. It's equally crucial for such systems to provide the capability to make exceptions to the rule, preventing certain permissions even if the hierarchy would naturally grant them.

Let's revert back to our GlobeCorp scenario. Jane, who has a 'readonly' role at GlobeCorp, and an 'admin' role at EcoLife, might find herself in a tricky situation. Perhaps, due to new data protection regulations, she shouldn't have access to individual health records within GreenHealth, even though her 'admin' role for EcoLife would naturally grant her that access.

This scenario underscores the necessity of an authorization system that can make nuanced decisions. It should not only grant permissions based on hierarchical roles but also be adept at carving out exceptions based on specific business rules or compliance requirements. Such capabilities ensure that a company remains compliant with both its internal policies and external legal standards.

Embracing these hierarchical principles with the capacity for exception handling enables businesses to craft an authorization system that is structured yet adaptable, reflecting the real-world complexities of roles and resources.

Conclusion & What's Next

In "Requirements for a Scalable Authorization Architecture," we've taken a deep dive into the essential requirements, both technical and functional, needed for robust authorization. One of the key components we’ve highlighted is the ever-important realm of Fine-Grained Access Management, shedding light on its complexities and indispensable role in modern systems.

However, our journey is just beginning. As we move forward, subsequent articles will delve deeper into the other requirements, ensuring you have a comprehensive understanding of what it takes to build an effective, scalable, and secure authorization system.

Stay with us for more insights and deeper explorations into the expansive world of authorization.

Authorization Basics

SWCode — Mon, 18 Dec 2023 12:56:35 +0000

Welcome back to our Scaling Authz blog series, where we're documenting our journey toward a scalable and efficient authorization architecture.

In this second installment of our blog series, we'll delve into the foundational aspects of authorization. We'll explore various models from simple authentication-based authorization, through role-based access control, and up to complex relationship-based models.

To ensure a hands-on experience, each segment of our series will be accompanied by practical examples and implementation snippets. For those enthusiastic about diving deeper, all resources, code samples, and detailed walkthroughs are readily available in our dedicated repository. We believe that the synergy of theoretical knowledge and hands-on practice paves the way for profound understanding and mastery.

Authentication vs Authorization

Authentication: Think of it as the grand entrance—the main gateway to your application. It's about ascertaining "who" stands at the gates. Whether you employ the age-old username-password duo, modern techniques like OAuth, or even the increasingly popular OpenID Connect (often recognized in functionalities like "Login with Google"), the essence remains the same: confirming the identity of the entity trying to gain access.

Authorization: Once past the main gates, what parts of the castle is one permitted to explore? That's the realm of authorization. It delineates what you're allowed to do once you've made your entrance. The rooms you can enter, the secrets you can access, all fall under its domain.

In other words, authentication answers the question, "Who are you?" while authorization answers the question, "What are you allowed to do?"

In real-world applications, the distinctions between authentication and authorization begin to blur as they work hand-in-hand. Once a user's identity is verified, this information is frequently used to determine the permissions they have. For instance, after successfully authenticating, a system might use the user's ID or other attributes to determine which resources or actions they can access.

This series will primarily center on the nuances of authorization. However, we'll touch upon authentication when it's relevant to the topic at hand. We assume readers are already familiar with, or have in place, a basic authentication system, and there's a bunch of resources available for those looking to further delve into authentication mechanisms.

Authentication as Authorization

The simplest form of authorization is when the mere identity of a user is sufficient to grant access to resources. Such scenarios commonly surface in systems engineered for limited functionality or catering to a compact user base. In these instances, user authentication effectively doubles as a form of authorization.

A Glimpse into the Process

User Accesses Web Application: A user attempts to access a protected resource or service within a web application.
Authentication Required: Recognizing that the user is not yet authenticated, the application redirects the user to a login page.
User Authenticates: The user provides their credentials, such as a username and password.
Token Granted: Upon successful authentication, the identity provider issues a token to the user. This token acts as a proof of the user's identity and might contain claims, roles, or other pertinent information.
Token Utilized: The web application uses this token to grant the user access to its services and to communicate with backend APIs. It is essential to note that the backend verifies the integrity of this token to ensure it's genuine and hasn't been tampered with.

The flow above is reminiscent of the OAuth2 "Authorization Code Flow", albeit in a simplified manner. It primarily aims to illustrate the core concept of authentication. For those looking for an in-depth dive into OAuth2, consider exploring Auth0's comprehensive guide on the Authorization Code Flow.

Authorization Models

In the intricate world of authorization, one size seldom fits all. As applications grow and requirements become more nuanced, the mechanisms to decide who gets to do what need to evolve as well. The diversity of use-cases and the need for granularity have given rise to various models of authorization. These models serve as conceptual frameworks, providing structured methods to grant or deny access to resources based on different criteria. From the rudimentary setups that hinged primarily on lists to the more intricate designs that account for dynamic attributes and relationships, the evolution of these models mirrors the increasing complexity of our digital ecosystems.

Let's dive deeper into some of the prevailing models in authorization:

ACL (Access Control List): Stemming from its historical roots in the Unix file system, ACLs provide a list of permissions attached to an object. They are inherently a means to define which user have access to specific objects and the operations they can perform on them.
- Example: Think of ACLs like a club's guest list. If your name (and perhaps a specific criterion, like VIP) is on the list, you're allowed entry.
- Application Context: ACL would be apt for file-sharing platforms, where individual files/folders can be shared with specific users.
RBAC (Role-Based Access Control): A more structured approach, RBAC assigns permissions to specific roles rather than individual users. Users are then assigned to roles. This model offers simplicity and scalability, especially in larger systems where defining permissions for each user can become unwieldy.
- Example: Imagine a hospital. There are doctors, nurses, and administrators. Each role has different access levels in the patient database.
- Application Context: RBAC might be ideal for corporate intranets where access needs to be defined by job roles like manager, employee, HR, etc.
ABAC (Attribute-Based Access Control): A more dynamic approach, ABAC bases access decisions on policies derived from various attributes—be it of the user, the resource, or even the surrounding environment. For instance, a document might only be accessible during working hours or if a user is located in a specific country. This approach provides nuanced, context-aware authorization.
- Example: A confidential company document that's only accessible from company premises (using location as an attribute) and during working hours.
- Application Context: ABAC is well-suited for complex environments like financial systems where access to resources might depend on various dynamic factors.
ReBAC (Relationship-Based Access Control): Building on ABAC, ReBAC incorporates relationships. Access decisions take into account both attributes and their interrelationships. Examples include:
- Example: On Facebook, you can view a friend's private post, but you can't necessarily view the private posts of a friend of a friend unless you're also friends with that person.
- Application Context: Social networking platforms where user relationships play a pivotal role in determining access would greatly benefit from ReBAC.

While each model brings its distinct approach, in practice, they often intermingle. Implementations commonly weave elements from multiple models to create a tailored authorization mechanism that comprehensively fulfills specific requirements. The blending of these models in real-world setups reflects the flexibility and complexity required in today's digital ecosystems.

Hands-On

Diving into the practicalities, we'll use a Spring Boot application to breathe life into our conceptual understanding of authorization. Our canvas for this exercise is a modest web service dedicated to managing a list of pets. This service, while simple, will help us demarcate the boundaries between an authenticated user and one that comes adorned with specific roles or permissions.

For the scope of this chapter, our emphasis will mainly be on authentication as authorization" and on the facets of Role-Based Access Control (RBAC). As we progress further in the series, we'll delve deeper into intricate authorization mechanisms, but for now, these two paradigms serve as our focal points.

The Pet Controller

Consider this Spring Boot PetsController. It offers two endpoints:

An endpoint to fetch a list of pets, accessible by any authenticated user.
An endpoint to add a new pet, restricted to only those users with the 'admin' role.

@RestController
class PetsController {
    private val pets = mutableListOf(Pet("dog"))

    @GetMapping("/pets")
    fun getPets(): List<Pet> {
        return pets
    }

    @PostMapping("/pets")
    @PreAuthorize("hasAnyRole('admin')")
    fun addPet(pet: Pet): Pet {
        pets.add(pet)
        return pet
    }
}

data class Pet(val species: String)

In the code above, the @GetMapping annotation is standard for any Spring Boot application, denoting an HTTP GET endpoint. However, the @PostMapping is a bit special. It's decorated with the @PreAuthorize annotation, a component of Spring Security, which states that the following method (in this case, addPet) can only be accessed if the authenticated user has an 'admin' role.

When a user attempts to post a new pet to our service, the @PreAuthorize annotation intercepts the request and evaluates the user's roles. If they have the 'admin' role, the method is executed. Otherwise, an access denied response is returned.

You may wonder: where does this 'admin' role come from and how does the system recognize whether the user possesses this role or not?

The answer lies in the JSON Web Token (JWT) used by the application for authentication and authorization.

{
  "exp": 1692800905,
  "iat": 1692800845,
  "iss": "http://localhost:8090/realms/master",
  "aud": [
    "master-realm",
    "account"
  ],
  "sub": "84059de0-00e4-43ab-82f5-5b7b217cf8dd",
  "typ": "Bearer",
  "azp": "postman",
  "acr": "0",
  "allowed-origins": [
    "https://oauth.pstmn.io"
  ],
  "realm_access": {
    "roles": [
      "admin"
    ]
  },
  "scope": "openid profile email",
  "sid": "7395ed9c-09d4-449b-9f95-60958afe5f91",
  "email_verified": false,
  "preferred_username": "admin"
}

This JWT is a compact, URL-safe means of representing claims between two parties. In the context of our application, the claims within the JWT relay information about the authenticated user to the application. Let's break down some of these claims:

exp: This stands for "expiration time". It specifies the time after which the JWT will no longer be accepted. It's a mechanism to ensure that old tokens get discarded and are not misused if intercepted.
iat: The "issued at" time claim identifies the time at which the JWT was issued. This can be used to determine the age of the JWT.
iss: This is the "issuer" claim. It indicates the issuer of the JWT. In our token, the issuer is specified as http://localhost:8090/realms/master. Knowing the issuer is vital for validating the authenticity of the token, ensuring it comes from a trusted authority.
sub: Short for "subject", it identifies the principal entity that is the subject of the JWT. Often, this is used to hold the user ID.
preferred_username: This claim provides a human-readable string that identifies the user, which can be used to give a personalized user experience.

Amongst these claims, the realm_access claim is particularly interesting for our case. This claim reveals the roles associated with the user within the context of a realm in Keycloak, an open-source identity and access management solution. In our provided token, the user possesses the admin role within the realm, which correlates with the role our @PreAuthorize annotation is checking for.

Now, it's essential to draw attention to the mechanics of how this token is used. As highlighted in our previous chapter on authentication, this JWT token isn't a one-off; it accompanies every request made to the application. It's transmitted as a part of the Authorization header. This persistent presence means that the token, and by extension its claims, must be evaluated repeatedly. Every time the application receives a request, it needs to validate the token and identify the user and his permissions to ensure the right levels of access. This continuous validation plays a pivotal role in maintaining the security and integrity of the application's operations.

For those interested in a deeper dive into tokens and their functionalities, Auth0's JWT Basics provides an extensive overview.

This example highlights a basic use of Role-Based Access Control (RBAC). In this case, roles within the user's token are evaluated by the Spring framework to determine authorization. However, as systems grow and become more complex, challenges can arise. A common issue is "role explosion", where the number of roles becomes too large to manage effectively. In some cases, there may be too many roles to fit within the token's constraints.

To address this, advanced systems often separate roles or permissions from the token. Instead of including every role directly, they use a user identifier, like the sub claim (user ID). When a user requests access, this identifier is checked against an external source to determine the user's roles or permissions. This approach is more flexible and scalable, making it easier to handle complex authorization setups.

Upcoming

As we journey through this series, we'll delve deeper into more intricate mechanisms with practical exercises, especially Relationship-Based Access Control (ReBAC). This blog is largely centered on the latter, given our keen interest in harnessing the capabilities of Google's Zanzibar, a powerful system known for its robust ReBAC features. By familiarizing ourselves with these advanced techniques, we aim to demonstrate the potential and flexibility that advanced authorization mechanisms can bring to contemporary applications.

Scaling Authz: A Journey into Authorization Architectures

SWCode — Tue, 31 Oct 2023 08:46:53 +0000

Welcome to "Scaling Authz," a new series dedicated to exploring the complexities and challenges of implementing scalable authorization architectures, especially within the context of microservice environments. In this series, we will delve into the nuances of authorization, various approaches to solve common problems, and the techniques used by industry leaders to scale their systems.

The lessons shared here are not purely theoretical. Currently, I am engaged in a customer project where we are actively developing a scalable authorization architecture based on Ory Keto. We've moved beyond the conceptual phase and are now diving into the implementation process. I look forward to sharing the progress, challenges, and successes from this hands-on experience with all of you.

Before we delve into the specifics of our current journey, it's essential to highlight the invaluable experience our team at SWCode has garnered over the years. This isn't our first encounter with the challenges and intricacies of authorization at scale. In fact, we've been running a scalable system based on Casbin successfully for several years.

This prior experience at SWCode has offered us practical insights and deep knowledge about scalable authorization. The lessons we learned have been instrumental in shaping our approach and decisions as we build and enhance our authorization frameworks. We intend to share these insights throughout this blog series, in the hope that they might aid others navigating the same path.

If there's one lesson I've learned from my 8 years of cloud-native development, it's this: Never underestimate the power and complexity of authorization. The authentication stage of a user's journey, while crucial, is just the tip of the iceberg. Beneath the surface, a labyrinth of permissions, roles, and policies is waiting to be navigated and understood. Unfortunately, it's a journey many of us embark on too late.

From Authentication to Authorization

In the early stages of most projects, the primary focus tends to revolve around authentication - confirming that the user is who they claim to be. Upon successfully validating their identity, the gates to the system swing wide open, granting them access to a broad set of features and data. Initially, this may seem like an adequate solution, particularly for applications with a limited scope or user base.

However, as business requirements expand and evolve, the needs of the system inevitably become more complex. What was once a modest application might now be dealing with multiple types of users, each requiring distinct levels of access and control. A system that started with a handful of services might have transformed into a microservices landscape, with each microservice managing its own set of resources. This deepening resource hierarchy and the need to precisely control access at various levels herald the onset of a new challenge: authorization.

At this point, many projects introduce roles and permissions in an attempt to meet their growing authorization needs. A 'role' is a descriptor for a user's function within a system, while 'permissions' determine what actions this role can perform. For instance, an 'admin' role might have permission to 'create', 'read', 'update', and 'delete' all types of resources, while a 'user' role might only have 'read' permission.

When Simple Isn't Enough: The Pitfalls of Authorization Workarounds

For a time, this roles-and-permissions model may serve well. It's a flexible system that can handle a variety of scenarios. However, as the application continues to grow, the cracks start to show. A flat model of roles and permissions soon struggles to keep up with the intricacies of a deepening resource hierarchy and an expanding user base with diverse needs. What happens when a specific user needs access to a particular set of resources, but not others? Or when a user's permissions need to vary depending on the context or the specific attributes of the resources they're trying to access?

That's when we find ourselves in murky waters. The simple roles-and-permissions model, initially perceived as sufficient, starts to buckle under the weight of growing business requirements. The reality we face is that business requirements never stop growing. The resource hierarchy deepens and becomes more complex. And soon enough, we realize that our initial roles, often embedded within the token, no longer suffice. We start finding workarounds, implementing special handling for authorization, digging into the database, and examining roles. Some of us might even modify the business code to accommodate new requirements

The initial sense of relief that comes with implementing quick fixes and workarounds can quickly give way to a myriad of new problems. As we start to manipulate our authorization mechanisms to fit new, unanticipated requirements, we find ourselves in a precarious situation. Our previously clean and straightforward authorization process now starts to resemble a complex, intertwined web of conditions and exceptions, scattered throughout the codebase.

One significant downside to this approach is that it makes our system harder to understand and maintain. With special handling and exceptions woven into the business logic, the complexity of our codebase grows. This complexity can create cognitive overhead for developers, increasing the time it takes to onboard new team members and slowing down the debugging and development process.

Additionally, these workarounds can make our system more fragile. Since these patches are often rushed and not part of the initial design, they may not be covered by our existing test suite. This lack of test coverage can lead to unintended side effects and obscure bugs, which can be difficult to trace back to their source.

A perhaps more insidious problem is that with every workaround we implement, we start to erode the separation of concerns in our system. The business logic becomes entangled with authorization concerns. This entanglement can lead to a 'spaghetti code' situation, where the business logic and the authorization logic are so intertwined that changing one can inadvertently affect the other.

Finally, workarounds scattered throughout the business code can easily get lost or forgotten, leading to potential security vulnerabilities. When authorization checks are not centralized in an authorization layer, it's easy to miss some when making changes or additions to the system.

Arriving at this point in a project often feels like driving at full speed and suddenly spotting a red light in the distance. You must hit the brakes. It's a moment for pause, introspection, and reassessment. We must step back and ask ourselves: "How did we end up here?" It's a pivotal realization, a turning point where we understand that our approach needs a serious reevaluation.

Speaking from personal experience, I've encountered this scenario numerous times. Project after project, I've seen the same pattern emerge - the struggle to accommodate growing authorization needs within an unprepared system. Each time, it underscores the reality that this is a widespread and commonly underappreciated problem.

My message to those moving in this direction is clear and direct: Stop. Evaluate your current trajectory and consider its implications. If your project is becoming bogged down in a quagmire of workarounds and patchy fixes to accommodate for growing authorization needs, it's a clear warning sign that your current approach is not sustainable.

Realize that authorization is not an afterthought, or a 'nice-to-have'. It's not something that can be tacked onto a system with duct tape and good intentions. Authorization is a fundamental building block for any scalable, secure, and flexible system. It should be one of the cornerstones of your architecture, designed and planned for from the start.

Ignoring or underestimating this fact only leads to a technical debt that becomes increasingly burdensome to pay off. It's a road I've seen many teams travel down, and it always leads to the same destination: a complex, hard-to-maintain system that doesn't meet its users' needs or the security standards of today's digital landscape.

Upcoming

In this introductory post of our series, we aim to underscore the criticality of authorization and the complexities it introduces, particularly in microservice ecosystems. As we navigate through this series, we will explore in-depth topics such as the requirements for a scalable authorization architecture, fundamental concepts of authorization, roles, claims, tokens, permission structures, and the impact of Google's Zanzibar paper, among others.

Whether you're a seasoned architect seeking to solidify your knowledge or a beginner starting your journey in the world of scalable architectures, this series aims to be a comprehensive guide to mastering the art and science of authorization. So sit back, relax, and join me as we explore the intricate world of authorization in the era of scalability and microservices.