DEV Community: AuthZed

SpiceDB Office Hours

AuthZed — Thu, 19 Sep 2024 12:55:26 +0000

💡 SpiceDB office hours are a great way to learn more about authorization!

🚀 Live demos, announcements, & expert Q&A.

Navigating the Developer Landscape: Building, Selling, and Securing Tools | That Tech Pod

AuthZed — Tue, 20 Aug 2024 16:30:00 +0000

In this episode of That Tech Pod, Jake Moshenko, co-founder and CEO of AuthZed, discusses his journey from Amazon and Google to building a leading permissions system.

Key takeaways:

From Big Tech to Startup: Jake shares insights into working at both Amazon and Google, including Amazon's controversial firing practices.
Building for Developers: Learn about the challenges and rewards of creating developer tools and the importance of focusing on developer experience.
AuthZed's Journey: Discover how Jake and his co-founders, with their extensive enterprise software background, leveraged the Zanzibar papers to create AuthZed and solve critical authorization challenges.

Permissions Systems: Building an Authorization Lexicon

AuthZed — Wed, 07 Aug 2024 15:57:00 +0000

Why words matter

Every field develops its own linguistic shortcuts – jargon – to streamline communication around complex ideas. Consider browser cookies: a term that conjures images of tasty treats, but actually refers to bits of data that can be used for tracking online activity. While convenient for insiders, such jargon can be confusing for the uninitiated.

Common language, including jargon, usually settles organically as an area of practice matures. Sometimes early innovators intentionally agree to a particular framework for convenience.

While the practice of building authorization in applications is not new, modern approaches are evolving rapidly, spurred by advancements like Google's Zanzibar paper and the increasing demand for sophisticated collaborative features.

However, the language used to describe authorization solutions hasn't kept pace. We often rely on terminology borrowed from the lexicon of identity and access management (IAM), a field focused on resource control rather than application development. This mismatch can hinder clear communication and can lead to confusion.

How do we start clearing the air? Common language helps the community converse more productively about the tradeoffs involved when comparing authorization services and permissions systems. The first step: establish a shared vocabulary, starting with foundational elements.

Semantics

Often used interchangeably, the terms permissions and authorization represent distinct, yet related concepts. Permissions establish and reason about the rules and logic that govern permissible actions, defining the boundaries of what can and cannot be done under specific conditions and contexts. Authorization is the process of answering questions about permissions. In short, permissions determine whether or not an action is authorized.

Permissions systems

To determine whether any particular action is allowed or not, a decision maker requires the following:

A defined set of guidelines governing allowable actions
Relevant data about the action, including the actor (who's doing it), the action itself, the object of the action, and sometimes additional context or attributes
The method for applying those guidelines to the relevant data

In a legal trial, the court determines whether the defendant's actions violated the law. The laws and policies serve as the set of guidelines, while the facts of the case provide the information relevant to the action in question. The mechanism that applies these laws and policies to the facts of the case would be the judge or jury.

Coming back to software, an authorization decision requires similar information:

A permissions model that defines the logic and rules governing allowable actions
Data pertinent to the decision being made, either from a datastore or provided at runtime
Engine capable of applying logic and rules to provided data and returning a decision

These components make up a permissions system. The system itself may be wholly contained outside of an application, be dispersed throughout a monolithic application, or some combination of both. A system might have multiples of a component, such as multiple data sources or permissions models.

Applying SpiceDB concepts to this terminology, a SpiceDB permissions system uses a schema to structure relationship data, creating a graph database capable of returning answers about authorization.

The permissions system concept not only applies to other Zanzibar implementations, but any other system that may follow a different paradigm.

Authorization service

While a permissions system can function independently to provide authorization to an application, an authorization service provides a layer of abstraction on top of a permissions system.

This service manages the business logic of authorization and permissions systems while also providing additional operational functionality such as caching decisions, auditing and logging, and managing supportive resources.

Used in context, the AuthZed Cloud authorization service deploys, manages, and provides additional operational capabilities for SpiceDB permissions systems. Other organizations choose to build their own authorization service for their SpiceDB permissions systems within their own infrastructure.

By establishing a clear and consistent lexicon for implementing authorization, we foster a shared understanding that allows us to compare disparate approaches, evaluate tradeoffs with precision, and ultimately architect more robust and secure collaborative workflows and sharing capabilities within applications.

Glossary

Permissions establish and reason about the rules and logic that govern permissible actions, defining the boundaries of what can and cannot be done under specific conditions and contexts.

Authorization is the process of answering questions about permissions.

Permissions systems construct a mechanism for defining, storing, and evaluating permissions.

Authorization services provide a layer of abstraction on top of permission systems that provide answers to an end consumer, such as an application.

Permissions Management with the Privacy Files Podcast

AuthZed — Wed, 21 Feb 2024 10:02:46 +0000

Jake joined The Privacy Files podcast, where he discussed the world of permissions and access control. Throughout the episode, listeners are treated to an in-depth exploration of the evolution of permissions management technologies, underscoring the essential role of advanced access control systems in ensuring digital privacy.

Introduction to Permissions and Access Control in Airports

Rich shares observations from airports on controlled access and permissions.
Examples include security at drop-off/pickup points, boarding pass requirements for gate access, TSA regulations on liquid amounts, and restricted areas for authorized personnel.
Access to airport lounges requires a special pass, indicating controlled entry based on permissions.

Introduction to Permissions Management

Jake focuses on permissions management technology.
Discussion on the importance of not hand-rolling security, especially permissions management, due to the high risk of introducing vulnerabilities.
Permissions management is critical yet challenging, and it's advisable to rely on experts in the field.

The Importance and Challenges of Permissions Management

Jake emphasizes the shift towards outsourcing non-core competencies, including permissions management, similar to cloud adoption.
Permissions management examples include bank account access rights and Google Docs permissions.
The conversation touches on the significance of permissions in ensuring privacy and control over one's data.

Jake's Background and Interest in Permissions Management

Jake's experiences at Amazon and Google influenced his interest in building large services focused on the developer experience.
The origin of AuthZed was inspired by challenges faced in managing permissions in previous ventures, highlighting the need for a specialized solution in permissions management.

Common Pitfalls in Developing Permissions Code

Jake discusses the dangers of developing custom permissions code, including security vulnerabilities and scalability issues.
Broken access control is highlighted as a top security vulnerability, underscoring the importance of expertly managed permissions systems.

Transitioning to AuthZed and the Concept of Cloud-Native

Jake's journey from IBM to founding AuthZed, driven by entrepreneurial spirit and the desire to address permissions management challenges.
Explanation of cloud-native versus cloud-based applications, focusing on security responsibilities and innovations in cloud services.

SpiceDB and AuthZed's Mission

AuthZed aims to provide a comprehensive solution for permissions management inspired by Google's Zanzibar paper.
SpiceDB, as an open-source implementation of Zanzibar, represents AuthZed's contribution to simplifying permissions management for companies.

Challenges in Selling Infrastructure to Enterprises

Discusses the challenges of capturing attention and convincing enterprises of the need for specialized permissions management solutions.
The trend towards proactive engagement with permissions management solutions, rather than reactive responses to breaches.

Real-World Implications of Permissions Mismanagement

Examples of permissions-related breaches and the silent nature of some breaches pose significant risks to both companies and individuals.
The evolving landscape of permissions management and the increasing recognition of its importance in corporate security and compliance.

Personal Insights and Advice on Privacy and Security

Jake shares personal experiences and the impact of working in the permissions management space on his privacy consciousness.
Advice for individuals on enhancing their digital privacy and security, emphasizing the importance of being selective and cautious with data sharing.

Authentication vs. Authorization

AuthZed — Wed, 10 Jan 2024 10:04:18 +0000

Authz and Authn, a primer

As humans, we’ve been guarding our stuff since we first invented locks 6,000 years ago.

Guarding assets and sharing access to them is human nature. There are two major steps involved in this process: authentication and authorization. First, one must authenticate that they are who they say they are. Then, once someone’s credentials verify their identity, the next question is whether or not that individual is allowed access to the asset.

Think of it this way:

You want to go on a trip to another country. You need a passport to verify that you are who you say you are - this is an example of authentication. You also need a visa that gives you permission to enter the other country. This is an example of authorization. The same principles can be applied in computing:

Authorization and authentication get confused OFTEN - I mean, look at how similar the words are! Instead of having the perspective of authentication vs. authorization, let’s understand how they work together. As you’ll see, you can’t have one without the other.

Authentication

What is Authentication?

Authentication, in computing, involves verifying the identity of a user, process, or machine. To authenticate a user, we can use one or a combination of three key factors: Knowledge: Something a person knows – i.e. a password/username, a passcode or PIN, or a security question Ownership: Physical token or something a person owns – i.e. a Yubikey, hardware capable of an encrypted handshake (like ACME for Apple devices), or Google Authenticator Inherence: A vital piece of something a user is or does – i.e. biometrics or keystrokes

Whenever you sign into social media, your email, or any Web-based application, you are using authentication to prove you are who you say you are.

The History of Authentication

Authentication has been a part of our world since the dawn of time, but that is another conversation for another day. For now, let’s dive into the history of authentication in modern computing.

Authentication started all the way back in the 1960’s (60 years ago!) with the first passwords in databases. From there it moved to passwords with a hash, and eventually, encryption. Modern encryption has been around since the 1970’s with RSA asymmetric encryption.

Timeline

[Timeline is based on an excellent piece - “Digital authentication: The past, present and uncertain future of the keys to online identity” on GeekWire, with a few new additions by the authors]

We don’t know where authentication will go in the future but as technology continues to advance, the ability to identify an individual user will get more complicated. This rings true for all Web applications and services, but even more so for highly secure or zero trust environments such as government platforms and healthcare data. Authentication is the passport to our digital world.

Authorization

If authentication is our passport, authorization is our visa. Authorization is when one entity grants permission to another entity to engage with a resource within a set of boundaries. Authorization has become increasingly difficult for application developers, often causing performance issues when deployed at scale and blocking feature development.

As our digital footprint has grown, experts have developed new access control and permissions management models to meet the needs of feature hungry teams. Since the advent of the original Access Control List Model (ACL) in the 1960s – similar to an “invite only” guest list – we have designed increasingly abstract authorization systems.

Timeline

Moving forward, interest will likely trend toward dynamic, context-aware systems that can adjust permissions in real time. The exploration of decentralized models for authorization that integrates concepts from ReBAC and systems like Zanzibar suggests a future with more secure, efficient, and flexible access control mechanisms.

As technology develops, so will both authentication and authorization. We at AuthZed are excited to be on the forefront of the authorization space, bringing Google’s Zanzibar to the masses with SpiceDB.

You can try out a schema here and get started with fine-grained access control across all of your environments and applications.

Zed File #Z-4902: Spanner Spikes

AuthZed — Thu, 07 Dec 2023 20:32:33 +0000

Zed File # Z-4902: Spanner Spikes

💼 Case facts

Customer: [redacted] // Product: AuthZed Dedicated // Datastore: Spanner

A storm was brewing at AuthZed customer [redacted]- as their traffic grew, so did the number of permission requests, driving check latencies up to a full second at the 95th percentile. This spike in latency degraded the client system's performance - it was time to call in the experts.

AuthZed special agents were on the case.

Read on to discover how our agents used AuthZed’s extensive extant observability toolkit and quickly rolled out additional tooling and optimizations to find the solution among a complex myriad of contributing factors.

Bonus - all improvements are now available in SpiceDB v1.28!

First aid

As an emergency stopgap, AuthZed immediately scaled up resources and tuned Spanner’s configuration, improving overall performance. However, latencies were still spiking every few hours, indicating the system was not yet fully stable. With these initial measures in place, the team started investigating.

Dispatch deduplication

Agent V knew he had to fist check the customer’s schema - it might reveal complex permissions that could lead to many subproblems being dispatched. With enough capacity, the system should be capable to handle it, but customer’s cluster was relatively small.

The venerable manual of Agent Operations, the Zanzibar paper, described a clever strategy to reduce fan out of complex permissions: request deduplication. Authzed Agent X introduced improvements in the SpiceDB dispatch with a deduplication middleware that yielded a 30% reduction in dispatched requests, which in turn, led to a proportional reduction in database usage. Despite this notable overall improvement in system performance, persistent latency spikes suggested there still were other underlying issues to address.

Tracking consistency

Initial investigation efforts were hindered due to insufficient observability, leaving key data shrouded in mystery. The situation improved when Agent J improved SpiceDB’s request consistency middleware to produce metrics describing the requested consistency level. This tool uncovered a correlation between the latency spikes and increases in ' fully_consistent' traffic, providing a new lead - one what would crack the case.

Schema spotting

Pursuing the lead, Agent V knew the team was onto something- they just needed more data. He set to improve the readability of OpenTelemetry traces across the dispatch graph and to expose additional metadata, particularly schema traversal details. Direct tracking of Spanner server-side query latency posed a challenge, as using QueryWithStats API method increased Spanner’s CPU usage significantly. Undeterred, Agent V successfully integrated Spanner Client metrics with OpenTelemetry and exposed the metrics through SpiceDB’s Prometheus endpoint. And to add the last bit of information missing, Agent V exposed request and response payloads via new CLI flags.

The solution

Newly equipped with observability data, the team utilized Google Cloud Platform’s log aggregator to locate the timing and source of the latency spikes. They discovered that [redacted] was intermittently issuing up to 800 CheckPermission calls within milliseconds, overwhelming the processing capacity of the provisioned Spanner units.

As it turned out, the database simply had insufficient capacity to handle these spikes.

A crucial log entry showed a single user making over 600 requests within 250 milliseconds for the same permission check, indicating inefficient ACL filtering that fanned out individual checks too quickly for the allocated capacity, all of which is more efficiently computed with BulkCheck or LookupResources.

AuthZed advised the customer to move from fanout CheckPermission calls to BulkCheck, which led to significant latency reduction and improved cache efficiency. As a result, the SpiceDB cluster has achieved stability, eliminating the problematic latency spikes and enhancing the performance of the Permissions System.

These improvements are now available to everyone in SpiceDB v1.28!

Policy-Based Access Control (PBAC) vs Relationship-Based Access Control (ReBAC): When You Should Use One or the Other

AuthZed — Wed, 29 Nov 2023 09:51:37 +0000

At Authzed, we are building a platform around SpiceDB - our open-source, Google Zanzibar-inspired authorization service. In the past, we have written about what Zanzibar is, why more businesses are adopting it, and some of the different ways you can implement Relationship-Based Access Control (ReBAC) in your own applications.

We have even published our own annotated copy of the Zanzibar whitepaper - to make it more accessible, add some outside perspective, and to help illuminate the Zanzibar model independent of Google and Google-scale problems.

But we have not spent much time talking about alternatives to Zanzibar and SpiceDB. Authorization is a field with a long and storied history, and there are several modern alternatives to Google Zanzibar that come with different trade-offs in terms of performance, deployment topology, and useability.

Understanding where SpiceDB fits in the landscape of authorization is important for anyone evaluating different solutions for authorization.

In this post we are going to focus on one specific class of modern authorization technologies: Policy-Based Access Control (PBAC).

If you're not sure what PBAC is, how it differs from SpiceDB and Google Zanzibar, or when you should use one, the other, or both, then this post is for you.

What is policy role-based access control?

A PBAC solution generally has at least two separate components:

A policy engine, a software component that takes policy rules and data as input, and produces authorization decisions as output.
A policy language, which is often a simple, declarative language for defining authorization rules.

Typically, these solutions also espouse a philosophy that the policies written in the policy language should be managed using standard software development lifecycle tools and techniques, though the degree to which that happens is up to the user.

Though there are many ways to deploy a policy engine, they are usually deployed in one of three ways:

Embedded directly into an application
As a sidecar, serving queries over the local network
As a reverse proxy or as an authorization step in a proxy, authorizing requests before sending them on to their destination.

Some examples of Policy-Based access control systems include:

Styra's Open Policy Agent and Rego Language
AWS's Verified Permissions and the Cedar language
Hashicorp's Sentinal framework and language

Due to its ubiquity, we'll use OPA and Rego in the examples below. If the idea of using policies for authorization is totally new to you, it might be worth checking out what some of these projects have to say about themselves before reading on.

What is the difference between ReBAC and PBAC?

On the surface, ReBAC (Google Zanzibar) and SpiceDB may look very similar to PBAC:

SpiceDB is an engine that computes authorization results over data.
SpiceDB uses a declarative schema language that defines how permission is computed. Many teams choose to manage this schema with standard tools like git.

But there are some major differences in how Zanzibar approaches access that sets it apart.

ReBAC is a Database

One of the defining features of Zanzibar is its use of a backing database to store authorization data and an explicit Write API for storing auth data for Zanzibar to use in later request evaluations.

At evaluation time, Google Zanzibar makes queries against the data in the database to evaluate authorization requests. This is what puts the "DB" in SpiceDB — in the Zanzibar model, authorization data must be explicitly written into the backing datastore ahead of query time, just like an RDMS.

ReBAC Stores Relationships

Most PBAC engines put very few restrictions on what authorization data can look like. Rego is a powerful and ergonomic datalog implementation that can be used as a general-purpose tool for computation over a set of facts stored as JSON.

In Zanzibar, all "facts" are relationship tuples: they define a relationship between a subject (user or group of users) and a resource.

The schema (or set of namespaces in Zanzibar parlance) defines how these base relationships relate to each other to form a graph.

A document is in a folder, a role grants access to a group, and a share grants a role access to a folder - these relationships form a graph that allows an application to ask a simple question like "Can Alice access the document?" by following the links in the graph.

Since everything is a graph, most authorization questions can fit into this model, including RBAC and ABAC.

ReBAC Scales

Though policy engines generally require the user to send the data that the policy operates over alongside the authorization request, they sometimes offer other mechanisms for syncing relevant data down periodically, or pulling data at evaluation time, which could be used to get a similar approach to data as Zanzibar. And you could certainly choose to store data as relationships in a policy engine.

If you start to go down this road, though, it's good to be aware of the rest of the Zanzibar design, since it addresses things like consistency guarantees, hotspot caching, and general scaling strategies which don't generally have direct or simple answers in a policy engine.

When should you use a policy engine?

Policy engines are usually very fast at evaluating policies over small sets of local data. They do this very well, and a lot of time is invested in making the ergonomics developer-friendly.

Any time all of the data for a policy decision can be statelessly derived from a policy engine, they tend to shine. Here is an example of a great use of PBAC from OPA (playground):

rego!
package kubernetes.admission
import future.keywords
deny contains msg if {
input.request.kind.kind == "Pod"
some container in input.request.object.spec.containers
image := container.image
not startswith(image, "hooli.com/")
msg := sprintf("image '%s' comes from untrusted registry", [image])
}

Note that:

The policy is not likely to update very often. A developer might need to add a new untrusted registry maybe once a year, if ever.
Everything required to evaluate the policy is right there in the API request: request.kind, input.request.object.spec.containers.
Because this is a Kubernetes example, the policy is likely running with Gatekeeper, which acts as an authorization hook for the kube-apiserver in the cluster. There are no edge deployments that need synchronization.
The evaluation should be fast - the rules are simple and the data is small.

Many of the examples in the OPA playground are similar - they highlight the use of policy engines that are always fast and sensible to manage as code. But there is an entire section of the OPA docs dedicated to external data and the ways it can be fed into OPA policies for evaluation.

Most of these options require a connection to a database, or making networked API calls instead. Once you need to do one of these things, policy evaluation is no longer guaranteed to be fast.

Generally speaking, in 2023, and regardless of the particular stack, you can expect a local policy evaluation over a small local dataset to be on the order of microseconds. But even then, there can be cases where evaluation takes milliseconds.

Here's an excerpt from an OPA profile (source):

+----------+-------------+-------------+-------------+-------------+----------+----------+--------------+------------------+
| MIN | MAX | MEAN | 90% | 99% | NUM EVAL | NUM REDO | NUM GEN EXPR | LOCATION |
+----------+-------------+-------------+-------------+-------------+----------+----------+--------------+------------------+
| 43.875µs | 26.135469ms | 11.494512ms | 25.746215ms | 26.135469ms | 1 | 1 | 1 | data.rbac.allow |
+----------+-------------+-------------+-------------+-------------+----------+----------+--------------+------------------+

A p90 of 25ms means that even when the policy engine has all of the data it needs, it may not beat a network call to a nearby SpiceDB. And the time does not include the time it took to gather the data up to feed into the engine in the first place.

What follows is an example of an OPA policy that is not a great use for a policy engine (playground):

package kubernetes.validating.ingress
import future.keywords.contains
import future.keywords.if
deny contains msg if {
    input_host := input.request.object.spec.rules[_].host
    some other_ns, other_name
    other_host := data.kubernetes.ingresses[other_ns][other_name].spec.rules[_].host
    [input_ns, input_name] != [other_ns, other_name]
    input_host == other_host
    msg := sprintf("Ingress host conflicts with ingress %v/%v", [other_ns, other_name])
}
input_ns := input.request.object.metadata.namespace
input_name := input.request.object.metadata.name
is_ingress if {
    input.request.kind.kind == "Ingress"
    input.request.kind.group == "extensions"
    input.request.kind.version == "v1beta1"
}

This is a more complex policy that ensures that no two Ingress Kubernetes objects try to bind to the same host name.

At first glance, it may not be clear what the issue is - it mostly comes down to this line:

other_host := data.kubernetes.ingresses[other_ns][other_name].spec.rules[_].host

This generates a list of all hosts referenced by other Ingress objects. But where did this data come from? It's left as an exercise to the reader, but the full list of all ingress objects needs to be queried first and provided to OPA at evaluation time.

This approach can work, but it removes the best property of policy engines: authorizing requests is no longer a fast process that runs at the edge, you have to deal with real networks and real databases in your authorization hotpath.

Once you need external data to answer authorization questions, you might start to wonder if your policy engine is the best tool for the job.

You got Policy in ReBAC. You got ReBAC in Policy.

SpiceDB strives to be a faithful Zanzibar implementation, deviating primarily when it seems best for usability or for de-Googling. One thing that we found, though, was that although Zanzibar was great for general authorization, there were often a few cases that users had that didn't lend themselves to relationship data stored ahead of time in SpiceDB.

This led us to develop Caveats, which uses CEL to conditionally define whether a relationship exists. Caveats offer some of the benefits of policy engines without sacrificing the benefits of Zanzibar and maintaining its caching and sharding strategies.

Using Caveats doesn't remove the need for a network roundtrip to SpiceDB. But it does mean you may not need to run a separate policy engine if you're already running SpiceDB, which can simplify your deployment.

What do we do at Authzed?

We run a significant amount of infrastructure at Authzed to power our Serverless and Dedicated products. In the end, we use a mix of different approaches:

A dedicated SpiceDB instance powers the tenancy layer and permissions for the Authzed Serverless platform.
Although we don't use projects like Gatekeeper or Kyverno (at the moment), we do use some policy-like features of several Kubernetes projects to enforce standards in the clusters we run.
The Fine Grained Access Management (FGAM) feature in SpiceDB Enterprise and SpiceDB Dedicated uses a hybrid approach: an embedded SpiceDB instance uses Caveats to evaluate requests at runtime, with no network roundtrips.

Whether or not you want to use policy-based access control, Zanzibar, or both, depends on the authorization problems you have. Hopefully, this post has helped distinguish when one approach may be better suited than the other.

SpiceDB Tracing: How to interpret our OpenTelemetry traces

AuthZed — Tue, 21 Nov 2023 14:03:54 +0000

Transcription

This is Victor from AuthZed speaking and today we're going to look into some of the recent changes introducing the OTel instrumentation in SpiceDB.

Before these changes, folks needed a bit of knowledge on the internals of SpiceDB in order to interpret what was happening in requests, right?

It should be possible to visualize how SpiceDB walks the graph and that represents your schema. So you can better understand the sequence of steps taken to respond to an API request.

We're going to look into an example CheckPermission request using a simple schema with user and resource and the usual "view" permission, as you can see here which is granted if either user has a "reader" or "edit" relationship over a resource.

I have my SpiceDB instance running in the background with a Postgres datastore and I've spun up an instance of Jaeger, a popular distributed tracing service, and I've also just issued a check request with SpiceDB companion CLI tool, Zed. Let's have a look at the trace.

As you all probably know SpiceDB has a tunable consistency model which allows clients to decide the levels of consistency of their request.

By default, Zed uses minimize_latency which offers a good compromise between performance and security with a bounded staleness of up to five seconds maximum by default.

OptimizedRevision, ayou can see here at the top, is what indicates that minimized latency has been used as the server computes and a snapshot of the system and then SpiceDB validates the request parameters and making sure that it adheres to the schema that is currently in the system. As you can see in these two checks on the name spaces or the definitions on your schema.

Here's where the interesting bits start- this is at the first dispatch or we usually refer as sub problems being computed and we've issued a "view" over a specific resource for a user and so this is the first dispatch, this is the first sub-problem being computed.

This subproblem then SpiceDB determines that is composed of two parts or two sub problems: the "reader" side and the "edit" side. In turn, SpiceDB is going to turn these two guys into a new dispatch.

This is happening in one single instance so it's not actually dispatching to a different physical node, but for all intents and purposes, it should be the same. Here we can see then the subproblems of the first "view" permission check being computed.

We have "reader" on one side and then "edit" on the other side and the plus sign indicating this is a union. A "view" is a permission is granted to a user of resource over a resource if either they have "reader" or "edit" relationships.

When this is dispatched, then SpiceDB determines that this is a terminal or indicates that this is a relationship and the only thing it has to do is to actually query the database.

As you can see here it accesses the QueryRelationships datastore API and then issues a query "select." You can see details here of the actual SQL query issued to the Postgres database in case you're interested to know what's happening under the hood.

This is the new structure of the Otel traces that were introduced in SpiceDB 1.27.

What you need to know about Permissions Management

AuthZed — Thu, 09 Nov 2023 10:50:03 +0000

What you will learn

1. The problem with permissions management, especially RBAC

RBAC examples and challenges
How companies start permissions management

2. Authorization models applied by companies

Custom code Permissions Management: example, pros and cons
Policy-based Permissions Management: example, pros and cons
Google Zanzibar: example, pros and cons

3. Google Zanzibar open-source implementations:

SpiceDB, Google Zanzibar’s most mature open source implementation
Google Zanzibar vs Ory Kratos vs Hydra vs Oathkeeper
How does SpiceDB stack against Google Zanzibar?
Scalability and Compatibility in Permissions Management
Open FGA vs SpiceDB

4. How to choose your permissions management system

Transcript

I'm Jake Moshenko, CEO of a company called AuthZed, and AuthZed is an authorization company.

So, why should you listen to me? Why do I even know anything about this topic? First of all, as I already mentioned, I'm a co-founder and CEO of an authorization company. In the past, I also founded a company called Dev Table, and one of the things that came out of that company was something that we created called Quay. Quay was the first private Docker registry before Docker Hub allowed you to store private images. Quay was acquired by CoreOS. Then at CoreOS, I was head of engineering for that same division.

Eventually, CoreOS was acquired by Red Hat. At Red Hat, I worked as a senior manager in the service delivery organization. You can think of that as like platform engineering and SRE all wrapped up into one. And prior to that, I also have some experience at some companies you've probably heard of like Google and Amazon. But outside of that CV, the real reason that you should listen to me is because I've actually lived the problem that I'm going to be talking about today.

Through my work at CoreOS, through my work at Quay, and even in the past at Google and Amazon, I've experienced this problem firsthand. So in today's talk, I'm going to talk about the problem that I'm referring to, a little bit about the solution space, and what some people are doing in this area.

I'll talk about ReBAC, and how Google Zanzibar is one example of ReBAC. ReBAC stands for Relationship-Based Access Control. I'll talk about some of the pros and cons of the different authorization approaches that we'll discuss in the solution space, and then I'll explain why we're all in on the Zanzibar model, which is what Google is doing internally.

To frame the problem, I've got here two graphical representations of permissions models, and I'm encouraging you to think about what these models have in common.

On the left side, we have what I call a straightforward RBAC system where user Suzy is an editor of document one. And then on the right, we've got this much more convoluted, much more interesting graph where user Suzy has a role within the context of an org, a role has dynamic grants based on what you're trying to do and the object types you're trying to do it to.

What do these models have in common? Well, they're both called RBAC. One of the challenges in the space of authorization is that everybody has different definitions of what these different acronyms stand for, not necessarily for what they mean in practice. And maybe you've heard other authorization terms like ABAC, so there's all kinds of work done around defining the problem and sort of giving it names in the space, but we often don't start here.

So companies will often start very simply when they're imagining their authorization or they're deploying their first authorization system for their application. Most of the time, they start by building their own. This could be as simple as storing a few pieces of data in a database, loading that up, and then interpreting it in code. At some point, that stops working. That point could soon or it could be very far in the future- but at some point, it stops working.

Maybe it stops working because you have too much traffic now. So the system was built to work for 100 queries per second, and now you're at a thousand or 10,000 queries per second. Or it was built to support 10 users, and now you have a thousand or 10,000 or a million users. So scale can be one of the reasons.

Another reason it stops working is that customers themselves start requesting things that you didn't anticipate. An example of this would be the model on the right in the past slide. I'll give an example of that from my past as well.

Another time it might fail is if you start entering new geographies, and the underlying data store that you're building this on top of is not a distributed data store. So imagine you built a permission system, it works super well, and everything is headquartered out of Amazon US East, and now you need to expand into Frankfurt or somewhere in APAC. If you need a single unified view of permissions data, often the homegrown solution won't follow you on that journey.

Now I want to talk a little bit about my lived experience developing permissions for Quay at CoreOS. This was the very first permissions model that we built when we created and launched Quay. We thought we were copying the GitHub model, I'll put it that way.

In general, everything was centered around the concept of a repository. A repository is where you store your container images, and then users could have various relationships to a repository. So a user could be a reader of the repository, a writer of the repository, or an admin. And then if you see the arrows pointing to the right between the different roles, those are saying that when you're considering who is a reader, you should also consider all writers as readers, and you should consider all admins as writers and therefore also readers. So if someone just has admin, we don't want to prevent them from reading the repository.

So this was actually what we shipped with. Within a week, everyone said, "It's great that you copied GitHub's permission system, or you think you did, but we also need all of the nifty organizational support that GitHub offers in their permission system."

So within a month, we built and shipped this much more complicated system. Everything that's new, I have there in red. And what we had to do is we had to build some concept of organizations. We had to nest repositories under organizations. Organizations had teams, teams had roles within an organization, and then teams could be directly given access to any of the other roles that a repository already had. So that's what that graph is meant to represent.

The real takeaway here is that it was a lot more complicated than we initially anticipated. We built and launched that, and the users who asked for it were happy for a little while. But what we failed to realize is that not even GitHub nailed it, and what the users wanted was the ability to have multiple namespaces under an organization.

Imagine you're like me and you used to work at Red Hat, and Red Hat had different business units, or maybe even Red Hat itself is a business unit of IBM now. They wanted to be able to nest namespaces and they wanted to be able to nest teams. They wanted to be able to break down and model teams according to the org chart, and they wanted to be able to nest repositories under a top-level parent company/business unit/eam/repository. And they wanted to be able to hang permissions off of any of those various places in this tree that they wanted to build up.

For example, if you had a top-level organization, maybe the admin for it was some superuser in the operations department, the COO at the organization. And then as you started to descend through the tree, the permissions would become more fine-grained and more granular and be federated out on a team-by-team or a person-by-person basis.

We never actually built and shipped this version of the authorization model. And the reason that we didn't do that is because we were somewhat naive in our early implementation.

We did our authorization, probably similar to how many of you are doing, by storing data in a database and then doing a bunch of SQL joins to figure out what you have access to. At the time, we were built on top of MySQL. MySQL didn't support recursive CTEs, so we couldn't do a recursive query to jump through, to walk through and gather up these permissions as we descended or ascended through the namespace tree or through the team hierarchy.

We were never actually able to ship this feature. It was one of those things that was always on the backlog, something that we knew we needed to do, but it was going to require a major refactoring of the way we were doing authorization. That's the problem, as we see it, in a nutshell.

And now let's step into some of the ways that companies are currently authorizing things. The most common thing that we see by far is that companies are using embedded custom authorization code in their applications. They're just writing some code to interpret some data, they're doing it directly in their codebase, and then they're making authorization decisions based on that.

If people have decided that this is a bad idea, sometimes they'll adopt a policy engine, and I'll talk a little bit about that as well. And then finally, the new paradigm or the new kid on the block is the Zanzibar paradigm that Google put forward.

First up, this is the embedded code in applications. The specifics of the code aren't really important, but what is important is that you can see that we're reaching out to the database every time a user is trying to handle a request for an object. We reach out to the database and we load who are the authorized users for the object, and then we usually have to reach back out to the same database again to get the data to send back to the user. And then we've got a little authorized method there. It's already abstracted out into its function, so it's not like a big if-else block embedded in the code, but it is still something that exists within our monolithic application.

No judgment if this is how your authorization code looks. Like I said, I've built and launched variations of this in the past, and it works for some scale.

So when we break down the pros and cons of writing custom code to do authorization, the biggest pro is it's infinitely flexible. You can do and express anything that your heart desires as long as you can figure out how to write the code for it. Another pro for custom code is once you've exceeded the single monolithic application and you start to break down your service into potentially other services or microservices, you can take that authorized method like I had on the last slide, and you can turn it into a shared library. You can share that library and compile that library into other services and other applications.

But if that code is living within a single service, one of the downsides is that you can't call it from other services. So one thing that we'll often see as a company is going through like a microservices or a service decomposition is that the authorization logic will get tied up into that monolith, and then they'll have to expose methods on for those other microservices to query it about authorization. This is obviously like a huge inversion of priority, right? You don't want microservices to be blocked on calling out to a monolith. This is not the ideal by a long shot.

Another disadvantage of this method is that if you split this out into a library and you happen to be a polyglot shop, which means you're writing your services in multiple different languages, you need an implementation of the library for all of the different languages that you're potentially implementing in. So for example, if you use JavaScript, and Node.js on the backend, maybe you use some Python, maybe some Java, maybe some Ruby, you need an implementation of that same authorization code for all of those languages. And that can often result in complicated rollouts as well. So if you need to update the authorization code, you need to make sure that you're updating it across all languages and rolling it out to services in such a way that the users are never given an inconsistent view or incorrect answer.

Another downside of this method is that you're going to your database, your main database, your source of truth, and fetching data on every single request that requires authorization. Not just data, but data specifically about authorization. This can add additional unwanted load to your database. This was something that we ran into for Quay. On Quay, we did those SQL joins as I mentioned, and I think the final version of the code, or the code that you can go see if you look at the open source right now, does a join with like 11 different tables. When we did some analysis, it turned out that we were spending far too much of our database CPU just joining on these same 11 tables over and over again. This could be like a tall tale; maybe every time I say it, the number of tables gets a little bit bigger, but it is open source, you can go check out how that works today.

And finally, authorization code isn't the kind of code that people like to open up and change all the time because it's tricky code. It's hard to verify that you haven't opened any security holes, and it's just not something that your average application developer is interested in getting down into the nitty-gritty and understanding the nuts and bolts of.

Moving on to the next solution, we have policy engines. So policy engines were kind of the hot new thing on the block probably about eight or nine years ago. So what about policy engines? Are they good? Are they bad? What do they do?

Often when people embark on adopting a policy engine, their goal is to abstract their authorization logic from their code, and this is a laudable goal. You can solve a lot of the same problems that we just talked about. You can write your policies in a robust logic language that's formally proven to be correct for a certain class of authorization operations.

Since it's usually a network request or you're running it in a sidecar, something like that, you can have one implementation of your policies for all languages, and then you just reach out and talk to those policies over the network. This works great when you already have all of the data ready.

Imagine if you're writing like a network appliance or an HTTP filter or something like that, and the data that you're making your request based on is already available to you in the form of the IP address of the caller or HTTP headers or the current date and the current time. If you already have all of that data that you want to feed to the policy engine, the policy engine can often give you an answer back in microseconds.

Some of the downsides of using a policy engine, though, are that you still have to fetch data from your main source of truth for every request and feed it to that policy engine. And sometimes you're reading a lot more data than you have to, depending on how you can constrain and how you can draw a boundary around the data that you need to feed to that policy engine. It can also cause a complicated rollout of new policies if you're using the sidecar model and you've baked the policy into the sidecar.

You need to make sure that you're rolling that out in a consistent way, or at least in a backward-compatible way, such that if two different services are evaluating the policy at two different times, they don't accidentally open up a security flaw in terms of having different data interpreted different ways at different places.

And finally, these languages that you write policies in are powerful. They're very powerful, and these can result in policies that are difficult to understand and eat up a lot of your budget in terms of latency, and difficult to write and difficult to maintain.

Just to show an example of one of those, I went to the OPA website and I just pulled a Rego script right off of the landing page. I'm going to give everybody about 30 seconds to read through this Rego script and try to have an idea in mind for what this script is doing or what this policy is doing.

When I look at this policy, I see eligible groups, something about locations, a lot of things about locations. And then we've got some kind of roles construct, and then we've got this block repeated three times which I'm guessing is binding a location to a role record.

So if you were able to figure it out, maybe you were, maybe you weren't, but what this is doing, is doing a role binding by location and group name. The first thing it does is it verifies that the groups are one of the supported groups, so that's group one and group two. And then it gives three different locations, and the locations have different roles for each group in each location. And then anything that doesn't match one of those locations or one of those groups results in an empty set of roles that you have access to.

So this table makes a lot of sense to me, but this policy was kind of hard for me to read through and for me to understand what exactly was happening.

As a result of having policies that are written in this domain-specific policy language, one thing that we're being told by our prospects and by our customers is that when you adopt a system like this, the platform or auth team usually ends up owning all of the policies.

We talked to a massive organization, and they told us that they were using a version of Prolog or a dialect of Prolog to do their authorization policy, and there was one person in this multi-tens of thousands of person organization who was capable and confident in writing their policies. This is usually kind of an anti-pattern for what we're trying to accomplish by bringing in an authorization policy engine.

Moving on to the next and final solution in the solution space is Google's Zanzibar. So in 2019, Google wrote a paper called Zanzibar: Google's Consistent Global Authorization System. The core concept of Zanzibar is let's build off on top of a globally distributed database. So internal to Google, there's a globally distributed database called Spanner. It's ACID, it seems to violate the CAP theorem and like the basic laws of physics, but it is a very solid thing on which to build an authorization system that might need to scale and that might be geographically distributed.

The whole paper is about 12 pages. It's an easy read, and I highly encourage you to go read it.

I will give an example of what adopting Zanzibar has allowed Google to do.If you've ever been in this case where you're writing an email, and in the email you include a link to a Google Document, and you think everything is great, and you go to hit the send button, and Google pops up a scary warning that says, "Hey, someone that you're trying to send this email to does not have access to the document that you've linked."

What is this magic? How does this work? How does Gmail possibly know what access my email recipient has to a document that I've merely linked?

The answer is that they use Google Zanzibar as a centralized authorization service for everything at Google. And Gmail is able to ask questions about permissions that exist ostensibly within the doc service.

A high-level overview of Google Zanzibar: it's a single authorization service for all of Google I mentioned. It deals primarily with storing relationships as edges in a directed graph, which is what I referred to earlier as ReBAC or Relationship-Based Access Control. Then they give their engineers a schema to interpret those relationships. So we're decoupling the relationships themselves from how they're interpreted to make authorization decisions.

The Google team spends about half the paper talking about performance, and it's throwing up some really impressive numbers. At Google, Zanzibar is doing 10 million queries per second at peak, and this is as of 2019, I'm sure it's grown since then. These are just the numbers in the paper. They're storing trillions of relationships, and they have a 95th percentile check query latency of 10 milliseconds.

If you know anything about writing applications that are human-facing, usually you want to keep those interactions under about 100 milliseconds. So 10 milliseconds isn't chewing up a lot of your budget in terms of latency. and maybe most impressively for this being a centralized service they've managed to keep 99.999% uptime, so five nines of uptime for this service which is super important for an authorization service which is usually in the line of fire for every request.

So, in an attempt to make this more concrete, and I hope I'm successful here, here's an example of using relationship-based Access Control to control policy decisions. I don't have access to a DSL if Google has one internally, but what I can show is how you express these things in our own Oso's DB schema language.

We have three different object types: users, organizations, and documents. And within those object types, we have relationships, those are in the red boxes or relations. So we can say that a user can be an owner of a document, a user can be a reader of a document, and a user can be an administrator of an organization. And then I only really have one document-level permission modeled here, but you could imagine read, write, view, delete, and many different permissions.

But the way we're expressing this permission is we're saying in order to be able to view a document, you need to either be a reader of the document, you need to be an owner of the document, or you need to be an administrator or have the admin permission on the organization that the document belongs to.

I don't know about you, but for me, this seems a straightforward way to express this kind of authorization logic. But then to try to make it more visual, I have a graphical representation of that same policy on the right-hand side. We can see, that I tried to color coordinate it so that you can see relations and how they relate between users and documents and organizations. And then the interesting one is there's the dotted line that connects view to admin, and so that's just showing you that you sometimes need to traverse the graph, and go through different objects to be able to make the decision that you need to make. In this case, we're considering what a user's role is in the organization to decide whether they have access to the document.

And then down at the bottom, I just have some example relationships. So these relationships are the way to read this is that user Sean has reader permissions on document some document, user Fred has reader on the same document, user Jill has owner on the same document. So we're setting up edges in a directed graph where we're relating a subject, which is often a user, back to a resource which in this case is either a document or an organization. And then you'll also see that we explicitly bind the organization and the document together with this relationship down on the bottom of the relationship grid.

By combining these relationships and the policy which allows us to interpret them, we can express in a natural way very, very complicated authorization schemes.

So Google Zanzibar, what is it good for? I've already mentioned that it's very scalable with traffic, with requirements, and also across geographies because it's built on top of a distributed database. It's very flexible. So within relationship-based Access Control, you can model multiple other authorization paradigms, for example, ABAC, which is attribute-based access control, or RBAC, role-based access control. Facebook wrote a paper about how you can model all of ABAC in RBAC.

You also get a single view of permissions for all services. So this is how Google is accomplishing that thing that I showed earlier where Gmail can understand access that only sensibly exists in the document service.

One of the super powerful things you can do when you have a system based on a graph like this is you do reverse index questions, for example, what can this user access, right? So if user Jake is talking to my web app, what documents can Jake see? If you think about how you would implement that in your own code or a policy engine, you can see how that would be pretty technically complicated.

Finally, it gives you a natural way to express permissions as long as the permissions that you're trying to express are dealing with relationships between people and other people, or people and data, or data and other data.

One of the downsides to adopting something like Zanzibar is that it's hard to do with data that's only available at request time. So for example, if your permission decision needs to take into account the IP address of the calling user, you don't have the IP address to store as a relationship ahead of time, so you can't use that in a graph walk or a graph traversal.

We do have a solution to that that I'll get to in a little bit, but in stock Zanzibar, in core Zanzibar that they write about in the paper, that's difficult. And then finally, it's just yet another distributed system to run. So if authorization is very important to you and a very critical piece of your business, then you'll run that distributed system. But often you might say, well, you know, I'm kind of up to my ears already in distributed systems, so I don't want to bring this in as well.

If you're interested in learning more about Zanzibar, I gave a full 60-minute talk just about the paper and the various sections of the paper for New York City's "Paper We Love" Meetup. There's a QR code there, but it's highly technical, and I break down all of the different distributed systems techniques that Zanzibar uses to get the performance and the uptime that we talked about a little bit earlier.

I'm an advocate of choosing the right tool for the right job. There's that phrase where if all you have is a hammer, everything looks like a nail. We don't just have hammers in authorization; we have other things. But just thinking about what the various things might be good for, I mentioned earlier that network-level or HTTP-level filtering because you have all of the data at request time is maybe a really good fit for a policy engine. And if you have to do a comparison of values, right, so if you're trying to say like is the content or the contents of this shopping cart greater than $100 and taking that into account when you do your permissions checks, that might also be more suited for a policy engine.

But where does Google Zanzibar shine?

Google Zanzibar shines when you're doing traditional role-based access control because you're able to associate the user with those roles and then interpret those roles all in a single place. It's great for RBAC with user-defined roles, so if you want to give your customers or your users control of what those things mean, you can do that with Google Zanzibar as well. We have an example of that in our playground.

If you want a single consistent view of permissions everywhere, Google Zanzibar is great for this, right? It's a network service, it's centralized, it has all of the data ahead of time, and it's great at being able to do that. Finally, if you need to feed billions or trillions of facts or relationships to your policy solution to make a decision, that's difficult to do with a policy engine. So if you try to check out what the research is on scaling Datalog to billions or trillions of facts, you'll find that this is actually cutting-edge Datalog stuff and isn't supported everywhere.

And then on the Zanzibar side, do have a little yellow warning sign there because Google Zanzibar isn't good at those things, but we've built a Google Zanzibar open-source service called SpiceDB, which is everything that the Zanzibar paper talks about plus a little bit more.

SpiceDB is our open-source love letter to Google Zanzibar. You can find it at Authzed or GitHub. It's fully open-source, it's Apache 2, which is a very liberal license.

SpiceDB has about 4,000 GitHub stars, and over 3,000 commits, we have 1,500 users on our Discord who are in there talking about authorization software, and we have over 40 contributors from multiple companies across the industry.

What I've been kind of alluding to is we have a thing called caveats which allows you to attach small bits of policy to individual relationships, and I have an example of that on the next slide.

We also have a few additional APIs that make it easier to work with this solution, giving you those reverse indexes that I talked about and giving you information about how permissions are changing over time. And we've also got a thoughtful, great schema language that we're constantly being told is sort of industry-leading.

And we want you to come to contribute, to use it, to do whatever you want to do, but just get involved in SpiceDB and the whole Google Zanzibar movement.

As promised, I have an example of caveats. So in this example, we just have a permission system where we're trying to decide who can unlock a car. In this case, the owner of the car can always unlock it, but the cleaner of the car might only be able to unlock it on certain weekdays. And then we have a single relationship where we say that user Jake, in this case, I'm the cleaner of the car, is only allowed to be the cleaner on Sundays. And then the car, the specific car that we're talking about in this case, is the Toyota Camry. I don't know why I picked that, I just did. So then we can see that as we're traversing this graph, we'll take into account what day of the week it's being called on.

This is really how we can fuse at scale. I can't talk about all of them because some of them are still in stealth mode or they haven't publicly disclosed that they're using us, but we have a few that are on our website under case studies. In terms of the metrics, we're very proud of our performance. We've done a lot of performance testing. We're not quite at the five nines that Google talks about with Zanzibar, but we're very, very close. And in terms of latency, we're often faster than what Google reports for Zanzibar, which we're very proud of.

We've also done a lot of work to make sure that SpiceDB is very operable, so it's very easy to run, it's very easy to scale. We have a lot of documentation about how to do that. And we're also very responsive on Discord if you run into any issues.

So, I think that's all the time I have. Thank you so much for your questions. If you have more questions, please come to Discord, please come to GitHub, file an issue, or start a discussion. We're very, very active in the community, and we're always looking to help. So, thank you so much.

Zed Tokens, Zookies, Consistency for Authorization

AuthZed — Wed, 18 Oct 2023 13:35:59 +0000

Secure authorization logic requires data consistency

If you've ever taken a course on databases or dove head-first into documentation, you may have bumped into the acronym "ACID". ACID describes the guarantees that are expected in order to effectively rely on a database system: atomicity, consistency, isolation, and durability. While it's rumored that the term "consistency" was fabricated in order to complete a catchy acronym, its colloquial usage by the software industry is no joke.

Taking a stroll across the Internet, one finds that many definitions of consistency are self-referential; they define consistency using the term "consistent". While one could attempt to define the term in a dense mathematical way, I'll leave that to Wikipedia. When most folks are referencing consistency, I find that they are trying to suss out the invariants that need to be preserved when their data is observed. This contract is largely driven by the problem and domain of the software being built and thus requires careful consideration.

The process of designing well-architected software often begins with determining the contracts and scope of the system. While the previous statement seems uncontroversial, I've seen time and time again that there is a critical component whose contract is often an afterthought: authorization systems.

Unless the domain is sensitive or regulated (e.g. healthcare), consideration of the design and requirements for access control is an exercise saved until a project has found product-market fit. By this time in the product's development, new functionality is assumed to be features that extend the existing foundation rather than something that entails its own end-to-end design.

Google's journey building authorization

When faced with the challenge of building access control for their products, the team at Google recognized what many had not: they needed to do their due diligence and design a system that considered the consistency model used for authorization data.

"Handling end-user access controls for distributed objects can be complex. The access controls must be handled consistently across all systems. This can be relatively straightforward if access is fixed at creation time (e.g. "this email may be read only by the recipient"). But if the ACL (access control list) can be updated, things get difficult quickly.

[..]

To solve this problem, we developed Zanzibar. Zanzibar has a central, highly-replicated, highly-available, and consistent ACL store."

Lea Kissner, co-author Google's Zanzibar authorization system

Kissner and their co-authors have done the world a favor by publishing the observed results of their design. The paper focuses on various aspects of the design, but explicitly chooses to focus on detailing the consistency model. The paper even goes so far as to give a name to a scenario that commonly plagues inconsistent authorization systems: The New Enemy Problem.

SpiceDB is the successor to Google Zanzibar

Fast-forward to today: an open source community is flourishing around SpiceDB, a re-imagining of Google's Zanzibar system that is not only faithful to the paper, but also inclusive in its design for those building software outside of a tech giant like Google.

SpiceDB is designed to start from a strongly consistent posture while providing users a way to relax overzealous consistency requirements in order to unlock higher performance.Zed Tokens, SpiceDB's analogue to the Zookie concept in Zanzibar, are a vital part of solving the New Enemy Problem. Zed Tokens are an encoded value that represents a particular point in time in SpiceDB. Requests to SpiceDB can optionally specify expected consistency when querying authorization data by providing Zed Tokens for when they need to query at or after a particular time (in addition to allowing the server to pick a time or specifying the most recent time in the system).

The most important benefit of designing for ad hoc consistency is that adopters can be confident that when they require strong consistency for a particular problem, they won't have to accept silent data corruption or re-architect their systems. SpiceDB is not the only system that takes this approach to consistency; the core databases powering Azure and AWS services, CosmosDB and DynamoDB respectively, also allow for this ad-hoc consistency model. And just as the cloud revolution moved the industry forward by democratizing elastic infrastructure, SpiceDB is now moving the industry toward a more secure way to build access control into our applications.

A Primer on Modern Enterprise Authorization (AuthZ) Systems

AuthZed — Wed, 06 Sep 2023 15:37:08 +0000

Introduction

A business operating complex software environments needs to provide end-user experiences that enable and delight users, but never at the cost of a poor security stance. In 2022, a lapse in security fetched on average 9.44M for a data breach in the US, and $4.35M globally according to IBM.

This post aims to help companies with their authorization decisions and systems and share what we see in the market through conversations with companies looking to solve their authorization challenges, specifically authorization that impacts end-user interaction with their products. I’ll cover how authorization became an issue for most companies, the different approaches, an introduction to Google Zanzibar–the solution the market is converging on, the prominent use cases driving businesses to adopt this new approach, and what you can expect when moving to a relationship-based access control system (ReBAC).

What is AuthZ?

Because the Internet is, by definition, a networked system that connects users, most Internet-facing software is designed with a multi-user experience in mind. These environments require constructs to facilitate a natural experience that protects users’ data: authentication , commonly referred to as authN, is the key that verifies an application-specific identity, typically called a user, and authorization , commonly referred to as authZ, dictates what doors that key can open for a user.

AuthZ plays a pivotal role in software security by ensuring that users have the appropriate level of access to different resources and functionalities within a system. At its core, authZ involves granting or denying access rights to specific resources or actions based on the identity and privileges of the user. It is crucial in preventing unauthorized access and verifying that only authorized users can perform specific actions within a system. It serves as a gatekeeper that protects sensitive data and functionalities. By implementing robust authorization mechanisms, developers can control the level of access granted to different users or roles, thereby safeguarding the system from potential security breaches.

Collectively in the context of a product’s end users, authN and authZ are called Customer Identity and Access Management (CIAM). Authorization is such a foundational part of the digital experience that its underlying design principles have become 99% invisible, even to developers, leading to fundamental challenges as a business scales.

Why AuthZ is an Issue

A company typically starts by aggregating all user requests into a single piece of software that tightly couples application logic with authZ. As the company’s product gains traction and its user base grows, the focus shifts to distributing the software and scaling infrastructure components, often ignoring a much-needed change to the authorization system. This further embeds an authorization construct not meant to handle a growing number of requirements.

From the business perspective, the two key limitations of a legacy authorization system are:

Permissions are inflexible : There isn’t a way to easily add additional constructs like user-defined roles, recursive relationships, attribute-based access control (ABAC), or fine-grained authorization (FGA).
Siloed permissions: as a company grows, it scales revenue by offering additional products; application teams then build bespoke authorization implementations that are hard to reason about and don’t consider a holistic user experience, especially at large scale.

Google set out to fix this problem, along with several “unique challenges involving data consistency and scalability.”

Google’s Solution to Authorization: Google Zanzibar

The confluence of business requirements driving the adoption of zero-trust architectures and 94% of consumers wanting more control of their data in near real-time galvanized the search for a modern authorization system. Google’s response was a modern approach that can scale with your business and maintain strict security requirements, now known as Google Zanzibar.

Among the requirements set forth by the Google Zanzibar team is “support for a rich set of access control policies as required by both consumer and enterprise applications” and “establish consistent semantics and user [developer] experience across applications.” Zanzibar powers authorization across hundreds of Google Products, including Google Calendar, Cloud, Drive, Maps, Photos, and YouTube. Notably, it unlocks unique experiences like cross-product authorization checks, e.g., Slack’s Gmail extension can check if a recipient has access to a Google Doc, unlocking growth through reduced friction points while maintaining user privacy.

Google Zanzibar is a relationship-based access control system (ReBAC), meaning that permissions are derived from the existence of a relationship between digital objects and users. This has positive performance implications, especially for recursive permissions, but, importantly, it’s a natural extension of sharing in the real world, which makes it intuitive for most developers.

Use-Cases Driving Adoption of Google Zanzibar

Product-Led Growth

To share something is a user choice and subsequent action. Companies we’re speaking with have learned that facilitating frictionless sharing can help onboard additional users to their platforms. For instance, a hiring platform we’re working with implements fine-grained authorization (FGA) so enterprise recruiters are comfortable exposing permissions to hiring managers related to the positions the managers are looking to fill. Hiring managers, in turn, can proactively engage with candidates and increase activity on the platform.

Another example is adding capabilities that foster more in-app experiences. For instance, a sharing economy company is boosting engagement with its platform by bringing resource management for users into their native applications instead of relying on third-party applications. Most companies we speak with share similar product-led growth initiatives built atop robust authorization.

Breaking into the Enterprise

Enhancing security and compliance is a key requirement for B2B companies looking to scale revenue within an enterprise market segment, and OWASP’s 2021 report cites broken access control as a top security concern. One of the main risks is “exposure of sensitive information to an unauthorized actor,” which is a core tenet of the Google Zanzibar paper; the system "must ensure consistency of access control decisions to respect user intentions.”

Enterprise users also require increased flexibility; these manifest as the following requirements for product teams:

Fine-grained authorization (FGA): the ability to control resources down to a granular level, e.g., a page in a document, though there is a balance, see How small is too small?
User-defined Roles and Permissions: beyond a typical application-defined Role-Based Access Control (RBAC) system, product teams need to allow end-user admins to create roles and associated permissions for delegating to internal teams.
Recursive relationships: at a certain scale, teams start owning teams. This is challenging for a traditional authorization system dealing with permissions stored in a relational database alongside application data.
Attributes-Based Access Control (ABAC): support for dynamic time-bound or otherwise caveated access.

What to Expect When Adopting Relationship-Based Access Control (ReBAC)

A crucial part of ReBAC systems like Google Zanzibar and our own Zanzibar-inspired authorization system, SpiceDB, is storing permissions data in a separate database; product-specific data (e.g. content of a social media post) is stored in the application database, while the data that drives who can edit that data live in the permissions database. If you have an existing authorization flow, you’ll have to translate that data into permissions data.

Modeling data is probably the most fun and intuitive part. SpiceDB, like other solutions, has a domain-specific language (DSL) called the SpiceDB Schema Language for defining the objects you want to create an authorization system for. The permissions schema defines the objects, e.g., users and documents, how they relate to each other, and the permissions those relationships define.

Since you’re writing permissions data, integration is a big part of the journey. A ReBAC authorization system is delivered over a gRPC or HTTP API; SpiceDB has libraries available in multiple languages to help developers get up to speed quickly. You’ll want to make sure whatever solution you choose delivers a solid developer experience.

Google Zanzibar doesn’t mention policy, but we’ve learned through our collaboration building Attribute-Based Access Control (ABAC) for Netflix that pairing policy with ReBAC is a powerful paradigm. An example of this capability is SpiceDB Caveats: Caveats: A Scalable Solution for Policy.

A common practice is to organize a core team of developers tasked with architecting and executing an overhaul to your authorization system. The effort must be cross-functional; you’ll want platform engineers, application engineers, and product managers to work together to ensure smooth adoption.

Get Started

Given how popular Google's approach to authorization has become, there are a number of new companies and projects looking to provide Zanzibar-aaS. At AuthZed, we've created a faithful open-source implementation of Google Zanzibar called SpiceDB, and offer managed commercial offerings that make it easy to get into production. Join the community on Discord or schedule a call to learn more!

Additional Reading

If you’re interested in learning more about Authorization and Google Zanzibar, we recommend reading the following posts:

Authorization Must Scale

AuthZed — Mon, 17 Jul 2023 12:05:53 +0000

As someone working at a YC startup, we are constantly reminded of Paul Graham’s classic advice "do things that don't scale". While we take this advice to heart for operating our business, we deliberately ignore it for the development of SpiceDB, the open source foundation of our company's products. Developing software is expensive, so why wouldn't you want to follow this advice? As it turns out, scalability is a requirement for authorization systems. There really hadn't been designs for scalable authorization solutions until Google published a paper documenting their internal system, Zanzibar. The Zanzibar paper is dense and includes many details that often cause the reader to miss the forest for the trees, but at its core it describes a system that scales. Rightfully, this breakthrough has gotten a lot of people excited—some of us are excited enough that we quit our jobs and started open source projects and a company 😉.

Your MVP's authorization doesn't scale and that's okay

Experienced software developers and entrepreneurs have probably raised an eyebrow: before making anything scale, you must first test your assumptions by building the minimum viable product. I wholeheartedly agree. I'm not going to sit here and try to sell you software complexity if your MVP can prove its viability with limited authorization functionality; MVPs are about building as little as possible to validate your idea. There are a few industries (e.g. healthcare, finance) that require complex access models on day one, but many MVPs need very little to serve their initial users. As long as there's technology, balancing complexity will always be the name of the game.

Scaling isn't exclusively a measure of traffic

It's easy to forget that organizations scale on various dimensions. While scale has become almost synonymous with software handling user traffic, it's more likely that you will experience the need to scale in one or more of the other dimensions first. Here's the typical order in which we see some dimensions of scale crop up in organizations:

Feature Development: amount of feature requests you can satisfy
Development Velocity: the speed at which teams can deliver complex features
Geographies: the localities of users you can serve
Traffic: the number of users you can serve

When the feature requests start coming in, that's the ideal time to consider your foundation for authorization. You might be tempted to extend your MVP authorization logic to add support for the new requirement, but doing so means setting yourself up an endless loop where implementing the next feature first requires refactoring security-critical authorization code to support it.

While building SpiceDB, we've interviewed thousands of software developers across industries. What we've concluded is that developers are struggling to keep their head above water when implementing anything beyond the basics of authorization. For the few that have gotten past the basics, they never have the time and focus needed to build tooling around their solution that empowers developers with the confidence to iterate on their designs.

The downstream effect of what we've observed is that most teams are limited in their ability to scale in all dimensions. Eventually this pain builds up until a breaking point has been met. At best, a new authorization system must be built and all feature development is paused while each individual product integrates with this new system. But the question is, who will build the new solution and how will you know it has any chance of being an improvement over the previous?

Friends don't let friends build authorization

Despite being a pillar of computer science with research beginning equally as early as other topics like database systems, authorization experts in the industry are an order of magnitude more rare to employ than database experts. Today, the developer zeitgeist acknowledges that it'd be foolish in most cases to implement your own database for an application, but has yet to internalize that this is equally if not more true for authorization systems. Meta and Google can afford to staff a team of experts to build an authorization platform for their product teams to consume, but pretty much everyone else cannot.

SpiceDB: the community for scalable authorization

Because there are so few authorization experts and everyone needs scalable authorization, it can only make sense to share expertise. The beauty of open source is that it's not only a great venue for sharing code, but also knowledge. That's why we created SpiceDB and are continuously investing in the community around it. By fostering an ecosystem that connects the folks solving everyday problems with the experts, we can build secure systems that scale with our needs. For real time discussion, you can join us on Discord or those that prefer asynchronous communication can ask questions and raise topics on SpiceDB's GitHub issue tracker.

DEV Community: AuthZed

SpiceDB Office Hours

Navigating the Developer Landscape: Building, Selling, and Securing Tools | That Tech Pod

Permissions Systems: Building an Authorization Lexicon

Why words matter

Semantics

Permissions systems

Authorization service

Glossary

Permissions Management with the Privacy Files Podcast

Introduction to Permissions and Access Control in Airports

Introduction to Permissions Management

The Importance and Challenges of Permissions Management

Jake's Background and Interest in Permissions Management

Common Pitfalls in Developing Permissions Code

Transitioning to AuthZed and the Concept of Cloud-Native

SpiceDB and AuthZed's Mission

Challenges in Selling Infrastructure to Enterprises

Real-World Implications of Permissions Mismanagement

Personal Insights and Advice on Privacy and Security

Authentication vs. Authorization

Authz and Authn, a primer

Authentication

What is Authentication?

The History of Authentication

T﻿imeline

Authorization

T﻿imeline

Zed File #Z-4902: Spanner Spikes

Zed File # Z-4902: Spanner Spikes

Policy-Based Access Control (PBAC) vs Relationship-Based Access Control (ReBAC): When You Should Use One or the Other

What is policy role-based access control?

What is the difference between ReBAC and PBAC?

ReBAC is a Database

ReBAC Stores Relationships

ReBAC Scales

When should you use a policy engine?

You got Policy in ReBAC. You got ReBAC in Policy.

What do we do at Authzed?

SpiceDB Tracing: How to interpret our OpenTelemetry traces

What you need to know about Permissions Management

What you will learn

1. The problem with permissions management, especially RBAC

2. Authorization models applied by companies

3. Google Zanzibar open-source implementations:

4. How to choose your permissions management system

Transcript

Related reading:

Understanding Google Zanzibar: A Comprehensive Overview

Google Zanzibar’s annotated paper

Zed Tokens, Zookies, Consistency for Authorization

Secure authorization logic requires data consistency

Google's journey building authorization

SpiceDB is the successor to Google Zanzibar

A Primer on Modern Enterprise Authorization (AuthZ) Systems

Introduction

What is AuthZ?

Why AuthZ is an Issue

Google’s Solution to Authorization: Google Zanzibar

Use-Cases Driving Adoption of Google Zanzibar

Product-Led Growth

Breaking into the Enterprise

What to Expect When Adopting Relationship-Based Access Control (ReBAC)

Get Started

Additional Reading

Authorization Must Scale

Your MVP's authorization doesn't scale and that's okay

Scaling isn't exclusively a measure of traffic

Friends don't let friends build authorization

SpiceDB: the community for scalable authorization

Related Reading

Timeline

Timeline