Three words that get used interchangeably until the day you have to design a platform with them. Here is the hierarchy, top down.
The account
The account is the container for your whole organisation, and you normally have one per cloud provider. You manage it in the account console, where billing and account admins live. Identities live here too: users, groups and service principals.
The workspace
A workspace is a single deployment of the Databricks UI, with its own notebooks, clusters, jobs and dashboards. Most teams run separate workspaces for development, test and production.
The one-line difference: the account is where you manage identities, billing and metastores for the organisation. A workspace is where people actually do the work.
The Unity Catalog metastore
The metastore is the top-level container for data objects and their permissions, and you create it at account level. The documentation is direct about the count: you must have one metastore for each region in which your organisation operates, and each regional metastore can be linked to any number of workspaces in that region.
So dev, test and prod in the same region attach to the same metastore, rather than each carrying its own legacy Hive metastore.
The three-level namespace
Inside the metastore, objects follow a three-level namespace: a catalog holds schemas, and a schema holds tables, views and volumes. You address a table as catalog.schema.table, for example prod.sales.orders.
Identity federation
With identity federation you configure users, service principals and groups once in the account console rather than repeating it in every workspace, then assign them to the workspaces that need them. One group, granted permissions on data once, applies everywhere it works. Workspace-local groups still exist in non-federated workspaces, but they cannot be granted Unity Catalog permissions.
Where the data actually sits
Metastore-level managed storage is optional, and managed tables are written there when it is configured. Data you already have in a bucket or container is registered as an external location, backed by a storage credential, and you grant access to that.
The one-line answer
The account holds identities and billing, workspaces hold the tools and the compute, and the metastore holds the data and who can see it, shared across every workspace in the region.
Watch it drawn step by step
I drew this as a 2:32 animated diagram: https://youtu.be/2yy9-BL_RDo
It is episode 3 of a data engineering interview prep series, in order here: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4
Top comments (0)