Data Products in Practice: Output Ports, Versioning, SLAs & Discoverability

Data products are what you get when you stop treating a dataset as a byproduct of a pipeline and start treating it as something you ship — an owned, versioned, governed unit of data that another team can find, understand, trust, and consume without ever messaging you on Slack to ask "is this table still populated?" The raw table is only the beginning. A table is bytes on disk with a name; a data product is those bytes plus an explicit interface, a contract that says what the fields mean and guarantees, a version that lets it change without breaking anyone, a published promise about freshness and quality, and an entry in a catalog so the rest of the company can discover it. The gap between the two is the difference between a data platform that scales to fifty teams and one where every new consumer is a bespoke, hand-held integration.

This guide is the senior-data-engineering walkthrough for closing that gap — for turning a modelled table into a real data product the way a data mesh operating model demands, framed the way interviewers actually probe it. It works through the four pillars that separate a product from a table: output ports (the SQL, API, file, and stream interfaces a product exposes behind one data contract), versioning (semantic versioning of schemas with backward and forward compatibility and a deprecation clock), SLAs and SLOs (measurable freshness, availability, and quality guarantees with monitors that page an owner), and discoverability (a data catalog entry with ownership, lineage, and self-serve access). Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the data product design practice library →, rehearse interface work on the API integration practice library →, and sharpen the contract axis with the data validation practice library →.

---

On this page

• Why a data product is more than a table
• Output ports — the interfaces a data product exposes
• Versioning — semantic versioning, compatibility, deprecation
• SLAs and SLOs — freshness, availability, quality
• Discoverability — catalogs, metadata, ownership, self-serve
• Cheat sheet — data products in practice
• Frequently asked questions
• Practice on PipeCode

---

1. Why a data product is more than a table

A table is data; a data product is data plus a contract, an owner, an SLO, and a way to find it

The one-sentence invariant: a data product is the smallest independently-consumable, independently-owned unit of data on a platform, and what promotes a raw table into one is not more rows but four added properties — an explicit output port and contract (how consumers access it and what the fields mean), a version (so it can evolve without breaking anyone), a published SLA/SLO (so consumers can trust it), and discoverability with clear ownership (so the rest of the org can find and self-serve it) — because under a data mesh operating model the bottleneck is never the query, it is the coupling, and a product exists precisely to decouple its consumers from its producer. Point ten teams at a raw table and every schema change is an incident and every question is a Slack ping; wrap it as a product and the contract, the version, and the catalog entry answer the questions for you.

The four pillars interviewers actually probe.

• Output ports and the contract. How do consumers access the data, and what is guaranteed about its shape and meaning? A product exposes deliberate interfaces (a SQL view, an API, a file drop, a stream) over a data contract — a typed schema plus semantics — rather than letting people SELECT your internal tables. Naming "the contract is the product boundary, not the storage" is the senior signal.
• Versioning. How does the product change without breaking its consumers? The senior answer names semantic versioning of the schema, backward/forward compatibility, and a deprecation window with parallel versions — not "we altered the table and told people in standup."
• SLAs and SLOs. What does the product promise, and can you measure it? A dataset is only trustworthy if freshness, availability, and quality are measured SLIs against published SLOs with an owner who gets paged. "It usually updates every morning" is not an SLA.
• Discoverability. Can a stranger find it, understand it, and trust it without asking you? The senior answer names a data catalog entry with owner, schema, semantics, lineage, SLO, and self-serve access — the properties that make a product usable by people you have never met.

The product mindset under data mesh.

• Domain ownership. A data product is owned by the domain team that knows the data, not by a central pipeline team — ownership is a named team with an on-call, not "the platform."
• The usability attributes. The data-mesh baseline says a product must be discoverable, addressable, understandable, trustworthy, natively accessible, interoperable, secure, and valuable on its own — a scorecard you can literally check a candidate product against.
• Product thinking. You treat internal consumers as customers: you version for them, you publish an SLA for them, you document for them, and you measure their satisfaction — the same discipline a public API team applies.
• Decoupling is the point. Every property (port, contract, version, catalog) exists to let a consumer depend on the promise and not the implementation, so you can re-model storage without breaking anyone.

The 2026 reality — the tooling has standardised.

• Data contracts. Open standards like the Open Data Contract Standard (ODCS) give a portable YAML shape for a product's schema, semantics, SLAs, and access — so the contract is machine-readable, testable, and version-controlled.
• Catalogs. DataHub, OpenMetadata, Unity Catalog, and Amundsen provide the discoverability layer — search, lineage, ownership, glossary — and all accept metadata pushed as code from your pipeline.
• Product descriptors. A data product is increasingly described by a single manifest (a descriptor file) that lists its ports, version, SLOs, and owner, checked into the same repo as the transformation that builds it.
• Governance as code. Access policies, PII tags, and quality tests live beside the descriptor and run in CI, so "governed" means "enforced in the pipeline," not "documented in a wiki."

What interviewers listen for.

• Do you distinguish the contract from the storage and call the contract the product boundary? — senior signal.
• Do you name a specific owner (a team with an on-call), not "the platform"? — required answer.
• Do you frame change through semantic versioning and deprecation, not "we altered the table"? — senior signal.
• Do you treat freshness/quality as measured SLOs, not vibes? — required answer.
• Do you make it discoverable in a catalog with lineage, so a stranger can self-serve? — senior signal.

Worked example — the table-to-product promotion checklist

Detailed explanation. The single most useful artifact for a data-product interview is a memorised checklist of what you add to a raw table to make it a product. Every senior discussion converges on it: given a table nobody outside your team should couple to, what do you bolt on before you publish it? Walk through promoting a raw orders table into a governed product.

• The starting point. raw.orders — an internal table, no docs, no version, no owner outside "whoever wrote the DAG."
• The tension. Consumers want to use it now; publishing it raw couples them to every future refactor.
• The rule. Add the four pillars — port/contract, version, SLA, discoverability — before you let anyone depend on it.

Question. For the raw orders table, list what each pillar contributes and what breaks if you skip it.

Input.

Pillar	What you add	What breaks if skipped
Output port + contract	a curated view/API + typed schema + semantics	consumers couple to internal columns
Versioning	a schema version + compatibility policy	every change is a silent break
SLA / SLO	measured freshness/quality + owner	nobody can trust or escalate
Discoverability	catalog entry + lineage + access	it is tribal knowledge

Code.

-- BEFORE: a raw internal table. No contract, no version, no owner, no guarantees.
-- Anyone who SELECTs this is coupled to its exact columns and refresh timing.
SELECT * FROM raw.orders;

-- AFTER: a curated PRODUCT surface — a stable view is the output port,
-- the column list + comments are the contract, and the product is versioned.
CREATE VIEW product.orders_v1 AS
SELECT
    order_id,                       -- stable key (contract: never reused)
    tenant_id,                      -- partition / access key
    order_status,                   -- enum: 'pending'|'paid'|'refunded'
    total_cents::bigint AS total_cents,   -- money as integer cents (semantics)
    order_ts                        -- event time, UTC (semantics)
FROM raw.orders
WHERE order_status IS NOT NULL;     -- product invariant: status is always set

COMMENT ON VIEW product.orders_v1 IS
  'Data product: orders v1. Owner: analytics-platform. SLO: fresh<=15m, quality>=99%.';

Step-by-step explanation.

1. The raw SELECT * is the anti-pattern: any consumer who runs it is coupled to the physical table — every renamed column, every added internal field, every re-partition becomes their problem. There is no boundary between your implementation and their dependency.
2. product.orders_v1 is the output port: a curated view that exposes only the columns that are part of the contract, so you can refactor raw.orders freely as long as the view still resolves. The view name carries the version (_v1) so the surface can evolve.
3. The explicit column list is the contract's schema, and the comments encode semantics that a raw table cannot: total_cents is integer cents, order_ts is UTC event time, order_status is a closed enum. Semantics are what make data understandable without asking the owner.
4. The WHERE order_status IS NOT NULL encodes a product invariant — a quality guarantee baked into the port, so a consumer never sees a half-written row that violates the contract.
5. The COMMENT is the seed of discoverability: owner and SLO travel with the object and can be harvested into a catalog. Skip any one pillar and the table stays a liability; add all four and it becomes something a stranger can safely build on.

Output.

Property	raw.orders (table)	product.orders_v1 (product)
Access boundary	physical columns	curated port/view
Meaning of fields	undocumented	semantics in the contract
Change safety	silent breaks	versioned + compatible
Trust	none	measured SLO + owner
Findable	tribal	catalog + lineage

Rule of thumb. Before you let anyone depend on a table, promote it: expose a curated port over a typed contract, stamp it with a version, publish a measured SLA with a named owner, and register it in the catalog. The four pillars are a checklist — a table missing any one of them is a liability, not a product.

Worked example — what interviewers actually probe

Detailed explanation. The senior data-product interview has a predictable escalation: an innocuous opener ("expose this dataset to another team"), then progressive narrowing to test whether you understand contracts, versioning, trust, and ownership. The candidates who name output ports, semantic versioning, measured SLOs, and catalog discoverability score highest.

• Ambiguous opener. "Team B wants your orders data. Give them access?"
• Follow-up 1. "You need to rename a column next sprint. Now what?" — probes versioning/compatibility.
• Follow-up 2. "Team B's dashboard was wrong for a day and nobody noticed. Why?" — probes SLAs/monitoring.
• Follow-up 3. "A third team wants it too but doesn't know it exists. How do they find it?" — probes discoverability.
• Follow-up 4. "Who is responsible when it breaks?" — probes ownership.

Question. Draft a senior data-product answer that pre-empts all four follow-ups without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Access	"grant them SELECT on the table"	"publish an output port over a contract"
Schema change	"alter the table, tell them"	"semantic version + backward-compat + deprecation"
Silent errors	"we'll notice eventually"	"freshness/quality SLOs with a monitor that pages"
Findability	"I'll send a Slack link"	"a catalog entry with lineage and access"
Ownership	"the pipeline team, I guess"	"a named domain team with an on-call"

Code.

Senior data-product answer template
===================================

1 — publish a port, not a table
  "I don't grant SELECT on my internal table; I publish an output port —
   a curated view/API over a typed data CONTRACT — so Team B depends on the
   promise, not my storage. I can refactor internals without breaking them."

2 — version everything
  "A rename is a MAJOR change. I don't mutate v1 in place; I ship v2 alongside
   it, keep v1 backward-compatible during a deprecation window, and track who
   still reads v1 before I sunset it."

3 — make trust measurable
  "The product publishes SLOs — freshness <= 15m, quality >= 99% — as MEASURED
   SLIs with a monitor. A day-long silent error is a monitoring gap, not bad
   luck; the monitor pages the owner the moment freshness or a quality test
   breaches."

4 — make it discoverable and owned
  "It lives in the data catalog with its schema, semantics, lineage, SLO, and
   a named owner team with an on-call. A third team searches, finds it,
   understands it, and self-serves access — no Slack archaeology, no me."

Step-by-step explanation.

1. Point 1 frames the whole answer around the contract as the boundary. Weak candidates hand out a table grant; naming "publish a port over a contract, depend on the promise not the storage" signals you understand decoupling, not just access.
2. Point 2 pre-empts the change follow-up. Volunteering semantic versioning and a deprecation window before the interviewer raises the rename shows you have evolved a product that other teams depend on.
3. Point 3 reframes a silent error as a monitoring gap: naming measured SLIs against published SLOs with a pager is the difference between "we'll notice" and "the owner is paged in minutes."
4. Point 4 closes on discoverability and ownership — a catalog entry and a named on-call team — which is the sentence that separates a platform engineer from someone emailing table names around.
5. The through-line is that every answer replaces a person (you, answering pings) with a product property (a contract, a version, a monitor, a catalog entry). That substitution is the entire point of the data-product model.

Output.

Grading criterion	Weak score	Senior score
Port/contract over raw table	rare	mandatory
Semantic versioning + deprecation	rare	senior signal
Measured SLOs with paging	occasional	mandatory
Catalog discoverability + lineage	rare	senior signal
Named owner with on-call	rare	required

Rule of thumb. The senior data-product answer replaces every "I'll handle it" with a product property: a contract instead of a table grant, a version instead of an in-place alter, a monitored SLO instead of hope, and a catalog entry instead of a Slack link. Rehearse the four substitutions; deploy them every interview.

Worked example — a data-product descriptor as the machine-readable contract

Detailed explanation. The artifact that makes all four pillars concrete and testable is a data-product descriptor: a single YAML manifest, checked into the repo beside the transformation, that declares the product's ports, version, SLOs, owner, and access. It is the thing CI validates and the catalog harvests. Write a descriptor for the orders product.

• One file. Ports, version, SLOs, owner, tags — all declared, all version-controlled.
• Machine-readable. CI can validate it; the catalog can ingest it; consumers can read it.
• The single source of truth. The descriptor, not tribal knowledge, defines the product.

Question. Write a data-product descriptor that captures the four pillars in one machine-readable file.

Input.

Section	Captures	Pillar
output_ports	interfaces + contract ref	ports/contract
version	semver of the product	versioning
slos	freshness/availability/quality	SLA/SLO
owner + catalog	team + discoverability	ownership/discovery

Code.

# data-product.yaml — one manifest, four pillars, checked into the repo.
apiVersion: dataproduct/v1
kind: DataProduct
metadata:
  id: orders
  version: 1.2.0                       # SEMANTIC VERSION of the product
  domain: commerce
  owner:
    team: analytics-platform          # a NAMED team...
    on_call: "#oncall-analytics"      # ...with an escalation path
  tags: [gold, pii-safe, revenue]
spec:
  description: >
    Curated, deduplicated orders with monetary values in integer cents,
    one row per order, event-timestamped in UTC.
  output_ports:                       # PILLAR 1 — the interfaces
    - name: sql
      type: table
      location: product.orders_v1
      contract: contracts/orders.yaml # -> the typed data CONTRACT
    - name: api
      type: rest
      location: https://api.example.com/orders
      contract: contracts/orders.yaml
  slos:                               # PILLAR 3 — the measured promises
    freshness: "<= 15m"
    availability: "99.9%"
    quality: ">= 99% rows pass contract tests"
  catalog:                            # PILLAR 4 — discoverability
    glossary: [order, revenue, tenant]
    lineage_upstream: [raw.orders, raw.payments]

Step-by-step explanation.

1. metadata.version: 1.2.0 puts the semantic version at the top level — the whole product is versioned, not just a schema file, so consumers can pin to a compatible range and CI can enforce that a breaking change bumps MAJOR.
2. owner.team plus on_call encode ownership as a named team with an escalation path — the descriptor makes "who do I page?" answerable from the manifest, not from git blame.
3. output_ports declares pillar 1: each port names its type (table, rest), its location, and — critically — points at a shared contract file, so every port serves the same typed contract rather than drifting apart.
4. slos declares pillar 3 as machine-readable targets a monitor can assert against; because they are in the manifest, a CI job can verify a monitor exists for each SLO before the product is allowed to publish.
5. The catalog block seeds pillar 4: glossary terms and upstream lineage let a catalog ingest the product and render its meaning and provenance, so the descriptor is simultaneously the contract, the config, and the documentation — one source of truth for all four pillars.

Output.

Question a consumer asks	Answered by	In the descriptor
How do I access it?	output_ports	sql + rest ports
What version is this?	metadata.version	1.2.0
Can I trust it?	slos	fresh/avail/quality
Who owns it?	owner	analytics-platform
Where did it come from?	catalog.lineage_upstream	raw.orders, raw.payments

Rule of thumb. Describe every data product with a single version-controlled descriptor that declares its ports, semantic version, SLOs, owner, and catalog metadata. When the manifest is the source of truth, CI can validate the contract, a monitor can assert the SLOs, and the catalog can harvest discoverability — the four pillars stop being a wiki page and become enforced code.

Senior interview question on promoting a table to a data product

A senior interviewer often opens with: "Another team wants to build on your internal orders table. Instead of granting them access, design it as a proper data product: what interface you expose and what contract backs it, how you'll change the schema later without breaking them, what you promise about freshness and quality and how you'd prove it, and how a third team that has never met you would discover it and self-serve — and who owns it when it breaks."

Solution Using a curated port, a semantic version, published SLOs, and a catalogued owner

-- 1. The output PORT: a curated, versioned view — the boundary consumers depend on.
CREATE VIEW product.orders_v1 AS
SELECT order_id, tenant_id, order_status,
       total_cents::bigint AS total_cents, order_ts
FROM raw.orders
WHERE order_status IS NOT NULL;      -- product invariant, not a raw dump

# 2. The CONTRACT the port serves — typed schema + semantics + quality rules.
contract: orders
version: 1.0.0
schema:
  - { name: order_id,    type: string,  required: true,  unique: true }
  - { name: tenant_id,   type: string,  required: true }
  - { name: order_status, type: string, enum: [pending, paid, refunded] }
  - { name: total_cents, type: long,    required: true,  min: 0 }   # integer cents
  - { name: order_ts,    type: timestamp, required: true }          # UTC event time
quality:
  - rule: not_null(order_id, tenant_id, order_status, total_cents, order_ts)
  - rule: unique(order_id)

# 3. The PRODUCT descriptor — version, SLOs, owner, catalog (discoverability).
kind: DataProduct
metadata: { id: orders, version: 1.0.0, owner: { team: analytics-platform, on_call: "#oncall-analytics" } }
spec:
  output_ports: [{ name: sql, type: table, location: product.orders_v1, contract: orders }]
  slos: { freshness: "<= 15m", availability: "99.9%", quality: ">= 99%" }
  catalog: { tags: [gold, pii-safe], lineage_upstream: [raw.orders, raw.payments] }

Step-by-step trace.

Decision	Before (raw table)	After (data product)
Access	GRANT SELECT ON raw.orders	curated product.orders_v1 port
Meaning	undocumented columns	typed contract + semantics
Change	in-place ALTER	semver + deprecation window
Trust	"usually fresh"	measured SLOs (fresh/avail/quality)
Findability	Slack link	catalog entry + lineage
Ownership	"the pipeline"	analytics-platform + on-call

After the promotion, Team B reads product.orders_v1 — a curated port whose typed contract fixes column names, types, the order_status enum, and the integer-cents semantics of total_cents, so the raw table underneath can be re-modelled freely. The product carries version 1.0.0, so a later rename ships as v2 alongside v1 rather than mutating it. The descriptor's SLOs are measurable promises a monitor asserts, the catalog block makes a third team able to search, understand lineage, and self-serve access, and owner.on_call names exactly who is paged when something breaks.

Output:

Metric	Raw-table access	Data product
Consumer coupling	to physical columns	to a stable contract
Cost of a schema change	breaks every consumer	additive or versioned
Trust	anecdotal	measured SLO + pager
Onboarding a new consumer	asks the owner	self-serves via catalog
Blast radius of an incident	unbounded, unowned	scoped, owned, escalatable

Why this works — concept by concept:

• Output port over a contract — a curated view/API backed by a typed contract is the product boundary, so consumers depend on the promise (names, types, semantics) and the producer can refactor storage underneath without breaking anyone.
• Semantic versioning — versioning the product means a breaking change ships as a new major alongside the old one during a deprecation window, converting "every change is an incident" into "changes are planned migrations."
• Measured SLOs — freshness, availability, and quality as SLIs against published SLOs with a monitor turn trust from an anecdote into a paged, enforceable guarantee with a clear owner.
• Catalogued ownership — a catalog entry with lineage, tags, and a named on-call team makes the product discoverable and self-serve, so a stranger onboards without the producer as a bottleneck.
• Cost — a curated view, a contract file, a descriptor, and a monitor, versus a table grant and a growing pile of coupling. The eliminated cost is the O(consumers) hand-holding a raw table demands — one product publish replaces N bespoke integrations, so onboarding is O(1) self-serve instead of O(N) Slack threads.

Design
Topic — design
Design problems on data products and platform boundaries

Practice →

Dimensional modeling Topic — dimensional-modeling Dimensional modeling problems on curated, consumer-ready datasets

Practice →

---

2. Output ports — the interfaces a data product exposes

One dataset, many doors: SQL, API, file, and stream output ports behind one contract

The mental model in one line: an output port is a typed, governed access interface of a data product — a SQL/table port, a REST/GraphQL API port, a file/object-export port, or a stream/topic port — and the crucial discipline is that every port serves the same data contract (a typed schema plus semantics, quality rules, and an access policy), so ports are additive (you add a door for a new consumer without changing the product) while the contract is the invariant (the one promise all doors keep) — which is what lets a batch team, an app, and an ML pipeline each consume the product in their native shape without the producer maintaining three divergent datasets. Expose the contract, not the storage; add ports, never fork the truth.

The output-port types.

• SQL / table port. A curated view or table (Postgres view, a Snowflake/BigQuery table, an Iceberg table) consumers query directly — the native shape for analysts and BI. The port is the view, not the raw table behind it.
• API port. A REST or GraphQL endpoint (PostgREST, Hasura, a service) for applications that need request/response access with authorization — the native shape for product surfaces.
• File / object port. A columnar file drop (Parquet on S3/GCS, a partitioned Iceberg/Delta location) for bulk consumers, partners, and downstream pipelines — the native shape for batch and portability.
• Stream / topic port. A Kafka/Pulsar topic (or a CDC feed) for consumers that need events as they happen — the native shape for real-time and event-driven consumers.

The contract behind every port.

• Typed schema. Field names, types, nullability, and keys — the structural promise, ideally expressed once (ODCS YAML, an Avro/Protobuf schema) and reused by every port.
• Semantics. What the fields mean: units (total_cents is integer cents), time semantics (event time vs processing time, UTC), enums, and grain (one row per order). Semantics are what make the data understandable without the owner.
• Quality rules. Not-null, uniqueness, referential, and range constraints that the product guarantees — the same rules that back the quality SLO.
• Access policy. Who may read which columns/rows, PII classification, and masking — governance that travels with the contract, not bolted onto one port.

Ports are additive; the contract is the invariant.

• Add a door, not a fork. A new consumer type gets a new port over the same contract — never a second, divergent copy of the data that drifts out of sync.
• One schema, many serialisations. The SQL port serves rows, the file port Parquet, the stream port Avro records — but all project the same logical contract, so a field means the same thing everywhere.
• The port is replaceable. Because consumers depend on the contract, you can swap the storage behind a port (Postgres view → Iceberg table) without a consumer noticing, as long as the contract holds.
• Decouple grain from delivery. The contract fixes the grain and meaning; the port decides batch vs stream vs request/response delivery — two orthogonal decisions.

The failure modes senior engineers pre-empt.

• Leaking the internal schema. Exposing raw tables as the port couples consumers to your implementation. Mitigation: a curated view/API is the port; the raw table stays private.
• A port with no contract. A file drop or topic with no declared schema/semantics is just bytes — consumers reverse-engineer meaning and break on every change. Mitigation: every port references the typed contract; no schemaless doors.
• Forked datasets per consumer. Cutting a bespoke copy for each team creates N drifting truths. Mitigation: add a port over one product; never a second source of truth.

Worked example — write a data contract for an output port

Detailed explanation. The canonical first step: express the product's contract once as a typed, machine-readable document — schema, semantics, quality, and access — that every port then serves. Write an ODCS-style contract for the orders product.

• One document. Schema + semantics + quality + access, version-controlled.
• Reused by all ports. SQL, API, file, and stream ports all point at this.
• Testable. CI can assert produced data satisfies it.

Question. Write a data contract that declares the typed schema, semantics, quality rules, and access policy for the orders product.

Input.

Contract section	Declares	Example
schema	fields, types, keys	order_id: string, unique
semantics	units, grain, time	cents; one row/order; UTC
quality	guaranteed rules	not-null, unique, range
access	classification + policy	tenant_id row scope; PII tags

Code.

# contracts/orders.yaml — the ONE contract every output port serves (ODCS-style).
apiVersion: odcs/v3
kind: DataContract
id: orders
version: 1.2.0
description: One row per order; monetary values in integer cents; event time in UTC.

schema:
  - name: order_id
    logicalType: string
    required: true
    unique: true
    description: Immutable order identifier; never reused.
  - name: tenant_id
    logicalType: string
    required: true
    classification: internal          # access key, not PII
  - name: order_status
    logicalType: string
    required: true
    allowedValues: [pending, paid, refunded]
  - name: total_cents
    logicalType: long
    required: true
    minimum: 0
    unit: cents                       # SEMANTICS: integer cents, not dollars
  - name: order_ts
    logicalType: timestamp
    required: true
    timezone: UTC                     # SEMANTICS: event time, UTC

quality:                               # the guarantees behind the quality SLO
  - rule: not_null
    columns: [order_id, tenant_id, order_status, total_cents, order_ts]
  - rule: unique
    columns: [order_id]
  - rule: range
    column: total_cents
    min: 0

access:                                # governance travels with the contract
  row_policy: "tenant_id = caller.tenant_id"
  pii: false
  classification: gold

Step-by-step explanation.

1. The contract lives in one file (contracts/orders.yaml) with its own version — so the schema is versioned independently and every port references this document rather than restating the schema and drifting.
2. The schema block is the structural promise: names, logicalType, required, unique, and allowedValues fix the shape. A logical type (not a Postgres or BigQuery type) keeps the contract portable across the SQL, file, and stream ports.
3. The unit: cents and timezone: UTC fields encode semantics the raw types cannot — the two most common data bugs (dollars-vs-cents and local-vs-UTC) are prevented by making meaning explicit in the contract, not implicit in a column name.
4. The quality rules are the same assertions that back the quality SLO in the descriptor — not-null, unique, range — so "the product guarantees X" and "CI tests X" are the same list, and the contract is testable, not aspirational.
5. The access block puts governance in the contract: a row policy scoping reads to the caller's tenant, a PII flag, and a classification. Because access lives in the contract, every port enforces the same policy — you cannot accidentally expose a stricter port and a looser one.

Output.

Consumer question	Contract answers	Applies to
"Is total_cents dollars?"	unit: cents	all ports
"Can order_id repeat?"	unique: true	all ports
"Is order_ts local?"	timezone: UTC	all ports
"Can I see other tenants?"	row_policy	all ports

Rule of thumb. Express the contract once as a typed, machine-readable document — schema, semantics (units, grain, time zone), quality rules, and access policy — and have every output port reference it. A contract that lives in one versioned file is portable across ports, testable in CI, and the single answer to every "what does this field mean?" question.

Worked example — expose one product through multiple ports

Detailed explanation. The feature that makes a data product interoperable is serving several ports over one contract: an analyst queries the SQL port, an app hits the API port, a partner pulls the file port, and a streaming consumer subscribes to the topic — all the same logical data. Wire four ports over the orders contract.

• One contract. contracts/orders.yaml from the previous example.
• Four ports. SQL view, REST API, Parquet export, Kafka topic.
• The invariant. A field means the same thing at every door.

Question. Expose the orders product through SQL, API, file, and stream ports, all serving the same contract.

Input.

Port	Serialisation	Consumer	Delivery
SQL / table	rows	analysts / BI	query
REST API	JSON	applications	request/response
File / object	Parquet	partners / batch	bulk pull
Stream / topic	Avro	real-time consumers	event push

Code.

-- PORT 1 (SQL/table): a curated view projecting the contract's columns.
CREATE VIEW product.orders_v1 AS
SELECT order_id, tenant_id, order_status, total_cents, order_ts
FROM raw.orders WHERE order_status IS NOT NULL;

# PORT 2 (REST API): PostgREST/Hasura over the SAME view, same contract.
api:
  path: /orders
  source: product.orders_v1
  contract: contracts/orders.yaml
  authz: row_policy   # tenant scoping from the contract's access block

# PORT 3 (file/object): a scheduled Parquet export of the SAME contract.
# The column list and types are DERIVED from the contract, not hand-written.
(spark.table("product.orders_v1")
      .repartition("order_status")
      .write.mode("overwrite")
      .parquet("s3://data-products/orders/v1/"))   # partitioned object port

# PORT 4 (stream/topic): a Kafka topic whose Avro schema IS the contract schema.
topic: orders.v1
key: order_id
value_schema: contracts/orders.avsc   # generated from contracts/orders.yaml
compatibility: BACKWARD                # registry enforces safe evolution

Step-by-step explanation.

1. Port 1 is the SQL view — the curated projection of the contract's five columns. It is the canonical materialisation; the other ports derive from it or from the same contract, so there is one source of truth, not four.
2. Port 2 puts a REST API over the same view, and its authorization comes from the contract's access block — so the app port and the SQL port enforce identical tenant scoping. The API is a door, not a different dataset.
3. Port 3 exports Parquet whose columns are derived from the contract, not re-typed by hand — partitioned for bulk consumers. Because it projects product.orders_v1, a partner's file has the same fields and semantics as the analyst's query.
4. Port 4 is a Kafka topic whose Avro value_schema is generated from the same contract, so a streamed record and a queried row agree field-for-field. The registry's BACKWARD compatibility ties the stream port into the versioning discipline of section 3.
5. The invariant across all four: total_cents is integer cents and order_ts is UTC event time at every door, because every port projects one contract. Adding a fifth consumer type means adding a fifth port — never forking a fifth divergent copy that drifts.

Output.

Consumer	Uses port	Gets
Analyst / BI	SQL view	rows, live query
Application	REST API	JSON, tenant-scoped
Partner / batch	Parquet file	bulk, portable
Real-time consumer	Kafka topic	events, as they happen

Rule of thumb. Serve every consumer type through its own output port — SQL, API, file, stream — but derive all of them from one contract so a field means the same thing at every door. New consumer, new port; never a new forked dataset. Ports are how a single product stays interoperable across batch, request/response, and streaming worlds.

Worked example — schema enforcement at the port

Detailed explanation. A port is only trustworthy if produced data actually satisfies the contract before it is published. The discipline is a CI/pipeline gate that validates the output against the contract and blocks the publish on violation. Add a contract-enforcement check to the orders product build.

• The gate. Validate produced rows against contracts/orders.yaml.
• The action. Fail the build (do not publish the port) on any violation.
• The result. The port never exposes data that breaks the contract.

Question. Add a build-time check that validates the produced orders data against its contract and blocks publication if any rule fails.

Input.

Check	Contract rule	On failure
not-null	not_null(...)	block publish
unique	unique(order_id)	block publish
range	total_cents >= 0	block publish
enum	order_status in (...)	block publish

Code.

# enforce_contract.py — run in CI/the pipeline BEFORE the port is published.
import yaml, sys
from pyspark.sql import functions as F

contract = yaml.safe_load(open("contracts/orders.yaml"))
df = spark.table("product.orders_v1")

violations = {}

# not-null rules from the contract
required = [c["name"] for c in contract["schema"] if c.get("required")]
for col in required:
    n = df.filter(F.col(col).isNull()).count()
    if n: violations[f"null:{col}"] = n

# uniqueness
dupes = df.groupBy("order_id").count().filter("count > 1").count()
if dupes: violations["dup:order_id"] = dupes

# range + enum
neg = df.filter(F.col("total_cents") < 0).count()
if neg: violations["range:total_cents"] = neg
bad_status = df.filter(~F.col("order_status").isin("pending","paid","refunded")).count()
if bad_status: violations["enum:order_status"] = bad_status

if violations:
    print("CONTRACT VIOLATION — port NOT published:", violations)
    sys.exit(1)          # BLOCK the publish; the old port stays live
print("contract OK — publishing orders port")

Step-by-step explanation.

1. The check loads the same contracts/orders.yaml the ports reference, so enforcement and publication share one definition — there is no second, drifting copy of "the rules" to keep in sync.
2. It derives the required columns from the contract's schema and counts nulls in each — the not-null guarantee is tested, not assumed, so a bug upstream that starts emitting null order_status is caught before any consumer sees it.
3. Uniqueness, range, and enum checks assert the remaining contract rules; each is a cheap aggregate over the produced data, and any non-zero count is recorded as a violation.
4. The sys.exit(1) is the load-bearing line: on any violation the build fails and the port is not republished, so consumers keep reading the last-good version instead of a contract-breaking one — failing closed, not open.
5. Wiring this into CI (or the pipeline's publish step) makes the quality SLO real: "≥ 99% rows pass contract tests" is enforced at the gate, so the SLA in the descriptor and the check in the pipeline are the same promise — measured, not aspirational.

Output.

Build	Contract check	Port state
clean data	passes	published (new version)
null order_status	not-null fails	blocked; last-good stays
duplicate order_id	unique fails	blocked; last-good stays
negative total_cents	range fails	blocked; last-good stays

Rule of thumb. Gate every port behind a contract-enforcement check that runs in CI/the pipeline and fails closed — on any violation, block the publish and keep the last-good version live. Enforcement that reads the same contract file the ports serve is what turns "the product guarantees quality" from a sentence into a build step.

Senior interview question on output ports and contracts

A senior interviewer might ask: "Your orders data product has three very different consumers — an analytics team that wants SQL, an app that wants a low-latency API, and a partner that wants bulk files — and a fourth, a fraud service, that wants events in real time. Design the output ports so all four consume the same product without you maintaining four divergent datasets: what contract backs the ports, how a field keeps one meaning across SQL, JSON, Parquet, and a stream, and how you stop a bad build from publishing data that violates the contract."

Solution Using one contract, four derived ports, and a fail-closed enforcement gate

# 1. ONE contract — the invariant every port serves (typed schema + semantics).
id: orders
version: 1.2.0
schema:
  - { name: order_id, logicalType: string, required: true, unique: true }
  - { name: tenant_id, logicalType: string, required: true }
  - { name: order_status, logicalType: string, allowedValues: [pending, paid, refunded] }
  - { name: total_cents, logicalType: long, minimum: 0, unit: cents }
  - { name: order_ts, logicalType: timestamp, timezone: UTC }
access: { row_policy: "tenant_id = caller.tenant_id", pii: false }

-- 2. The canonical materialisation the ports derive from.
CREATE VIEW product.orders_v1 AS
SELECT order_id, tenant_id, order_status, total_cents, order_ts
FROM raw.orders WHERE order_status IS NOT NULL;

# 3. Four PORTS over the one contract — additive, never forked.
output_ports:
  - { name: sql,    type: table,  location: product.orders_v1 }
  - { name: api,    type: rest,   location: /orders, authz: row_policy }
  - { name: file,   type: object, location: s3://data-products/orders/v1/ (parquet) }
  - { name: stream, type: topic,  location: orders.v1, schema: orders.avsc, compatibility: BACKWARD }

# 4. Fail-closed enforcement — validate against the contract BEFORE publishing.
if contract_violations(spark.table("product.orders_v1"), "contracts/orders.yaml"):
    sys.exit(1)          # block publish; last-good ports stay live
publish_ports()          # only reached when the data satisfies the contract

Step-by-step trace.

Layer	Component	Purpose
Invariant	one contract	schema + semantics + access, versioned
Canonical	product.orders_v1 view	single source the ports derive from
SQL port	curated view	analysts / BI query it
API port	REST over the view	app, tenant-scoped by the contract
File port	Parquet export	partners / batch, same columns
Stream port	Kafka topic (Avro)	fraud service, events as they happen
Gate	fail-closed check	no contract-breaking publish

After the design, all four consumers read one product: the analyst queries product.orders_v1, the app hits the REST port with tenant scoping from the contract's access block, the partner pulls Parquet whose columns are the contract's columns, and the fraud service subscribes to orders.v1 whose Avro schema is the contract's schema. total_cents is integer cents and order_ts is UTC at every door because every port projects the one contract. The enforcement gate blocks any build whose data violates the contract, so no port ever exposes a broken row — and adding a fifth consumer is a fifth port, not a fifth dataset.

Output:

Metric	Four bespoke datasets	One product, four ports
Sources of truth	4 (drift)	1 (contract)
Meaning of a field	may differ per copy	identical at every port
Cost of a new consumer	a new pipeline	a new port
Bad-data exposure	per-copy, ad hoc	blocked at one gate
Governance	per-copy	one access policy, all ports

Why this works — concept by concept:

• One contract, many ports — expressing schema, semantics, and access once and having every port serve it makes the contract the invariant and the ports additive, so consumers get their native shape without the product forking into divergent copies.
• Canonical materialisation — a single curated view the ports derive from means there is one place the data is defined and four projections of it, not four independently-maintained truths that drift apart.
• Semantics in the contract — unit: cents and timezone: UTC travel to the SQL row, the JSON field, the Parquet column, and the Avro record identically, killing the dollars-vs-cents and local-vs-UTC bugs at the source.
• Fail-closed enforcement — validating produced data against the contract and blocking the publish on violation means a port never exposes a row that breaks its promise; consumers read the last-good version instead of a broken one.
• Cost — one contract, one canonical view, four thin ports, and one gate, versus four pipelines each with its own schema, tests, and governance. The eliminated cost is O(consumers) duplicated datasets — one product serves all four, so maintenance is O(1) contract instead of O(N) forks.

API integration
Topic — api-integration
API integration problems on output ports and data contracts

Practice →

Data processing Topic — data-processing Data processing problems on multi-port serialisation and enforcement

Practice →

---

3. Versioning — semantic versioning, compatibility, deprecation

Semantic versioning for schemas: evolve without breaking, deprecate on a clock

The mental model in one line: versioning a data product means applying semantic versioning (MAJOR.MINOR.PATCH) to its contract — a MAJOR bump for a breaking change (drop, rename, retype, tighten), a MINOR bump for an additive backward-compatible change (a new nullable/optional field), a PATCH for a non-schema fix (docs, metadata) — governed by an explicit compatibility mode (backward, forward, or full) that a schema registry can enforce, so a producer can never silently ship a break; and when a break is genuinely needed, you run the old and new majors in parallel behind a deprecation clock, migrate consumers off the old one using read telemetry, and only then sunset it. Additive changes are free; breaking changes cost a migration — and semantic versioning is how you price the difference so nobody is surprised.

Semantic versioning mapped to schema changes.

• MAJOR — breaking. Dropping a field, renaming a field, changing a type incompatibly, tightening a constraint (adding NOT NULL, narrowing an enum), or changing semantics (dollars → cents). Consumers must change to keep working. Bump the major; do not do it in place.
• MINOR — additive, backward-compatible. Adding a new optional/nullable field, adding a new enum value a consumer can ignore, relaxing a constraint. Existing consumers keep working unchanged. Bump the minor.
• PATCH — non-schema. Documentation fixes, tag changes, description edits, a bug fix in the pipeline that does not change the schema or semantics. Bump the patch.
• The test. "Will an existing consumer break if I ship this without telling them?" Yes → MAJOR. No, but there is new stuff → MINOR. No schema change at all → PATCH.

Backward vs forward vs full compatibility.

• Backward compatibility. New schema can read old data — the common default for consumers: you can add an optional field or drop... actually, in registry terms, backward-compatible changes let a consumer on the new schema read data written with the old schema. Achieved by adding optional fields with defaults.
• Forward compatibility. Old schema can read new data — important when producers upgrade before consumers: an old consumer must tolerate data written by a newer producer. Achieved by only adding optional fields the old reader ignores.
• Full compatibility. Both directions — the safest and most constrained mode, where changes are limited to adding/removing optional fields with defaults.
• Where it is enforced. A schema registry (Confluent, Apicurio) checks a proposed schema against the compatibility mode and rejects an incompatible one at registration time — so a break cannot reach the topic/port.

Deprecation — retiring a version safely.

• Parallel versions. A breaking change ships as a new major (v2) alongside v1; both are live, both serve consumers, so nobody is forced to migrate on your schedule.
• A deprecation clock. v1 is marked deprecated with a sunset date in the descriptor/catalog, giving consumers a known window to migrate.
• Migration telemetry. You track who still reads v1 (query logs, API keys, consumer groups) so you know when it is safe to sunset — you retire on evidence, not on a calendar alone.
• The sunset. When reads of v1 reach zero (or only known stragglers remain), you retire it — and the deprecation window, not a surprise, is what made that safe.

The failure modes senior engineers pre-empt.

• Silent breaking change. Renaming a column in place breaks every consumer with no warning. Mitigation: registry-enforced compatibility + a MAJOR bump + parallel versions.
• Version nowhere in the contract. If the version is not in the schema/port/descriptor, consumers cannot pin to it. Mitigation: version the contract, the port name, and the descriptor.
• Forever-two-versions. Running v1 and v2 indefinitely doubles maintenance forever. Mitigation: a deprecation clock plus read telemetry to force and verify migration.

Worked example — classify a schema change as MAJOR, MINOR, or PATCH

Detailed explanation. The skill an interviewer tests first is classification: given a proposed change, is it MAJOR, MINOR, or PATCH? Get this wrong and you either break consumers (calling a break "minor") or over-version (calling docs "major"). Classify a batch of changes to the orders contract.

• The rule. Would an unchanged consumer break? MAJOR. New but ignorable? MINOR. No schema change? PATCH.
• The batch. Add nullable field, add enum value, rename, drop, tighten null, fix a description.
• The output. A semver bump per change and why.

Question. For each proposed change to the orders contract, assign MAJOR/MINOR/PATCH and justify it by consumer impact.

Input.

Proposed change	Breaks an unchanged consumer?	Bump
Add nullable discount_cents	no	MINOR
Add enum value disputed	maybe (if they switch-exhaustively)	MINOR*
Rename total_cents → gross_cents	yes	MAJOR
Drop order_status	yes	MAJOR
Make tenant_id NOT NULL (was nullable)	yes (tighten)	MAJOR
Fix a typo in the description	no schema change	PATCH

Code.

Classify by CONSUMER IMPACT, not by how big the diff looks.

MINOR  add nullable discount_cents
   old consumer ignores the new column -> keeps working.        1.2.0 -> 1.3.0

MINOR* add enum value 'disputed' to order_status
   old consumer that lists known values still reads rows, but a
   consumer that assumes a CLOSED set may mishandle 'disputed'.
   Safe as MINOR only if the contract documented the enum as OPEN. 1.3.0 -> 1.4.0

MAJOR  rename total_cents -> gross_cents
   every consumer selecting total_cents breaks.                  1.x   -> 2.0.0

MAJOR  drop order_status
   consumers filtering on it break.                              -> 2.0.0

MAJOR  tighten tenant_id to NOT NULL
   a producer emitting nulls now fails; a tightening is breaking.-> 2.0.0

PATCH  fix a description typo
   no schema, no semantics change.                              1.2.0 -> 1.2.1

Step-by-step explanation.

1. Adding a nullable discount_cents is MINOR because an existing consumer that never selects it is completely unaffected — the schema grew, but the old contract still holds for everyone who used it. Additive-and-optional is the definition of a backward-compatible minor.
2. Adding an enum value is the subtle case: it is safe as MINOR only if the enum was documented as open. A consumer that exhaustively switches on a closed enum can mishandle a new value, so the contract must have promised the set is open for this to be non-breaking — otherwise it is MAJOR.
3. Renaming and dropping are unambiguous MAJOR changes: any consumer referencing the old name breaks immediately. Rename is really "drop + add," and the drop half is the break — so it is never a minor, no matter how small the diff looks.
4. Tightening tenant_id to NOT NULL is MAJOR even though it removes nothing, because it narrows what the producer may emit — a tightening can break the producer or reject previously-valid data. Loosening (NOT NULL → nullable) would be backward-compatible; tightening is not.
5. The description typo is PATCH: no schema, no semantics, no consumer impact. The discipline is to classify by "does an unchanged consumer break?" — the size of the code diff is irrelevant, only the contract impact matters.

Output.

Change	Bump	Version
add nullable discount_cents	MINOR	1.3.0
add open-enum value	MINOR	1.4.0
rename total_cents	MAJOR	2.0.0
drop order_status	MAJOR	2.0.0
tighten tenant_id NOT NULL	MAJOR	2.0.0
description typo	PATCH	1.2.1

Rule of thumb. Classify every schema change by one question — "would an unchanged consumer break?" Yes is MAJOR (rename, drop, retype, tighten); no-but-additive is MINOR (new optional field, open-enum value); no-schema-change is PATCH (docs, tags). The diff size never decides the bump; consumer impact does.

Worked example — backward-compatible evolution enforced by a schema registry

Detailed explanation. Classification is a judgement; a schema registry makes it enforced. Register the contract's schema with a compatibility mode and the registry rejects any incompatible change at registration time — so a break physically cannot reach the stream/port. Set BACKWARD compatibility on the orders topic and evolve it.

• The mode. BACKWARD — a consumer on the new schema can read old data.
• The safe change. Add an optional field with a default → accepted.
• The unsafe change. Remove a required field / rename → rejected.

Question. Configure a registry with BACKWARD compatibility and show which evolution it accepts and which it rejects.

Input.

Change to Avro schema	Backward-compatible?	Registry verdict
add field with a default	yes	accepted
add field without a default	no	rejected
remove an optional field	yes	accepted
rename a field	no	rejected

Code.

// orders v1 (registered). Avro — the stream port's contract schema.
{ "type": "record", "name": "Order", "fields": [
    { "name": "order_id",     "type": "string" },
    { "name": "tenant_id",    "type": "string" },
    { "name": "order_status", "type": "string" },
    { "name": "total_cents",  "type": "long" },
    { "name": "order_ts",     "type": "long", "logicalType": "timestamp-millis" }
]}

// orders v1.1 — ADD an optional field WITH A DEFAULT. BACKWARD-compatible.
// A consumer on this schema reading OLD data gets discount_cents = 0.
{ "type": "record", "name": "Order", "fields": [
    { "name": "order_id",      "type": "string" },
    { "name": "tenant_id",     "type": "string" },
    { "name": "order_status",  "type": "string" },
    { "name": "total_cents",   "type": "long" },
    { "name": "order_ts",      "type": "long", "logicalType": "timestamp-millis" },
    { "name": "discount_cents","type": "long", "default": 0 }   // <- default is required
]}

# The registry ENFORCES the mode at registration time.
# Set the subject's compatibility, then try to register the new schema.
curl -X PUT $REG/config/orders-value -d '{"compatibility": "BACKWARD"}'

# v1.1 (adds field WITH default)  -> HTTP 200, registered (id 42)
# a schema that RENAMES total_cents -> gross_cents:
#   -> HTTP 409 Conflict: "Schema being registered is incompatible with BACKWARD"
#   the break NEVER reaches the topic; the producer's deploy fails fast.

Step-by-step explanation.

1. The BACKWARD mode means a consumer running the new schema must be able to read data written with the old schema. That is exactly what protects already-deployed consumers when the producer upgrades first — the common ordering in practice.
2. Adding discount_cents with a default is backward-compatible: a new-schema consumer reading an old record (which lacks the field) fills in the default 0, so it never fails. The default is the load-bearing detail — without it the field is required and the change is rejected.
3. The registry PUT ... /config sets the compatibility contract for the subject, and every subsequent registration is checked against it — governance moves from "reviewer remembered to check" to "the system refuses incompatible schemas."
4. A rename (total_cents → gross_cents) is rejected with 409 Conflict: the new schema cannot read old data (old records have total_cents, which the new schema no longer knows), so it violates BACKWARD and the producer's deploy fails fast instead of silently breaking consumers at read time.
5. This is the enforcement half of section 3's discipline: classification tells you a rename is MAJOR, and the registry guarantees you cannot slip it through as an in-place change — a MAJOR must go to a new subject/version, which is the parallel-run of the next example.

Output.

Evolution	Compatible with BACKWARD	Registry
add field + default	yes	registered
add field, no default	no	409 rejected
remove optional field	yes	registered
rename field	no	409 rejected

Rule of thumb. Put the contract's schema in a registry with an explicit compatibility mode (BACKWARD is the common default) so incompatible changes are rejected at registration, not discovered at read time. Add fields with defaults to stay compatible; a rename or removal is a MAJOR that must move to a new version, never an in-place edit.

Worked example — run v1 and v2 in parallel with a deprecation clock

Detailed explanation. When a change is genuinely breaking, you cannot force every consumer to migrate at once. You ship the new major alongside the old, mark the old deprecated with a sunset date, and use read telemetry to know when it is safe to retire. Run orders v1 and v2 in parallel through a rename.

• The break. total_cents → gross_cents (a MAJOR).
• The parallel run. orders_v1 and orders_v2 both live.
• The clock. v1 deprecated with a sunset date; telemetry tracks v1 readers.

Question. Ship a breaking rename as v2 while keeping v1 working, and decide when it is safe to sunset v1.

Input.

Element	v1	v2
Field name	total_cents	gross_cents
Status	deprecated	current
Sunset	2026-11-30	—
Readers	tracked (telemetry)	growing

Code.

-- v2 is the NEW major with the renamed field. v1 stays live, unchanged.
CREATE VIEW product.orders_v2 AS
SELECT order_id, tenant_id, order_status,
       total_cents AS gross_cents,      -- the breaking rename lives ONLY in v2
       order_ts
FROM raw.orders WHERE order_status IS NOT NULL;

-- v1 is preserved AS-IS so existing consumers keep working during the window.
-- (product.orders_v1 continues to expose total_cents.)

# Descriptor: BOTH majors are live; v1 carries a deprecation clock.
output_ports:
  - { name: sql_v2, type: table, location: product.orders_v2, version: 2.0.0, status: current }
  - { name: sql_v1, type: table, location: product.orders_v1, version: 1.4.0,
      status: deprecated, sunset: 2026-11-30,
      migrate_to: sql_v2, note: "total_cents renamed to gross_cents" }

-- Migration TELEMETRY: who still reads v1? Retire on evidence, not the calendar.
SELECT usename AS consumer, count(*) AS reads_7d, max(query_start) AS last_seen
FROM   pg_stat_activity_history           -- or query logs / audit table
WHERE  query ILIKE '%product.orders_v1%'
  AND  query_start > now() - interval '7 days'
GROUP  BY usename
ORDER  BY reads_7d DESC;
-- v1 is safe to sunset when this returns no rows (or only known stragglers).

Step-by-step explanation.

1. product.orders_v2 is a new view carrying the renamed gross_cents; the breaking change is isolated to v2. product.orders_v1 is left untouched, so every existing consumer keeps reading total_cents exactly as before — the parallel run is what makes a MAJOR non-disruptive.
2. The descriptor lists both ports with their versions, and marks v1 deprecated with a concrete sunset date and a migrate_to pointer — so a consumer looking at the catalog sees the clock and knows both when v1 goes away and what to move to.
3. The note documents the actual breaking difference (the rename), turning the migration into a mechanical, well-documented change for consumers rather than a reverse-engineering exercise.
4. The telemetry query answers the only question that makes a sunset safe: who still reads v1? Retiring on a calendar date alone risks breaking a straggler; retiring when reads reach zero (or only known, notified consumers remain) retires on evidence.
5. The discipline that avoids "forever-two-versions" is the combination of the clock (a deadline that creates urgency) and the telemetry (proof the deadline was met) — you keep v1 exactly as long as someone needs it and not one release longer.

Output.

Phase	v1	v2	Action
ship v2	live	live	consumers start migrating
deprecation window	deprecated	current	telemetry tracks v1 reads
reads → 0	retire	current	sunset v1
after sunset	gone	only version	single version again

Rule of thumb. Ship a breaking change as a new major alongside the old one, mark the old deprecated with a sunset date and a migrate_to pointer in the catalog, and drive the sunset with read telemetry — retire v1 only when reads reach zero. Parallel versions plus a clock plus evidence is how you break a contract without breaking a consumer.

Senior interview question on versioning a data product safely

A senior interviewer might ask: "You need to rename a field and change a unit in your orders data product — a genuinely breaking change — but a dozen teams depend on it and you don't control their release schedules. Walk me through it end to end: how you classify and version the change, how you keep a break from silently shipping, how you run old and new in parallel, and how you decide when it is finally safe to retire the old version."

Solution Using semantic versioning, registry-enforced compatibility, parallel versions, and read telemetry

# 1. CLASSIFY: rename total_cents -> gross_cents AND change unit is BREAKING.
#    An unchanged consumer breaks -> MAJOR. Version 1.4.0 -> 2.0.0.

# 2. ENFORCE: the registry makes a silent in-place break impossible.
curl -X PUT $REG/config/orders-value -d '{"compatibility": "BACKWARD"}'
# attempting to register the rename on the SAME subject -> 409 Conflict.
# a MAJOR must be a NEW subject/version -> orders-v2. The break cannot slip through.

-- 3. PARALLEL VERSIONS: v2 carries the break; v1 stays live and unchanged.
CREATE VIEW product.orders_v2 AS
SELECT order_id, tenant_id, order_status,
       total_cents AS gross_cents, order_ts
FROM raw.orders WHERE order_status IS NOT NULL;
-- product.orders_v1 continues to expose total_cents to existing consumers.

# 4. DEPRECATION CLOCK in the descriptor/catalog.
output_ports:
  - { name: sql_v2, version: 2.0.0, status: current }
  - { name: sql_v1, version: 1.4.0, status: deprecated, sunset: 2026-11-30, migrate_to: sql_v2 }

-- 5. TELEMETRY decides the sunset: retire v1 only when reads reach zero.
SELECT usename, count(*) FROM query_log
WHERE query ILIKE '%orders_v1%' AND ts > now() - interval '7 days'
GROUP BY usename;   -- empty (or only notified stragglers) => safe to sunset

Step-by-step trace.

Step	Mechanism	Guarantee
Classify	semver by consumer impact	rename+unit = MAJOR (2.0.0)
Enforce	registry compatibility mode	no silent in-place break
Parallel	v1 + v2 live	existing consumers unaffected
Deprecate	sunset date + migrate_to	consumers have a known window
Retire	read telemetry = 0	sunset on evidence, not calendar

After the rollout, the change is classified MAJOR and versioned 2.0.0; the registry's BACKWARD mode rejects any attempt to slip the rename into the existing subject, forcing it onto orders-v2; product.orders_v2 exposes gross_cents while product.orders_v1 keeps serving total_cents untouched; the catalog shows v1 deprecated with a November sunset and a pointer to v2; and the telemetry query proves when v1's reads reach zero so the sunset breaks nobody. A dozen teams migrate on their own schedules inside the window — a planned migration, not an incident.

Output:

Metric	In-place breaking ALTER	Versioned + parallel + telemetry
Consumers broken at change time	all twelve	zero
How a break is caught	in production, by users	at registration (409)
Migration control	producer forces it	consumers choose within the window
Retirement safety	hope	proven by read telemetry
Long-term maintenance	—	bounded by the clock

Why this works — concept by concept:

• Semantic versioning by consumer impact — classifying the rename-plus-unit change as MAJOR and versioning it 2.0.0 prices the change honestly, so nobody mistakes a break for a free additive change.
• Registry-enforced compatibility — a compatibility mode checked at registration rejects an incompatible in-place edit with a 409, converting "a reviewer should have caught it" into "the system refuses it," so a silent break is impossible.
• Parallel versions — shipping v2 beside an unchanged v1 lets a dozen teams migrate on their own release cadence, turning a forced, synchronised break into a set of independent, planned migrations.
• Deprecation clock plus read telemetry — a sunset date creates urgency and read telemetry proves the window was honoured, so v1 is retired exactly when it is safe — neither prematurely (breaking a straggler) nor forever (doubling maintenance).
• Cost — a second view, a registry check, a descriptor entry, and a telemetry query, versus an outage across twelve teams and a scramble to roll back. The eliminated cost is the O(consumers) simultaneous break an in-place alter causes — parallel versioning makes the cost O(1) per team, paid on each team's own schedule.

Data validation
Topic — data-validation
Data validation problems on schema compatibility and versioning

Practice →

ETL Topic — etl ETL problems on schema evolution and parallel-version migration

Practice →

---

4. SLAs and SLOs — freshness, availability, quality

Freshness, availability, and quality as measured SLOs — with monitors that page an owner

The mental model in one line: a data product earns trust only when its promises are measurable: an SLA is the promise you make to consumers, an SLO is the numeric target you hold yourself to, an SLI is the actual measurement, and the gap between the SLO and reality is your error budget — and for data the three dimensions that matter are freshness (is the data recent enough?), availability/completeness (is it there and whole?), and quality/correctness (does it satisfy the contract?), each expressed as an SLI a monitor computes and each wired to an alert that pages a named owner — because "it usually updates every morning" is not an SLA, it is a hope, and a product nobody can hold to a number is not trustworthy. Measure the promise, budget the misses, and page the owner — otherwise the SLA is decoration.

SLI, SLO, SLA, error budget.

• SLI (indicator). The measurement: "minutes since the latest order_ts," "percent of expected rows present," "percent of rows passing contract tests." An SLI is a number you can compute on a schedule.
• SLO (objective). The target you commit to internally: freshness ≤ 15 min, availability ≥ 99.9%, quality ≥ 99%. The SLO is stricter than the SLA so you have headroom.
• SLA (agreement). The promise to consumers, often with consequences: "data is at most 30 minutes stale." Consumers design against the SLA.
• Error budget. 100% − SLO — the allowed misses. Burning the budget fast is the signal to stop shipping features and fix reliability; an intact budget means you can take risks.

The three data SLO dimensions.

• Freshness / timeliness. How recently was the data updated relative to the event time it represents? Measured as lag between now (or the latest event) and the last successful load. The most-consumed data SLO.
• Availability / completeness. Is the product reachable and is the expected volume present? A view that returns but is missing half its rows is "available" yet incomplete — measure row counts against an expected range, not just uptime.
• Quality / correctness. Does the data satisfy the contract — not-null, unique, in-range, referentially valid? Measured as the pass rate of the contract's quality tests, the same rules from section 2.
• The trap. An SLA that names a dimension with no SLI behind it is unenforceable. Every promised dimension needs a computed SLI and a monitor.

Monitoring and escalation.

• Freshness checks. A scheduled query on max event/load time vs threshold — the canonical dbt source freshness or an explicit SQL monitor.
• Volume / anomaly checks. Row-count and distribution checks that catch a silent half-load or a schema drift the freshness check would miss.
• Quality tests. The contract's not-null/unique/range assertions run as tests, producing a pass rate to compare against the quality SLO.
• Ownership + escalation. Every SLO breach pages a named owner (the descriptor's on_call), with severity tied to the dimension — a stale gold product is a page, a stale sandbox one is a ticket.

The failure modes senior engineers pre-empt.

• SLA with no SLI. A promised freshness with nothing measuring it is fiction. Mitigation: no SLA dimension without a computed SLI and a monitor.
• Alert on everything. Paging on every minor blip trains people to ignore pages. Mitigation: alert on SLO breach and error-budget burn rate, not on every anomaly; tune severity.
• No owner / no escalation. A monitor that fires into an unwatched channel is not a monitor. Mitigation: page the descriptor's named on-call, with escalation.

Worked example — define SLIs and SLOs for a data product

Detailed explanation. The first step is turning vague promises into a table of SLI → SLO → alert. Every dimension gets a measurable indicator, a numeric target, and an action on breach. Build the SLO table for the orders product.

• Three dimensions. Freshness, availability/completeness, quality.
• Each row. An SLI you can compute, an SLO number, an alert.
• The SLA. A slightly looser consumer-facing promise derived from the SLOs.

Question. Define SLIs, SLOs, and alerts for the orders product across freshness, availability, and quality.

Input.

Dimension	SLI (how measured)	SLO (target)	Alert on
Freshness	minutes since last load	≤ 15 min	> 15 min for 2 checks
Availability	% expected rows present	≥ 99.9%	< 99.9%
Quality	% rows passing contract tests	≥ 99%	< 99%

Code.

# slo.yaml — measurable promises for the orders product (asserted by monitors).
data_product: orders
owner: { team: analytics-platform, on_call: "#oncall-analytics" }
sla: "Orders is at most 30 minutes stale, complete, and contract-valid."  # consumer-facing
slos:
  - dimension: freshness
    sli: "minutes_since_last_load"
    objective: "<= 15m"              # SLO stricter than the 30m SLA (headroom)
    error_budget: "1% of the month"  # allowed staleness
    alert: { condition: "> 15m for 2 consecutive checks", page: on_call }
  - dimension: availability
    sli: "rows_present / rows_expected"
    objective: ">= 99.9%"
    alert: { condition: "< 99.9%", page: on_call }
  - dimension: quality
    sli: "rows_passing_contract_tests / total_rows"
    objective: ">= 99%"
    alert: { condition: "< 99%", page: on_call }

Step-by-step explanation.

1. Each SLO names a concrete sli — a formula you can compute on a schedule (minutes_since_last_load, a ratio of present-to-expected rows, a test pass rate). Without a computable SLI, the objective is unmeasurable and the SLA is fiction.
2. The objective (SLO) is deliberately stricter than the consumer-facing sla: the SLA promises "≤ 30 minutes stale" but the SLO targets "≤ 15m," leaving headroom so a brief breach of the internal target does not immediately break the external promise.
3. The freshness SLO carries an explicit error_budget — the allowed fraction of time the product may be stale. Budget framing turns reliability into a resource: burn it slowly and you can ship; burn it fast and you stop and fix.
4. Availability is measured as completeness (rows_present / rows_expected), not mere uptime — this is the data-specific twist: a product can be queryable yet missing half its rows, and only a volume-based SLI catches that.
5. Every SLO's alert routes to the descriptor's on_call, so a breach pages a named owner. The freshness alert requires "2 consecutive checks" to avoid paging on a single transient blip — severity and debounce are tuned per dimension.

Output.

Dimension	Measured by	Target	On breach
Freshness	minutes_since_last_load	≤ 15m	page (2 checks)
Availability	present/expected rows	≥ 99.9%	page
Quality	contract-test pass rate	≥ 99%	page
SLA (consumer)	derived from SLOs	≤ 30m, complete, valid	—

Rule of thumb. Turn every promised dimension into a row of SLI → SLO → alert: a computable indicator, a numeric target stricter than the consumer SLA, and a page to a named owner on breach. An SLA dimension with no SLI behind it is decoration — measure freshness, completeness, and quality, or do not promise them.

Worked example — a freshness SLO monitor

Detailed explanation. Freshness is the most-consumed data SLO and the easiest to make concrete: a scheduled query comparing the latest data timestamp to now, alerting when the lag exceeds the threshold. Build a freshness monitor for orders.

• The SLI. now() − max(order_ts) (or last successful load time).
• The threshold. 15 minutes (the SLO).
• The action. Page the owner when lag > threshold for 2 checks.

Question. Write a freshness monitor that computes the lag SLI and pages the owner when the freshness SLO is breached.

Input.

Element	Value
SLI	minutes since max(order_ts)
SLO threshold	15 minutes
Debounce	2 consecutive breaches
Action	page #oncall-analytics

Code.

-- The freshness SLI as a query. Runs every 5 minutes on a schedule.
SELECT
    max(order_ts)                                   AS last_event,
    extract(epoch FROM now() - max(order_ts)) / 60  AS lag_minutes,
    (extract(epoch FROM now() - max(order_ts)) / 60) > 15 AS breaches_slo   -- SLO = 15m
FROM product.orders_v1;

# dbt source-freshness equivalent — declarative freshness SLO on the source.
sources:
  - name: product
    tables:
      - name: orders_v1
        loaded_at_field: order_ts
        freshness:
          warn_after:  { count: 15, period: minute }   # SLO target
          error_after: { count: 30, period: minute }   # SLA breach -> page

# The monitor: compute the SLI, debounce, and PAGE the named owner on breach.
lag = run_sql("SELECT extract(epoch FROM now()-max(order_ts))/60 FROM product.orders_v1")
STATE["breaches"] = STATE["breaches"] + 1 if lag > 15 else 0
if STATE["breaches"] >= 2:                       # debounce: 2 consecutive checks
    page(channel="#oncall-analytics",
         msg=f"orders freshness SLO BREACH: {lag:.0f}m > 15m (SLA 30m at risk)")

Step-by-step explanation.

1. The SQL computes the freshness SLI directly: now() − max(order_ts) in minutes, plus a boolean breaches_slo. Because the SLI is just a query, it runs anywhere a scheduler can run SQL — no special tooling required to measure.
2. The dbt source freshness block expresses the same SLO declaratively: warn_after: 15m matches the internal SLO and error_after: 30m matches the consumer SLA, so a warn is "watch it" and an error is "the promise is at risk — page." This ties the monitor to the numbers in slo.yaml.
3. The Python monitor is where measurement becomes action: it computes the lag, and — crucially — debounces by requiring two consecutive breaches before paging, so a single transient blip (a slow load that recovers next cycle) does not wake anyone.
4. The page targets the descriptor's named on_call channel with a message that states the SLI value, the SLO it breached, and the SLA it threatens — an actionable page, not "something is wrong."
5. The whole loop closes the SLA: the freshness promise in the descriptor, the target in the SLO, the measurement in the query, and the page to the owner are one connected chain — which is exactly what makes freshness an enforceable guarantee instead of a hope.

Output.

Lag (SLI)	vs SLO (15m)	Monitor action
6 min	within	none
18 min (1 check)	breach	debounced (wait)
18 min (2 checks)	breach	page owner
32 min	SLA at risk	page + escalate

Rule of thumb. Make freshness a scheduled SLI — minutes since the latest event or load — compared to the SLO threshold, debounced to avoid transient noise, and paged to the named owner on sustained breach. A declarative source freshness plus a debounced pager turns "it should be fresh" into a measured, escalatable guarantee.

Worked example — a quality SLO with a validation test and pass rate

Detailed explanation. Freshness says the data is recent; quality says it is correct. The quality SLI is the pass rate of the contract's rules — the same not-null/unique/range assertions from section 2, now measured as a percentage against the quality SLO. Build the quality monitor for orders.

• The SLI. rows_passing_contract_tests / total_rows.
• The SLO. ≥ 99% pass rate.
• The rules. The contract's not-null, unique, range, enum assertions.

Question. Compute a quality SLI as the contract-test pass rate and alert when it falls below the quality SLO.

Input.

Contract rule	Failing rows counted	Weight
not-null (5 cols)	rows with any null	high
unique (order_id)	duplicate rows	high
range (total_cents ≥ 0)	negative rows	high
enum (order_status)	out-of-set rows	high

Code.

-- Quality SLI: fraction of rows that satisfy EVERY contract rule.
WITH checked AS (
  SELECT
    -- a row PASSES only if it satisfies all contract assertions
    (order_id IS NOT NULL AND tenant_id IS NOT NULL
       AND order_status IS NOT NULL AND total_cents IS NOT NULL AND order_ts IS NOT NULL
     AND total_cents >= 0
     AND order_status IN ('pending','paid','refunded')) AS row_ok
  FROM product.orders_v1
),
dupes AS (   -- uniqueness handled separately (cross-row rule)
  SELECT count(*) AS dup_rows FROM (
    SELECT order_id FROM product.orders_v1 GROUP BY order_id HAVING count(*) > 1
  ) d
)
SELECT
  round(100.0 * sum(CASE WHEN row_ok THEN 1 ELSE 0 END) / count(*), 3) AS quality_pct,
  (SELECT dup_rows FROM dupes)                                          AS duplicate_orders,
  round(100.0 * sum(CASE WHEN row_ok THEN 1 ELSE 0 END) / count(*), 3) < 99.0 AS breaches_slo
FROM checked;

# Monitor: compare the quality SLI to the SLO and page on breach.
q = run_sql(QUALITY_SLI_SQL)             # -> {quality_pct, duplicate_orders, breaches_slo}
if q["breaches_slo"] or q["duplicate_orders"] > 0:
    page("#oncall-analytics",
         f"orders quality SLO BREACH: {q['quality_pct']}% < 99% "
         f"({q['duplicate_orders']} duplicate order_ids)")

Step-by-step explanation.

1. The checked CTE evaluates every single-row contract rule per row into one boolean row_ok — a row passes only if it is fully not-null, non-negative, and in the status enum. This is the per-row half of the contract from section 2, executed as a measurement.
2. Uniqueness is a cross-row rule, so it cannot be a per-row boolean; the dupes CTE counts duplicate order_ids separately. Recognising that some contract rules are per-row and some are set-level is the subtle part of building a quality SLI.
3. quality_pct is the SLI: the percentage of rows satisfying all per-row rules, rounded. It is directly comparable to the 99% quality SLO, so the number the monitor computes is the number the SLO promises.
4. breaches_slo flags when the pass rate drops below 99%, and the monitor also treats any duplicate as a breach — a uniqueness violation is a contract break regardless of the aggregate pass rate, so it is escalated on its own.
5. The page reports the actual pass rate and the duplicate count, so the owner knows which rule degraded and by how much — the same rules that back the port's enforcement gate (section 2) now double as the quality SLO's SLI, so "the product guarantees quality" and "the monitor measures quality" are one definition.

Output.

Quality SLI	Duplicates	vs SLO (99%)	Action
99.8%	0	within	none
98.5%	0	breach	page owner
99.9%	3	uniqueness break	page owner
92%	40	severe breach	page + escalate

Rule of thumb. Make quality a measured SLI — the pass rate of the contract's rules — and alert when it drops below the quality SLO, treating per-row rules (not-null, range, enum) and set-level rules (uniqueness) distinctly. When the quality monitor runs the same assertions as the port's enforcement gate, the product's guarantee and its measurement are the same list.

Senior interview question on data SLAs and monitoring

A senior interviewer might ask: "Your orders data product's SLA says 'fresh and reliable,' but a downstream dashboard was silently wrong for a full day and nobody noticed until a VP asked. Redesign the reliability story from scratch: what SLIs and SLOs you'd define across freshness, availability, and quality, how you'd measure each, how you'd alert without drowning the owner in noise, and how an error budget changes what the team does when reliability slips."

Solution Using SLIs and SLOs across three dimensions, monitors, error budgets, and owner paging

# 1. SLOs across three dimensions — each with a measurable SLI and an owner.
data_product: orders
owner: { team: analytics-platform, on_call: "#oncall-analytics" }
sla: "Orders is at most 30m stale, >=99.9% complete, and contract-valid."
slos:
  - { dimension: freshness,    sli: minutes_since_last_load,          objective: "<= 15m",  error_budget: "1%/month" }
  - { dimension: availability, sli: rows_present/rows_expected,       objective: ">= 99.9%" }
  - { dimension: quality,      sli: rows_passing_contract_tests/total, objective: ">= 99%" }

-- 2. The three SLIs, computed on a schedule (freshness / completeness / quality).
SELECT extract(epoch FROM now()-max(order_ts))/60                    AS freshness_min,
       (SELECT count(*) FROM product.orders_v1)::float
         / nullif((SELECT expected FROM ops.orders_expected_today),0) AS availability,
       avg(CASE WHEN total_cents >= 0
                 AND order_status IN ('pending','paid','refunded')
                THEN 1.0 ELSE 0 END)                                  AS quality
FROM product.orders_v1;

# 3. Alert on SLO BREACH and error-budget BURN RATE — not on every blip.
if freshness_min > 15 and sustained(2):           page(on_call, "freshness SLO breach")
if availability < 0.999:                          page(on_call, "completeness SLO breach")
if quality < 0.99:                                page(on_call, "quality SLO breach")
if budget_burn_rate() > 14:   # burning a month's budget in ~2 days
    freeze_feature_work()      # error budget policy: stop shipping, fix reliability

Step-by-step trace.

Dimension	SLI	SLO	Why the day-long bug is now caught
Freshness	minutes since load	≤ 15m	a stalled load pages within ~10 min
Availability	present/expected rows	≥ 99.9%	a half-load trips completeness
Quality	contract pass rate	≥ 99%	bad values trip the quality SLI
Error budget	1%/month, burn rate	—	fast burn freezes feature work

After the redesign, the "silently wrong for a day" failure is impossible to miss: whichever dimension degraded — a stalled load (freshness), a partial load (availability), or corrupt values (quality) — has an SLI computed every few minutes and an alert that pages the named owner on sustained breach. Alerts fire on SLO breach and budget burn rate, not on every anomaly, so the owner is paged for things that matter and not trained to ignore noise. The error-budget policy makes reliability a team decision: a fast burn freezes feature work until the product is healthy again.

Output:

Metric	"Fresh and reliable" (vibes)	Three SLOs + budget + paging
Silent day-long error	undetected	paged in minutes
What's measured	nothing	freshness, completeness, quality
Alert noise	none or all	tuned to SLO breach + burn rate
Owner on breach	unclear	named on-call, paged
Reliability vs features	ad hoc	governed by error budget

Why this works — concept by concept:

• Three measured dimensions — freshness, availability/completeness, and quality each become a computed SLI, so the three distinct ways data "goes wrong" (late, partial, corrupt) are all caught, not just uptime.
• SLO stricter than SLA — targeting tighter internal SLOs than the consumer-facing SLA leaves headroom, so a brief internal breach is caught and fixed before the external promise is broken.
• Breach-and-burn-rate alerting — paging on SLO breach and error-budget burn rate, not on every blip, keeps alerts actionable and stops the owner from being trained to ignore the pager.
• Error budget as policy — quantifying allowed misses turns reliability into a resource the team spends deliberately: intact budget means ship, fast burn means freeze features and fix — a decision rule, not a debate.
• Cost — three scheduled SLI queries and a paging rule, versus a VP discovering a day-old bug. The eliminated cost is the silent-failure blast radius — O(scheduled checks) monitoring replaces O(consumers × time-to-notice) undetected damage.

Data validation
Topic — data-validation
Data validation problems on freshness, completeness, and quality SLIs

Practice →

Real-time analytics Topic — real-time-analytics Real-time analytics problems on freshness monitoring and alerting

Practice →

---

5. Discoverability — catalogs, metadata, ownership, self-serve

If nobody can find it, own it, or trust it, it isn't a product — catalogs, metadata, self-serve

The mental model in one line: discoverability is the property that lets a stranger find, understand, trust, and access a data product without talking to its author — delivered by publishing rich metadata to a data catalog (DataHub, OpenMetadata, Unity Catalog, Amundsen): the schema and semantics (so it is understandable), the owner and on-call (so it is accountable), the lineage (so its provenance is visible), the SLOs and quality (so it is trustworthy), the tags and classification (so it is governed and searchable), and a self-serve access path (so it is usable) — and the discipline that keeps the catalog from rotting is publishing this metadata as code from the pipeline, so the catalog entry is generated from the same descriptor and contract that define the product, never hand-maintained. A product a stranger cannot find, understand, or trust is not a product — it is a private table with extra steps.

The catalog metadata a product must publish.

• Identity and ownership. A stable id, a human name, a description, and a named owner team with an on-call — the first question a discoverer asks is "who owns this and who do I ping?"
• Schema and semantics. The typed schema plus the glossary/business terms (what "order," "revenue," "tenant" mean) — so the data is understandable without the author.
• Lineage. Upstream sources and downstream consumers, ideally column-level — so a discoverer sees where the data came from and who depends on it (and so an owner sees blast radius before a change).
• Trust signals. The published SLOs, recent freshness/quality status, version, and deprecation state — so a discoverer can judge whether to build on it now.

Addressable, self-describing, self-serve.

• Addressable. A product has a stable, unique address (a catalog URN, a fully-qualified name) that never changes even as storage moves — so links and references stay valid.
• Self-describing. The metadata answers "what is this and how do I use it?" — schema, semantics, examples, and the output ports — without a human in the loop.
• Self-serve access. A discoverer requests access through a governed workflow (a policy grant, a request that routes to the owner), not a DM — so onboarding is a click, not a favour.
• Interoperable and secure. Standard identifiers and classifications (PII tags, sensitivity) let products compose and let access be governed by policy, not by tribal trust.

Publish metadata as code.

• One source, generated entry. The catalog entry is emitted from the product descriptor and contract in the pipeline — so schema, owner, SLO, and lineage in the catalog always match the ones in the repo.
• Emit on every run. The pipeline pushes metadata (via a DataHub/OpenMetadata emitter or ingestion) on each build, so freshness/quality status and schema changes are always current.
• Lineage from the pipeline. Emit lineage from the transformation graph (dbt, Spark, an orchestrator) rather than drawing it by hand — machine-captured lineage is the only lineage that stays true.
• No hand-maintained catalog. A wiki page someone updates manually is stale by definition; metadata-as-code is what keeps discoverability honest.

The failure modes senior engineers pre-empt.

• Tribal knowledge. The product exists only in the heads of its authors. Mitigation: emit a catalog entry with schema, owner, and semantics as code — findable by search.
• Stale catalog. A hand-maintained entry drifts from reality. Mitigation: generate the entry from the descriptor/contract on every pipeline run.
• No owner / unclassified PII. An unowned product with unclassified sensitive columns is a governance incident waiting to happen. Mitigation: mandatory owner and PII classification in the descriptor, enforced in CI.

Worked example — a catalog metadata document for a data product

Detailed explanation. Discoverability starts with the metadata document the catalog renders: identity, ownership, schema, semantics, lineage, trust signals, and access. Author the catalog metadata for the orders product.

• Identity + owner. Stable urn, name, description, on-call team.
• Schema + glossary. Typed fields plus business terms.
• Trust + access. SLOs, tags/classification, self-serve access path.

Question. Author the catalog metadata for the orders product so a stranger can find, understand, trust, and request access to it.

Input.

Metadata block	Purpose	Discoverer's question
identity + owner	accountability	"who owns this?"
schema + glossary	understandability	"what do the fields mean?"
lineage	provenance	"where did it come from?"
trust + access	usability	"can I trust it / get in?"

Code.

# catalog/orders.yaml — the metadata the catalog renders for discoverers.
urn: "urn:dataproduct:commerce:orders"     # stable ADDRESS, never changes
name: "Orders"
description: "Curated orders, one row per order, monetary values in integer cents (UTC)."
owner:
  team: analytics-platform
  on_call: "#oncall-analytics"             # who to page
domain: commerce
tags: [gold, revenue, pii-safe]           , searchable + governance
glossary:                                  # business SEMANTICS
  order: "A confirmed customer purchase."
  revenue: "sum(total_cents) where order_status='paid'."
  tenant: "The customer account the order belongs to."
schema:                                    # typed, understandable
  - { name: order_id, type: string, description: "immutable order id" }
  - { name: total_cents, type: long, description: "gross value, integer cents" }
  - { name: order_status, type: string, description: "pending|paid|refunded" }
lineage:                                    # provenance
  upstream:   [raw.orders, raw.payments]
  downstream: [dashboards.revenue, ml.churn_features]
trust:                                      # trust signals from the SLOs
  version: 1.2.0
  slos: { freshness: "<= 15m", availability: "99.9%", quality: ">= 99%" }
  status: healthy
access:                                     # SELF-SERVE
  request: "https://catalog.example.com/request/urn:dataproduct:commerce:orders"
  policy: "tenant-scoped; approval by owner"

Step-by-step explanation.

1. The urn is the product's stable address — a discoverer, a link, or another product references this identifier, and it stays valid even if the underlying storage moves from Postgres to Iceberg. Addressability is what makes references durable.
2. owner with an on_call answers the first discovery question — accountability — directly in the catalog, so nobody has to run git blame or ask around to find who is responsible.
3. The glossary block carries semantics the schema alone cannot: it defines "revenue" as sum(total_cents) where paid, so a discoverer computes the metric the same way the owner intends — killing the "everyone defines revenue differently" problem at the source.
4. lineage shows both upstream provenance and downstream consumers, so a discoverer sees where the data comes from and an owner sees blast radius before changing it — the same metadata serves discovery and change-safety.
5. The trust block surfaces the version, SLOs, and current health so a discoverer can judge fitness now, and the access block gives a self-serve request URL with the policy — turning onboarding from a DM into a governed click. Every block maps to a discovery question, which is why the document is the product's public face.

Output.

Discoverer asks	Catalog field	Answer
"Who owns it?"	owner.on_call	#oncall-analytics
"What is revenue?"	glossary.revenue	sum of paid cents
"Where's it from?"	lineage.upstream	raw.orders, raw.payments
"Can I trust it?"	trust.slos/status	fresh≤15m, healthy
"How do I get access?"	access.request	self-serve URL

Rule of thumb. Give every product a catalog metadata document with a stable urn, a named owner, a business glossary, the typed schema, upstream/downstream lineage, trust signals (version + SLOs + health), and a self-serve access path. Each block answers a discovery question so a stranger can find, understand, trust, and access the product without ever contacting its author.

Worked example — publish metadata as code

Detailed explanation. A hand-maintained catalog entry rots. The discipline is emitting metadata from the pipeline so the catalog is generated from the same descriptor and contract that define the product. Emit the orders metadata to a catalog on every run.

• The source. The descriptor + contract in the repo.
• The emit. A pipeline step pushes metadata (and lineage) to the catalog.
• The guarantee. Catalog always matches the repo — never stale.

Question. Emit the orders product metadata and lineage to the data catalog from the pipeline, so the catalog entry is generated, not hand-maintained.

Input.

Step	Source	Emitted to catalog
read descriptor	data-product.yaml	identity, owner, SLOs
read contract	contracts/orders.yaml	schema, semantics
read pipeline graph	dbt/Spark lineage	upstream/downstream
emit	catalog API/emitter	one generated entry

Code.

# emit_metadata.py — run at the END of the pipeline. Catalog is GENERATED, not typed.
import yaml
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import (
    DatasetPropertiesClass, OwnershipClass, OwnerClass, GlobalTagsClass, TagAssociationClass,
)

descriptor = yaml.safe_load(open("data-product.yaml"))
contract   = yaml.safe_load(open("contracts/orders.yaml"))
emitter    = DatahubRestEmitter("http://datahub:8080")

urn = f"urn:li:dataset:(urn:li:dataPlatform:warehouse,product.orders_v1,PROD)"

# 1. Properties (description + SLOs) straight from the descriptor — no hand editing.
emitter.emit_mcp(make_mcp(urn, DatasetPropertiesClass(
    description=descriptor["spec"]["description"],
    customProperties={
        "version":   str(descriptor["metadata"]["version"]),
        "freshness": descriptor["spec"]["slos"]["freshness"],
        "quality":   descriptor["spec"]["slos"]["quality"],
    },
)))

# 2. Ownership from the descriptor — the owner is never guessed.
emitter.emit_mcp(make_mcp(urn, OwnershipClass(owners=[
    OwnerClass(owner=f"urn:li:corpGroup:{descriptor['metadata']['owner']['team']}", type="DATAOWNER")
])))

# 3. Tags / classification (governance + search) from the descriptor.
emitter.emit_mcp(make_mcp(urn, GlobalTagsClass(
    tags=[TagAssociationClass(tag=f"urn:li:tag:{t}") for t in descriptor["metadata"]["tags"]]
)))

# 4. Lineage is captured from the transformation graph, not drawn by hand.
emit_lineage(urn, upstream=descriptor["spec"]["catalog"]["lineage_upstream"])
print("catalog entry emitted from descriptor + contract — guaranteed in sync")

Step-by-step explanation.

1. The emitter reads the same data-product.yaml and contracts/orders.yaml that define the product, so the catalog's description, version, SLOs, owner, and tags are derived from the source of truth — there is no second place to update and no opportunity to drift.
2. Properties and custom fields (version, freshness, quality) come straight from the descriptor's slos block, so the trust signals a discoverer sees in the catalog are exactly the promises the product actually makes — the SLA and the catalog cannot disagree.
3. Ownership is emitted from descriptor.metadata.owner, so the "who owns this?" answer is generated from the repo — an unowned product literally cannot be published, because the emit would have no owner to write.
4. Tags/classification (gold, pii-safe) are emitted for both search and governance; a policy engine can later refuse to publish if a required classification (like a PII tag) is missing, making "unclassified sensitive data" a CI failure rather than a leak.
5. Lineage is captured from the transformation graph, not drawn by hand — the only kind of lineage that stays true as pipelines change. Because the whole emit runs at the end of every pipeline run, the catalog is regenerated each build, which is what structurally prevents the stale-catalog failure mode.

Output.

Metadata	Source	Freshness
description, version, SLOs	descriptor	regenerated each run
owner	descriptor	always current
tags / PII class	descriptor	enforced in CI
lineage	pipeline graph	machine-captured

Rule of thumb. Emit catalog metadata as code from the pipeline — generated from the same descriptor and contract that define the product — on every run, and capture lineage from the transformation graph rather than by hand. A catalog entry that is regenerated each build cannot go stale, and a product with no owner or missing PII tag simply fails to publish.

Worked example — ownership and self-serve access

Detailed explanation. Discoverability ends at access: a stranger who found and trusts the product must be able to get in without a favour. The pattern is a governed self-serve request that routes to the owner and grants scoped access by policy. Wire self-serve access for orders.

• The owner. The named team approves (or auto-approval by policy).
• The request. A governed workflow, not a DM.
• The grant. Scoped by the contract's access policy (tenant, columns).

Question. Design a self-serve access flow so a discoverer can request and receive scoped access to orders through the owner, governed by policy.

Input.

Element	Value
Trigger	"Request access" in the catalog
Approver	owner team (analytics-platform)
Policy	tenant-scoped; PII-safe columns only
Grant	role/row-policy applied automatically

Code.

# access-policy.yaml — governed SELF-SERVE access, not a DM to the owner.
resource: "urn:dataproduct:commerce:orders"
request_flow:
  entrypoint: catalog                     # discoverer clicks "Request access"
  approver: analytics-platform            # routes to the OWNER team
  auto_approve_if:                        # policy can grant without a human
    - requester.domain == "commerce"
    - purpose in ["analytics", "reporting"]
grant:
  role: orders_reader                     # least-privilege role
  row_policy: "tenant_id = requester.tenant_id"   # scoped by the CONTRACT's access block
  columns: [order_id, tenant_id, order_status, total_cents, order_ts]  # PII-safe set
  ttl: 90d                                # access expires -> re-review
audit:
  log: true                               # every grant is recorded for governance

-- On approval, the grant is applied AUTOMATICALLY — access is a click, not a favour.
GRANT orders_reader TO "user:dana@commerce";           -- least-privilege role
-- the row policy from the contract is already enforced on the product view:
--   USING (tenant_id = current_setting('request.jwt.claims')::json->>'tenant_id')
-- so the new reader sees ONLY their tenant's rows, PII-safe columns only.

Step-by-step explanation.

1. The request entrypoint is the catalog itself — the discoverer who just found and evaluated the product clicks "Request access" in the same place, so discovery and access are one continuous flow, not a hand-off to email.
2. The request routes to the owner team, making the descriptor's ownership load-bearing again: the same named team that is paged on an SLO breach is the team that approves access — one accountable owner for both reliability and governance.
3. auto_approve_if encodes policy that can grant without a human when the requester and purpose are low-risk — so self-serve is genuinely self-serve for the common case, while sensitive requests still route to a person.
4. The grant is least-privilege and scoped by the contract's access policy: a role, a row policy tying reads to the requester's tenant, and a PII-safe column set — so a new reader physically cannot see other tenants or sensitive fields, and the access policy lives in one place (the contract) rather than being re-implemented per grant.
5. The ttl and audit close the governance loop: access expires and must be re-reviewed (no forgotten grants), and every grant is logged — so self-serve does not mean ungoverned. Access becomes a click for the requester and an audit trail for the owner, which is what "self-serve and secure" means.

Output.

Requester	Policy check	Result
commerce analyst	auto-approve match	granted (tenant-scoped)
external partner	no match	routed to owner
granted user, day 91	ttl expired	re-review required
any grant	audit	logged for governance

Rule of thumb. Make access self-serve through a governed request that starts in the catalog, routes to the named owner, auto-approves low-risk cases by policy, and grants least-privilege access scoped by the contract's access policy — with a TTL and an audit log. Self-serve turns onboarding into a click; the policy, TTL, and audit keep it governed.

Senior interview question on discoverability and self-serve

A senior interviewer might ask: "You have fifty data products and new teams constantly asking 'is there a dataset for X, and can I trust it?' — usually by DMing whoever they think owns it. Design the discoverability layer: what metadata each product publishes so a stranger can find, understand, and trust it, how you keep that metadata from going stale, and how a discoverer gets governed access without the owner becoming a bottleneck."

Solution Using a catalog entry, metadata as code, machine-captured lineage, and governed self-serve access

# 1. Rich catalog metadata per product — find, understand, trust.
urn: "urn:dataproduct:commerce:orders"
owner: { team: analytics-platform, on_call: "#oncall-analytics" }
tags: [gold, revenue, pii-safe]
glossary: { revenue: "sum(total_cents) where paid" }
schema: [ { name: order_id, type: string }, { name: total_cents, type: long } ]
lineage: { upstream: [raw.orders], downstream: [dashboards.revenue] }
trust:  { version: 1.2.0, slos: { freshness: "<=15m", quality: ">=99%" }, status: healthy }

# 2. Metadata as code — emitted from the descriptor on EVERY pipeline run (never stale).
emit_to_catalog(urn, from_descriptor="data-product.yaml", from_contract="contracts/orders.yaml")
emit_lineage(urn, from_pipeline_graph=True)   # lineage captured, not drawn by hand

# 3. Governed SELF-SERVE access — the owner sets policy, not per-request DMs.
access:
  entrypoint: catalog
  auto_approve_if: [ requester.domain == "commerce", purpose == "analytics" ]
  grant: { role: orders_reader, row_policy: "tenant_id = requester.tenant_id", ttl: 90d }
  audit: true

Step-by-step trace.

Concern	Mechanism	Effect
Findable	catalog entry + tags + search	a stranger finds it
Understandable	schema + glossary	meaning without the author
Trustworthy	version + SLOs + health	judge fitness now
Provenance	machine-captured lineage	source + consumers visible
Not stale	metadata as code, every run	catalog matches the repo
Accessible	governed self-serve + policy	access without a bottleneck

After the rollout, a new team searches the catalog, finds orders, reads its glossary and schema to understand it, checks its version/SLOs/health to trust it, sees its lineage for provenance, and clicks "Request access" — auto-approved by policy for a low-risk analytics use, granted a tenant-scoped least-privilege role with a 90-day TTL and an audit log. The metadata never goes stale because it is emitted from the descriptor and contract on every pipeline run, and lineage is captured from the transformation graph. The owner set the policy once and is no longer a bottleneck.

Output:

Metric	DM-the-owner discovery	Catalog + metadata-as-code + self-serve
Time to find a dataset	hours/days of asking	a catalog search
Understanding a field	ping the author	glossary + schema
Trust decision	guesswork	version + SLOs + health
Metadata accuracy	stale wiki	regenerated each run
Getting access	a favour	a governed click
Owner as bottleneck	yes	policy, not per-request

Why this works — concept by concept:

• Rich catalog metadata — publishing schema, glossary, lineage, owner, and trust signals lets a stranger find, understand, and trust a product without its author, which is the entire definition of discoverability.
• Metadata as code — generating the catalog entry from the same descriptor and contract on every pipeline run means the catalog always matches reality, structurally eliminating the stale-catalog failure that makes hand-maintained catalogs worthless.
• Machine-captured lineage — emitting lineage from the transformation graph rather than drawing it by hand produces provenance that stays true as pipelines evolve, serving both discovery and change-impact analysis.
• Governed self-serve access — a policy-driven request that routes to the owner, auto-approves low-risk cases, and grants least-privilege scoped access with a TTL removes the owner as a bottleneck while keeping access audited and secure.
• Cost — one emitted catalog entry and one access policy per product, versus an owner fielding endless "is there a dataset for X?" DMs. The eliminated cost is O(consumers × products) manual discovery and access requests — a catalog plus self-serve makes it O(1) search-and-click per consumer, with the owner setting policy once.

Design
Topic — design
Design problems on data catalogs, discoverability, and self-serve

Practice →

Data transformation
Topic — data-transformation
Data transformation problems on metadata, lineage, and glossary semantics

Practice →

---

Cheat sheet — data products in practice

• Table vs product. A table is bytes with a name; a data product is bytes plus an output port over a contract, a semantic version, a measured SLA, and a discoverable catalog entry with a named owner. Missing any one pillar means it is a liability, not a product. Under data mesh, ownership is a domain team with an on-call — never "the platform."
• Promotion checklist. Before anyone depends on a table: (1) expose a curated port over a typed contract, (2) stamp a semantic version, (3) publish freshness/availability/quality SLOs with a monitor, (4) register a catalog entry with lineage and owner. Describe all four in one version-controlled data-product descriptor.
• Output ports. SQL/table, REST/GraphQL API, file/object (Parquet), and stream/topic (Kafka) — every port serves the same contract. Ports are additive (add a door for a new consumer), the contract is the invariant (one meaning at every door). Never expose raw tables; never fork a divergent dataset per consumer.
• Data contract. Express once (ODCS-style YAML): typed schema + semantics (units, grain, UTC time) + quality rules + access policy. Every port references it; CI enforces it; the catalog harvests it. unit: cents and timezone: UTC in the contract kill the two commonest data bugs.
• Enforcement gate. Validate produced data against the contract before publishing a port, and fail closed — on any violation, block the publish and keep the last-good version live. The gate runs the same rules that back the quality SLO.
• Semantic versioning. Classify by consumer impact: MAJOR = breaking (rename, drop, retype, tighten), MINOR = additive backward-compatible (new nullable field, open-enum value), PATCH = non-schema (docs, tags). The diff size never decides the bump; "would an unchanged consumer break?" does.
• Compatibility + registry. Set an explicit mode (BACKWARD is the common default) in a schema registry so incompatible changes are rejected at registration, not at read time. Add fields with defaults to stay compatible; a MAJOR moves to a new version, never an in-place edit.
• Deprecation. Ship a breaking change as a new major alongside the old, mark the old deprecated with a sunset date and a migrate_to pointer, and drive the sunset with read telemetry — retire only when reads reach zero. Parallel versions + a clock + evidence = a break with no broken consumers.
• SLIs / SLOs / SLA / error budget. SLI = the measurement, SLO = the internal target (stricter than the SLA), SLA = the consumer promise, error budget = 100% − SLO. Three data dimensions: freshness (minutes since load), availability (present/expected rows), quality (contract-test pass rate). No SLA dimension without an SLI behind it.
• Monitoring. Scheduled SLI queries per dimension; alert on SLO breach and error-budget burn rate, not on every blip; debounce transient noise; page the descriptor's named on-call. A fast budget burn freezes feature work — reliability as a governed decision, not a debate.
• Discoverability. A catalog entry (DataHub/OpenMetadata/Unity Catalog/Amundsen) with a stable urn, named owner, glossary semantics, typed schema, upstream/downstream lineage, version + SLOs + health, tags/PII classification, and a self-serve access path. Answer every discovery question in the catalog, not in a DM.
• Metadata as code. Emit the catalog entry from the same descriptor + contract on every pipeline run, and capture lineage from the transformation graph. A regenerated entry cannot go stale; a product with no owner or missing PII tag fails to publish. Self-serve access = governed request → owner policy → least-privilege scoped grant with a TTL and audit.
• The eight usability attributes (scorecard). Discoverable, addressable, understandable, trustworthy, natively accessible, interoperable, secure, valuable on its own. Score a candidate product against all eight — a gap on any one is the next thing to build.

Frequently asked questions

What makes a dataset a data product (vs a raw table)?

A raw table is bytes with a name; a data product is that data promoted with four added properties that let other teams depend on it safely: an explicit output port over a typed contract (so consumers couple to a stable interface, not your internal columns), a semantic version (so it can change without breaking anyone), a published and measured SLA/SLO for freshness, availability, and quality (so it is trustworthy, not just present), and discoverability through a catalog entry with a named owner and lineage (so a stranger can find, understand, and self-serve it). Under a data-mesh operating model the bottleneck is never the query — it is the coupling — and every one of those properties exists to decouple a consumer from the producer, so you can re-model storage without breaking anyone and onboard a new consumer without a Slack thread. The concise test: if a table is missing a contract, a version, an SLA, or a catalog entry with an owner, it is a liability, not a product.

What is an output port and how many should a data product have?

An output port is a typed, governed access interface through which consumers read the product — a SQL/table port, a REST/GraphQL API port, a file/object-export port, or a stream/topic port. You expose as many ports as you have distinct consumer access patterns, and no more: an analytics team wants SQL, an application wants a low-latency API, a partner wants bulk files, a real-time consumer wants a stream. The discipline that keeps this from becoming a maintenance nightmare is that every port serves the same data contract — one typed schema, one set of semantics, one access policy — so the ports are additive doors onto a single product rather than divergent copies of the data. A field like total_cents means integer cents at every port, because every port projects the one contract. Start with the one port your first consumer needs; add ports as new access patterns appear, never a forked dataset per team.

How do I version a data product's schema without breaking consumers?

Apply semantic versioning to the contract and classify every change by consumer impact. Additive, backward-compatible changes — a new nullable field, a new value in an enum documented as open — are MINOR bumps that existing consumers can ignore, so you can ship them in place. Breaking changes — renaming or dropping a field, changing a type or unit, tightening a constraint — are MAJOR bumps, and the rule is that you never mutate the current version in place: you ship the new major alongside the old one, mark the old one deprecated with a sunset date and a pointer to the new version, and use read telemetry (query logs, consumer groups) to see who still depends on the old version. A schema registry with an explicit compatibility mode (BACKWARD is the common default) enforces this by rejecting incompatible changes at registration time, so a break physically cannot ship silently. You retire the old version only when its reads reach zero — parallel versions plus a deprecation clock plus evidence is how a dozen teams migrate on their own schedules without a single break.

What SLAs should a data product publish?

Publish SLAs across the three dimensions that describe how data actually fails: freshness (is it recent enough — measured as minutes since the last successful load relative to your target), availability/completeness (is it reachable and whole — measured as the ratio of present rows to expected rows, because a product can be queryable yet missing half its data), and quality/correctness (does it satisfy the contract — measured as the pass rate of the not-null, unique, range, and enum rules). The key discipline is that each SLA dimension must have an SLI (a computed measurement) and an SLO (a numeric target stricter than the consumer-facing SLA, for headroom) behind it, plus a monitor that pages a named owner on breach — an SLA with no SLI is decoration. Add an error budget (100% − SLO) so reliability becomes a resource the team spends deliberately: an intact budget means you can ship features, a fast burn means you freeze them and fix reliability. Alert on SLO breach and burn rate, not on every anomaly, so the pager stays trustworthy.

How do I make a data product discoverable?

Publish rich metadata to a data catalog (DataHub, OpenMetadata, Unity Catalog, or Amundsen) so a stranger can find, understand, trust, and access the product without contacting its author: a stable address (urn), a named owner with an on-call, the typed schema plus a business glossary defining what the fields and metrics mean, upstream and downstream lineage, trust signals (version, SLOs, current health), tags and PII classification for search and governance, and a self-serve access path. The discipline that keeps the catalog honest is publishing this metadata as code — emitting it from the same product descriptor and contract that define the product, on every pipeline run, with lineage captured from the transformation graph rather than drawn by hand. A hand-maintained wiki page is stale by definition; a regenerated catalog entry cannot drift from reality, and a product with no owner or an unclassified PII column simply fails to publish. Finish with governed self-serve access — a request that routes to the owner, auto-approves low-risk cases by policy, and grants least-privilege scoped access with a TTL and an audit log — so the owner sets policy once instead of fielding access DMs forever.

Data product vs data contract vs data mesh — how do they relate?

They are three levels of the same idea. Data mesh is the operating model: a decentralised approach where domain teams own and serve their data as products, instead of a central team owning one monolithic pipeline — it is the organisational principle that says "treat data as a product with a domain owner." A data product is the unit that model produces: a single owned, versioned, discoverable dataset with output ports, SLAs, and a catalog entry — the thing this article is about building. A data contract is one component of a data product: the typed schema plus semantics, quality rules, and access policy that the output ports serve and that CI enforces — the machine-readable promise at the product's boundary. So the containment is mesh ⊃ products ⊃ contracts: the mesh is why you build products, the product is what you ship, and the contract is the interface that makes the product safe to depend on. You can adopt data-product and data-contract discipline for a single important dataset without a full mesh — the practices stand on their own — but the mesh is the operating model that makes them the default across an organisation.

Practice on PipeCode

• Drill the data product design practice library → for the product-boundary, output-port, and discoverability trade-offs that turn a raw table into something fifty teams can depend on.
• Rehearse interface work on the API integration practice library → for the output-port and data-contract problems that PostgREST, Hasura, and file/stream ports make concrete.
• Sharpen the contract axis with the data validation practice library → for the schema-compatibility, freshness, completeness, and quality-SLI scenarios where versioning and SLAs earn their keep.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the contract, semantic-versioning, SLO, and catalog patterns against real graded inputs — ports, compatibility modes, freshness monitors, and metadata as code.

Lock in data-product muscle memory

Docs explain data mesh and data contracts. PipeCode drills explain the decision — when a table must become a product, when a schema change is a `MAJOR` bump that ships in parallel, when an SLA needs an SLI behind it, and when metadata-as-code beats a wiki page nobody updates. Pipecode.ai is Leetcode for Data Engineering — data-product practice tuned for the production trade-offs senior data engineers actually face.

Practice data product design problems →
Practice API integration problems →