Data Platform Team Structure: Central Platform + Embedded DEs + Analytics Engineers

data platform team structure is the decision that quietly determines whether your data org ships trustworthy pipelines in weeks or drowns in a two-quarter backlog of ticket-driven requests — and it is the one architectural choice leaders get wrong most often because they treat it as an HR reporting-line question instead of a systems-design question. The shape of the data platform team — a single central data platform everyone queues behind, embedded data engineers sitting inside product squads, or a hybrid where a platform team paves the road and embedded engineers drive on it — decides how fast a new data product ships, who is accountable when a dashboard is wrong, whether governance is a bottleneck or a guardrail, and how the org scales from ten engineers to a hundred without collapsing into either chaos or gridlock. The trade-off does not live in "should we have a data team"; every company past its first analyst needs one. It lives in which topology you pick and what that topology costs you in latency, ownership, and standards drift.

This guide is the leader-and-staff-engineer walkthrough you wished existed the first time someone asked "how should we structure the data org as we scale," or "central platform or embedded engineers — where do the DEs report," or "when do we adopt data mesh, and what does an analytics engineer actually own." It walks through the framing (Conway's Law for data and the three canonical archetypes), the central platform team run as an internal product, embedded data engineers and the dotted-line reporting problem, analytics engineers and the dbt-era boundary between engineering and analysis, and finally scaling with team topologies, data mesh, headcount ratios, and the anti-patterns that stall a data team org. Each section pairs a teaching block with a Solution-Tail interview answer — a concrete artifact (a charter, a RACI, an org map), a step-by-step trace, an output table, then a concept-by-concept breakdown of why the structure works.

When you want hands-on reps immediately after reading, drill the system design practice library →, rehearse the modelling foundations on the dimensional-modeling practice library →, and sharpen the pipeline axis with the ETL practice library →.

---

On this page

• Why org structure is a data-architecture decision
• The central platform team: paved roads and self-serve
• Embedded data engineers and the dotted-line problem
• Analytics engineers and the dbt-era boundary
• Scaling the org: topologies, data mesh and anti-patterns
• Cheat sheet — data-team-structure recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why org structure is a data-architecture decision

The org chart becomes the data architecture — Conway's Law is not a metaphor for data teams, it is the mechanism

The one-sentence invariant: your data platform team structure is not a management detail bolted onto the architecture after the fact — it is the architecture, because Conway's Law guarantees the pipelines, the ownership boundaries, the schemas, and the SLAs will mirror the communication structure of whoever builds them, and the three archetypes (centralized, embedded/decentralized, hybrid platform + embedded) each hard-code a different set of trade-offs between delivery speed, domain accuracy, and governance that cannot be undone by tooling alone. A company that centralizes every data engineer into one team ships a warehouse that looks like a single monolith with a request queue in front of it. A company that scatters engineers into product squads with no coordinating function ships fifteen incompatible definitions of "active user." The structure you pick in year one becomes the migration you fight in year three, because every domain, dashboard, and downstream consumer hard-codes assumptions about who owns what.

The three archetypes every data leader must be able to name.

• Centralized. All data engineers, analytics engineers, and analysts report into one data organization that serves the whole company. Strong standards, one warehouse, one set of tools — but the central team becomes a bottleneck the moment demand outpaces headcount, and it sits far from the domain knowledge locked inside product teams.
• Decentralized / embedded. Data engineers are distributed into product or business squads, reporting to those squads. Fast iteration, deep domain context, clear product accountability — but standards drift, tools fragment, and there is nobody who owns the shared platform, so every squad rebuilds ingestion from scratch.
• Hybrid (platform + embedded). A central platform team owns the shared substrate (ingestion, orchestration, catalog, governance) and runs it as an internal product; embedded data engineers and analytics engineers sit in the domains and build on the paved road. This is the topology most scaling companies converge on, and the one senior interviewers expect you to defend.

Conway's Law for data — the mechanism, stated precisely.

• The law. Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure. For software this is famous; for data it is under-appreciated and more brutal, because data systems are integration systems — their whole job is to join across domains.
• The consequence. If two teams do not talk, their data will not join cleanly. Two squads that own "orders" and "payments" separately, with no shared platform team forcing a common customer_id contract, will produce two customer dimensions that cannot be reconciled without a heroic quarter of remediation.
• The lever. Because the law is a mechanism and not a curse, you can use it deliberately. Want a single source of truth for the customer? Give one team clear ownership of the customer data product and a contract everyone consumes. Want fast domain iteration? Embed engineers where the domain lives. The org chart is the API surface you are actually designing.

What breaks at each stage of growth.

• Early (centralized, 3–10 people). Works well. One team, one warehouse, tight standards. Breaks when the request queue grows faster than the team — every stakeholder is waiting behind a ticket, and the central team is a hostage to its own backlog.
• Mid (embedding starts, 10–40 people). Companies push engineers into domains for speed. Works until nobody owns the shared platform — five squads maintain five Airflow instances, three metric definitions for revenue, and no shared lineage. This is the "silo stage."
• Scale (hybrid or mesh, 40+ people). The platform team is carved out to own the substrate; domains own their data products on top. Breaks only when governance is under-invested (ungoverned mesh = a thousand data products nobody can find) or the platform team drifts back into being a bottleneck by owning too much.

What interviewers and leaders actually probe.

• Do you name all three archetypes and their failure modes without prompting? — senior signal.
• Do you invoke Conway's Law as the reason the structure matters, not as a buzzword? — senior signal.
• Do you tie structure to ownership and accountability ("who is paged when this dashboard is wrong") rather than to headcount? — required answer.
• Do you describe the hybrid platform-plus-embedded model as "platform paves the road, domains drive on it" rather than as "some central, some embedded"? — senior signal.
• Do you connect the org decision to a concrete failure the current structure is causing rather than reorganizing for its own sake? — required answer.

Worked example — the three-archetype comparison table

Detailed explanation. The single most useful artifact for a data-org-structure discussion is a memorised 3×N comparison table. Every serious conversation about how to structure a data team converges on it within the first ten minutes; having it in your head is what separates a fluent leader from someone reciting org-chart clichés. Walk through building the table for a hypothetical mid-size company with three product lines (checkout, growth, logistics) that currently has one central data team of eight people and a two-quarter backlog.

• Delivery speed. How fast can a squad get a new data product or metric shipped without waiting on another team.
• Domain accuracy. How correct the semantics are — does "active user" mean the same thing everywhere, and does whoever built the pipeline understand the business.
• Governance and standards. Whether there is one warehouse, one set of tools, consistent lineage, and enforced contracts.
• Failure mode. The specific way each archetype breaks under load.

Question. Build the three-archetype comparison for this company and pick the archetype it should move toward given a two-quarter backlog and three domains.

Input.

Archetype	Delivery speed	Domain accuracy	Governance	Primary failure mode
Centralized	slow (queue-bound)	weak (far from domain)	strong (one team)	central bottleneck
Decentralized / embedded	fast (in-squad)	strong (in the domain)	weak (fragmentation)	silos + drift
Hybrid (platform + embedded)	fast on the paved road	strong (embedded)	strong (platform owns substrate)	platform re-centralizes if it owns too much

Code.

Data-org archetype decision memo (1 page)
=========================================

Context
  - 8-person central data team, 2-quarter backlog.
  - 3 product lines: checkout, growth, logistics.
  - Symptom: every squad waits behind one ticket queue;
    domain teams complain the central team "does not
    understand our data."

Diagnosis (Conway's Law)
  - Centralized structure => one integration point => one
    queue => bottleneck. Architecture mirrors org: a monolith
    warehouse with a request desk in front.

Options
  A. Stay centralized, add headcount.      -> queue shrinks
                                              but returns.
  B. Fully embed 1 DE per squad, no center. -> fast now,
                                              silos in 2 quarters.
  C. Hybrid: carve a 3-person platform team
     (ingestion, orchestration, catalog),
     embed 1 DE + 1 AE per domain.          -> paved road +
                                              domain speed.

Recommendation
  - Move to (C). Platform team owns the substrate as a product;
    domains own their data products on the road.
  - 90-day plan: publish paved-road ingestion + a single
    semantic layer contract; embed engineers into checkout and
    growth first (highest backlog), logistics next quarter.

Step-by-step explanation.

1. The company is centralized, so by Conway's Law its architecture is a single integration point with a queue in front — exactly the bottleneck symptom the backlog describes. The structure is causing the delivery problem; adding headcount (Option A) treats the symptom, not the mechanism.
2. Full embedding (Option B) fixes speed immediately but removes the only thing keeping standards coherent. In two quarters the three domains will have three ingestion stacks and three definitions of revenue — the silo failure mode. Fast today, expensive to reconcile later.
3. The hybrid (Option C) splits the problem: extract a small platform team to own the shared substrate as an internal product, and embed engineers into the domains for speed. This is the only option that improves delivery speed and preserves governance.
4. The sequencing matters. You do not embed everyone at once — you pave the road first (a self-serve ingestion path and a single semantic contract), then embed into the highest-backlog domains, so embedded engineers have a road to drive on instead of paving their own.
5. The recommendation is written as a one-page memo, not an org chart. Leaders decide on the trade-off, not the boxes-and-lines; the boxes fall out of the trade-off once the platform-versus-domain boundary is agreed.

Output.

Domain	Structure after move	Owns
Platform team (new, 3 people)	central, product-run	ingestion, orchestration, catalog, contracts
Checkout squad	embedded DE + AE	checkout data products on the paved road
Growth squad	embedded DE + AE	growth metrics, experimentation tables
Logistics squad (next quarter)	embedded DE + AE	logistics data products

Rule of thumb. Never reorganize a data team by copying another company's org chart. Diagnose the failure mode first (bottleneck, silo, or ungoverned), map it to the archetype that is causing it via Conway's Law, then pick the structure whose failure mode you can afford. Write the trade-off as a one-page memo before you draw a single box.

Worked example — a Conway's-Law mapping exercise

Detailed explanation. The most senior move in a data-org discussion is to run the mapping in reverse: given the data architecture you want, derive the org structure that will naturally produce it. Codifying this makes the reorg defensible — you are not moving people for taste, you are aligning the communication structure with the target architecture. Walk through the mapping for a company that wants a single trusted "customer 360" data product consumed by every domain.

• Target architecture. One canonical customer data product with a published contract; every domain (orders, support, marketing) consumes it rather than rebuilding it.
• Conway's implication. A single, clearly-owned artifact requires a single, clearly-owned team with a stable contract and a communication channel to every consumer.
• The anti-pattern to avoid. Three domains each maintaining their own customer table means three teams, three schemas, no single owner — Conway's Law guarantees fragmentation.

Question. Map the desired "customer 360" architecture to the team structure and contract that will produce it, and name the coordination mechanism.

Input.

Target architecture property	Org structure it requires
One canonical customer data product	One team with clear, sole ownership
Stable schema every domain trusts	A published data contract + versioning
Domains consume, do not fork	A consumer channel (guild / contract review)
Changes do not silently break consumers	Contract tests in CI + deprecation policy

Code.

## Customer-360 ownership contract

**Owner:** Platform / Customer-data pod (single team, on-call rotation)
**Consumers:** orders, support, marketing, finance

### The contract
- `customer` data product exposes: `customer_id` (stable, never reused),
  `email_hash`, `first_seen_at`, `status`, `lifetime_value_cents`.
- Schema is versioned (`v1`, `v2`); breaking changes ship a new version
  and a 2-sprint deprecation window on the old one.
- SLA: freshness < 1h, availability 99.9%, published on the catalog.

### Coordination mechanism
- Any consumer needing a new field files a change request in the
  data-guild channel; the owning pod triages weekly.
- Contract tests run in the owner's CI: a PR that breaks `customer_id`
  stability or drops a field fails the build.

### Anti-pattern this prevents
- No domain maintains its own `customer` table. Forking the customer
  entity is a governance violation, not a shortcut.

Step-by-step explanation.

1. The target is a single canonical artifact, so Conway's Law says you need a single owning team. Spreading customer ownership across three domains would, by the same law, produce three customer definitions — the exact fragmentation you are trying to avoid.
2. A stable schema that other teams trust requires a published contract with versioning. Without it, the owning team ships a breaking change on a Tuesday and four dashboards break on Wednesday. The contract is the API between the owning team and its consumers.
3. The coordination mechanism — a guild channel plus a weekly triage — is the "communication structure" Conway's Law is really about. It is the deliberate channel that lets consumers influence the product without forking it.
4. Contract tests in the owner's CI turn the social agreement into an enforced one. A PR that breaks customer_id stability fails the build; enforcement at the pipeline level is far more reliable than enforcement in a wiki page nobody reads.
5. The explicit anti-pattern clause ("no domain maintains its own customer table") is what keeps the architecture from degrading. Without a stated prohibition, the first team under deadline pressure forks the entity and the single-source-of-truth is gone.

Output.

Property	Without the mapping	With the mapping
Number of customer definitions	3+ (one per domain)	1 (canonical)
Breaking-change blast radius	silent, discovered in prod	caught by contract tests
Consumer influence	forking	change request + triage
Ownership when it's wrong	ambiguous	one on-call pod

Rule of thumb. To get the architecture you want, design the org that Conway's Law will translate into it: one canonical artifact needs one owning team plus a published, versioned, test-enforced contract and a real communication channel for consumers. If you cannot name the single owner, you do not have a data product — you have a fork waiting to happen.

Worked example — diagnose this failing data team

Detailed explanation. Leaders and interviewers love a "here is a broken data team, tell me what is wrong and how you would fix it" scenario, because it tests whether you diagnose structure or just add tools. The skill is to map the symptoms to the archetype failure mode and prescribe a structural change, not a technology. Walk through a realistic broken team.

• Symptom set. A 25-person data org, fully embedded (each of six squads has 3–4 data people, no central team). Six Airflow deployments. Four definitions of "monthly active user." Nobody can produce a company-wide revenue number without a week of reconciliation. New hires take two months to find where anything lives.
• The trap. The naive fix is "buy a data catalog" or "standardize on one orchestrator." Tools do not fix a structural gap; they need an owner.
• The real diagnosis. This is the classic embedded-silo failure mode: fast local delivery, zero shared substrate, no owner of cross-domain semantics.

Question. Diagnose the failing team by archetype and prescribe the minimal structural change that fixes the root cause.

Input.

Symptom	Surface reading	Structural root cause
6 Airflow deployments	"tooling sprawl"	no platform owner
4 definitions of MAU	"data quality issue"	no owner of shared metrics
Week-long revenue reconciliation	"reporting is slow"	no canonical finance data product
2-month onboarding	"docs are bad"	no catalog, no single substrate

Code.

Failing-team diagnosis + remediation
====================================

Archetype: fully decentralized / embedded (no center).
Failure mode: silos + standards drift (textbook).

Root cause (NOT tooling):
  There is no team that OWNS the shared substrate or the
  cross-domain semantic contracts. Six squads each optimize
  locally; nobody is accountable for the whole.

Minimal structural fix (do not re-centralize everything):
  1. Carve a 4-person platform + enabling team from the
     existing 25 (do not hire first; reallocate).
       - Owns: one orchestration standard, one catalog,
         the semantic layer for shared entities (user,
         revenue), and the paved-road ingestion path.
  2. Keep DEs/AEs embedded in the six squads, but add a
     dotted line to the platform team (a guild) for standards.
  3. Declare 2 cross-domain data products canonical:
     `active_user` and `revenue`; one squad owns each with
     a contract; other squads consume, do not fork.

What NOT to do:
  - Do not fully centralize (kills the delivery speed the
    embedding bought).
  - Do not "buy a catalog" and hope; a catalog with no owner
    is an empty catalog in 3 months.

Step-by-step explanation.

1. The symptoms all point at one root: there is no owner of the shared substrate. Six Airflow deployments, four MAU definitions, and week-long reconciliation are not four separate problems — they are one structural gap (no platform + no cross-domain semantic owner) expressed four ways.
2. The trap is to prescribe tools. A catalog, a single orchestrator, or a metrics layer are all necessary — but each needs a team that owns it. Buying tooling without assigning ownership recreates the silo with nicer logos.
3. The fix is additive and minimal: carve a small platform + enabling team from existing headcount rather than hiring, so it lands this quarter. Reallocation signals the change is structural, not a growth bet.
4. Crucially, you keep the embedding. The embedded model bought real delivery speed; re-centralizing everything would trade one failure mode (silos) for another (bottleneck). The dotted-line guild preserves speed while restoring standards.
5. Declaring exactly two entities canonical (active_user, revenue) is deliberate scope control. You fix the highest-pain cross-domain semantics first with contracts, rather than trying to boil the ocean of governance on day one.

Output.

Before	After
6 orchestrators, no owner	1 paved-road standard, platform-owned
4 MAU definitions	1 canonical active_user contract
Week-long revenue reconcile	1 canonical revenue data product
No coordination	dotted-line guild + weekly triage

Rule of thumb. When a data team is failing, diagnose the archetype and its failure mode before you prescribe anything. Silos need a platform owner and shared contracts, not more tools; bottlenecks need embedding and self-serve, not more headcount. The fix is almost always structural, and the smallest structural change that removes the root cause beats the biggest tooling purchase.

Senior interview question on data org structure

A senior interviewer often opens with: "You join as head of data at a 120-person company. The data function is a single centralized team of twelve with a two-quarter backlog; product teams route every request through a ticket queue and complain the data team does not understand their domains. Walk me through how you would restructure the data org, what you would sequence in the first 90 days, and how you would prevent the new structure from breaking as the company doubles."

Solution Using a hybrid platform-plus-embedded reorg with a paved road and a 90-day sequence

Target structure (hybrid platform + embedded)
=============================================

  Central Platform team (4)          <- product-run substrate
    - paved-road ingestion, orchestration standard, catalog,
      semantic layer for shared entities, governance guardrails
    - runs as an internal product with an SLA and a roadmap

  Enabling team (2, time-boxed)      <- upskills domains, then dissolves
    - embeds temporarily to bootstrap each domain onto the road

  Embedded pods (1 DE + 1 AE per domain, ~6 pods)
    - solid line to the domain (delivery + accountability)
    - dotted line to platform via a data guild (standards, craft)

  Analytics engineers
    - own transformation + semantic models per domain, on the
      platform's shared semantic layer

## 90-day reorg sequence

### Days 0-30 — stabilize + pave
- Freeze the ticket queue triage into 2 lanes: platform work vs
  domain work. Stop being a request desk.
- Carve the 4-person platform team from the existing 12
  (reallocate, do not hire yet). Publish the paved-road
  ingestion path and one orchestration standard.
- Stand up a catalog + a single semantic layer skeleton.

### Days 30-60 — embed where the pain is
- Embed 1 DE + 1 AE into the top-2 backlog domains (checkout,
  growth). Solid line to the domain; dotted line to the guild.
- Migrate those 2 domains onto the paved road; delete their
  bespoke ingestion.
- Declare `active_user` and `revenue` canonical data products
  with contracts + contract tests in CI.

### Days 60-90 — scale + govern
- Embed into remaining domains. Enabling team rotates through
  each to bootstrap, then steps back.
- Publish SLAs/SLOs for the platform (freshness, availability).
- Establish the data guild: weekly standards triage, shared
  on-call for the platform, career ladder that values embedded
  and platform paths equally.

Step-by-step trace.

Step	Before (centralized)	After (hybrid)
Delivery model	ticket queue, one team	self-serve paved road + embedded pods
Domain accuracy	weak (team far from domain)	strong (DE/AE in the domain)
Governance	strong but bottlenecked	platform owns substrate + contracts
Ownership when wrong	ambiguous ("data team")	embedded pod (delivery) + platform (substrate)
Bottleneck	one queue for everything	platform work vs domain work split
Scaling behavior	linear headcount to keep up	road scales, pods add per domain

After the reorg, product squads stop queuing behind a central desk: they build on a self-serve paved road with embedded engineers who understand their domain, while a small platform team keeps the substrate and the shared contracts coherent. The two-quarter backlog drains because most requests are now self-serve; the platform team's roadmap is the queue for genuinely shared work, and cross-domain semantics stay consistent because active_user and revenue are owned data products, not per-squad forks.

Output:

Metric	Before	After
Median request lead time	weeks (queue-bound)	days (self-serve + embedded)
Cross-domain metric definitions	many, inconsistent	canonical, contract-enforced
Domain context in pipelines	low	high (embedded)
Governance owner	overloaded central team	dedicated platform team
Scaling response	add headcount to the queue	pave once, add pods per domain

Why this works — concept by concept:

• Conway's Law alignment — the reorg deliberately shapes the communication structure to match the target architecture: a shared substrate needs one owning platform team, domain speed needs embedded pods, and cross-domain truth needs owned data products with contracts. Structure and architecture stop fighting.
• Platform-as-product paved road — extracting the substrate into a self-serve product converts most of the backlog from "queue behind the central team" into "self-serve on the road," which is the only way delivery speed stops being headcount-bound.
• Embedded pods with a dotted line — solid line to the domain buys accountability and domain accuracy; the dotted-line guild buys standards without re-centralizing, so you get speed and coherence instead of choosing one.
• Canonical data products — declaring active_user and revenue as owned, contract-tested products stops the metric fragmentation at its source; consumers consume instead of forking.
• Cost — one small platform team (4) + an enabling team (2, time-boxed) reallocated from existing headcount, plus a guild and contract-test CI. The eliminated cost is the compounding O(domains²) reconciliation of divergent definitions and the perpetual central-queue tax. Net: near-constant coordination cost per new domain instead of linear-or-worse.

Design
Topic — design
System design problems on data org and ownership

Practice →

SQL Topic — sql SQL problems on canonical metrics and reconciliation

Practice →

---

2. The central platform team

Run the central data platform as an internal product — paved roads, golden paths, and self-serve, not a ticket queue

The mental model in one line: the central data platform team is the group that owns the shared data substrate — ingestion, orchestration, storage, catalog, governance, and the semantic layer — and its defining move is to run that substrate as an internal product with real users (the domain engineers), a paved-road golden path, published SLAs, and a funding model, rather than as a services desk that other teams file tickets against. The difference between a platform team that scales the org and one that becomes the new bottleneck is entirely whether it thinks "product" (build the road once, let everyone self-serve) or "service" (do the work for everyone, one ticket at a time). Every senior data leader has watched a well-intentioned central team collapse under its own request queue because it never made the product shift.

Platform-as-product — the mindset shift that defines the role.

• Users are engineers. The platform team's customers are the embedded DEs and AEs in the domains. Their "product" is measured by adoption and self-serve success, not by tickets closed. A capability nobody adopts is a failed feature, not a completed request.
• Golden path over bespoke. The team publishes an opinionated, well-supported "golden path" for the 80% case (ingest a source, schedule a job, register a dataset). Teams that stay on the golden path get support, upgrades, and SLAs for free; teams that go off-road own their own maintenance.
• Roadmap, not backlog. A product-run platform has a roadmap driven by user research (what are domains struggling with), not a backlog of one-off asks. This is the single most important cultural signal that a team has made the shift.
• Self-serve is the goal. Every recurring request should become a self-serve capability. "I need a new source ingested" should be a config change a domain engineer makes, not a ticket the platform team fulfils.

Paved-road capabilities — what the platform actually owns.

• Ingestion. A self-serve path to land a new source into the warehouse/lake with schema detection, incremental loads, and monitoring — so domains do not each rebuild ingestion.
• Orchestration. One blessed scheduler and a standard way to define, deploy, and observe pipelines. Domains write DAGs; the platform runs the runtime.
• Catalog + lineage. A single discoverable catalog with lineage, so a new hire can find the canonical revenue table in minutes, not weeks.
• Governance guardrails. Access control, PII tagging, data-contract enforcement, and quality checks provided as guardrails on the paved road, so doing the right thing is the easy path.
• Semantic / metrics layer. The shared definitions of the entities and metrics everyone uses, exposed so domains build on them instead of forking.

SLAs, SLOs, and the reliability contract.

• Publish them. A platform without published SLOs is a platform whose users cannot plan. Freshness (data is < 1h old), availability (the warehouse is up 99.9%), and support response are the three every platform commits to.
• Measure and report. SLOs mean nothing without dashboards and error budgets. When the platform burns its freshness budget, domain roadmaps slow — which is exactly the feedback loop that justifies platform investment.
• On-call. The platform team runs an on-call rotation for the substrate. If the shared ingestion path breaks at 2 a.m., the platform team is paged, not each domain individually.

The funding model — the question that decides the team's fate.

• Central cost center. The platform is funded centrally as shared infrastructure. Simple, avoids per-team friction, but risks the platform being under-invested because "nobody's budget line."
• Chargeback / showback. Domains are charged (or shown) the cost of their platform usage. Aligns incentives and surfaces waste, but adds accounting overhead and can push domains off the paved road to save money.
• The senior take. Most scaling orgs run showback (visibility without hard billing) plus central funding for the core substrate, so the platform is properly resourced and domains see the cost of their footprint.

Common interview probes on the central platform team.

• "What does a central data platform team own?" — required answer: the shared substrate (ingestion, orchestration, catalog, governance, semantic layer) run as a product.
• "How do you keep a platform team from becoming a bottleneck?" — self-serve golden paths, roadmap over backlog, adoption metrics.
• "How do you fund a platform team?" — central funding for the core + showback for usage visibility.
• "How do you measure platform success?" — adoption, self-serve rate, SLO attainment, domain lead time — not tickets closed.

Worked example — a platform-team charter and scope doc

Detailed explanation. The first artifact a new platform team ships is not code — it is a charter that says what the team owns, what it explicitly does not own, and how domains engage with it. A crisp charter is what prevents the team from silently sliding back into a services desk. Walk through writing one for a newly-carved 4-person platform team.

• In scope. The shared substrate and the golden paths on top of it.
• Out of scope. Domain-specific pipelines and metrics — those belong to embedded engineers.
• Engagement model. Self-serve first; guild for standards; roadmap for shared asks.

Question. Write the platform-team charter that defines ownership boundaries and the engagement model.

Input.

Charter section	Content
Mission	make domains productive and data trustworthy via a self-serve platform
In scope	ingestion, orchestration, catalog, governance, semantic layer
Out of scope	domain pipelines, domain metrics, dashboards
Engagement	self-serve golden path; guild for standards; roadmap for shared work
Success metrics	adoption, self-serve rate, SLO attainment, domain lead time

Code.

# Data Platform Team — Charter (v1)

## Mission
Make every domain team fast and every dataset trustworthy by
providing a self-serve data platform run as an internal product.

## What we own (in scope)
- **Ingestion**: self-serve source onboarding (config-driven),
  incremental loads, schema detection, ingest monitoring.
- **Orchestration**: one blessed scheduler + deploy/observe tooling.
- **Catalog + lineage**: single discoverable catalog for all datasets.
- **Governance guardrails**: access control, PII tagging, data
  contracts, quality checks — provided ON the paved road.
- **Semantic layer**: shared entity + metric definitions.

## What we do NOT own (out of scope)
- Domain pipelines, domain-specific transformations, dashboards,
  and domain metric logic. Those are owned by embedded DEs/AEs.
- We build the road; domains drive on it.

## How to engage with us
1. **Self-serve first**: golden-path docs cover the 80% case.
2. **Data guild**: standards, RFCs, and craft questions — weekly.
3. **Roadmap**: genuinely shared, cross-domain asks are triaged
   into the platform roadmap, not a ticket queue.

## Success metrics (reviewed monthly)
- % of domains on the golden path (adoption)
- self-serve rate (requests resolved without platform toil)
- SLO attainment (freshness, availability)
- domain lead time (idea -> shipped data product)

## Explicit anti-goal
- We are NOT a request desk. If a request recurs, we turn it into
  a self-serve capability rather than fulfilling it repeatedly.

Step-by-step explanation.

1. The mission is framed around domain productivity and trust, not around the platform's own outputs. This keeps the team honest: the platform exists to make others fast, so its success is measured by their speed, not its ticket throughput.
2. The in-scope list is the shared substrate — the things that are wasteful to rebuild per domain. Everything here has the property that centralizing it removes duplication without removing domain autonomy.
3. The out-of-scope list is the most important section. By explicitly disowning domain pipelines and metrics, the charter prevents the team from being pulled into doing domain work, which is the road back to being a bottleneck.
4. The engagement model puts self-serve first and demotes ticketed work to a roadmap for genuinely shared asks. This is the structural mechanism that keeps the team a product team rather than a services desk.
5. The success metrics and the explicit anti-goal ("we are NOT a request desk") turn the culture into something measurable. When a request recurs, the charter obliges the team to productize it — that is the flywheel that drains the backlog over time.

Output.

Question	Charter answer
Who ingests a new source?	domain self-serves on the golden path
Who owns the checkout revenue metric?	the embedded AE, not the platform
Who is paged if the scheduler is down?	platform on-call
Who decides a new shared capability?	platform roadmap via the guild
What if a request recurs?	platform productizes it (anti-goal)

Rule of thumb. A platform team's charter must state what it does not own as clearly as what it does. The out-of-scope list and the "not a request desk" anti-goal are what keep the team productized. If your charter only lists responsibilities, you have written a services-desk job description, not a platform charter.

Worked example — a paved-road capability map with SLOs

Detailed explanation. Once the charter defines the boundary, the platform team publishes a capability map: for each capability, the golden-path way to use it, the self-serve status, and the SLO. This map is the platform's product catalog — it tells domains what they can rely on and what they still have to build themselves. Walk through building it.

• Capability. Ingestion, orchestration, catalog, quality, semantic layer.
• Golden path. The blessed, supported way to use each.
• SLO. The reliability commitment for each.

Question. Build the paved-road capability map and attach an SLO to each capability.

Input.

Capability	Golden path	Self-serve?	SLO
Ingestion	config-driven source onboarding	yes	new source live < 1 day; freshness < 1h
Orchestration	standard DAG template + deploy CLI	yes	99.9% scheduler availability
Catalog	auto-registered on ingest	yes	100% of golden-path datasets cataloged
Data quality	declarative checks in the DAG template	partial	check failures alert < 5 min
Semantic layer	shared metric definitions repo	partial	metric change reviewed < 2 days

Code.

# paved-road ingestion — a domain engineer self-serves by adding
# a source config; the platform runtime does the rest.
source:
  name: checkout_events
  domain: checkout
  connector: postgres_cdc          # from the platform connector catalog
  destination: warehouse.raw.checkout_events
  load: incremental                # platform handles watermarking
  freshness_slo: 1h                # platform monitors + alerts
  pii:
    - column: email
      classification: pii          # governance guardrail: auto-masked
  owner: checkout-pod              # who is paged for domain logic
  contract: schemas/checkout_events.v1.yaml

-- Platform-provided SLO query: freshness for every golden-path source.
-- Runs on a schedule; breaches page the platform on-call.
SELECT
    source_name,
    domain,
    max(loaded_at)                                   AS last_load,
    now() - max(loaded_at)                           AS staleness,
    freshness_slo,
    (now() - max(loaded_at)) > freshness_slo         AS breaching
FROM   platform.ingest_audit
GROUP  BY source_name, domain, freshness_slo
HAVING (now() - max(loaded_at)) > freshness_slo
ORDER  BY staleness DESC;

Step-by-step explanation.

1. The capability map is a product catalog: each row tells a domain engineer exactly how to use a capability, whether they can self-serve, and what reliability they can plan around. This is what makes the platform legible to its users.
2. The ingestion golden path is config-driven — a domain engineer adds a source YAML and the platform runtime handles watermarking, freshness monitoring, and PII masking. The domain writes ~12 lines of config instead of a bespoke pipeline, which is the entire point of the paved road.
3. The pii block is a governance guardrail delivered on the road: because the golden path auto-masks tagged columns, doing the compliant thing is the default, not an extra step. Guardrails-on-the-path is how a platform enforces governance without policing.
4. The freshness SLO is not just a promise — the platform ships the query that measures it and wires breaches to on-call. An SLO the platform cannot measure is marketing; an SLO with a query and a pager is a contract.
5. The owner field encodes the split cleanly: the platform is paged if the ingestion runtime breaks, but the domain pod owns the source's business logic and contract. The capability map makes the DE-versus-platform boundary operational.

Output.

Scenario	Who acts	Why
New checkout source needed	checkout DE adds YAML	self-serve golden path
Ingestion runtime down	platform on-call	platform owns the runtime
checkout_events schema wrong	checkout pod	domain owns the contract
PII column added	auto-masked by guardrail	governance on the road
Freshness SLO breached	platform on-call, then domain if source-side	SLO query pages

Rule of thumb. Publish the platform as a capability map with a golden path and an SLO per capability. If a capability has no golden path, domains will build their own and you have a silo; if it has no SLO, domains cannot plan around it. The map is the platform's product; keep it honest by shipping the query that measures every SLO.

Worked example — a data contract as the platform's enforcement primitive

Detailed explanation. The platform's most powerful governance tool is the data contract: a machine-checkable declaration of a dataset's schema, semantics, and quality expectations, enforced in CI so a producer cannot ship a breaking change silently. Contracts are how a platform team enforces standards without becoming a review bottleneck. Walk through defining and enforcing one.

• Declare. The producing team writes a contract for the dataset it owns.
• Enforce. The platform provides a CI check that fails a PR violating the contract.
• Consume. Downstream teams read the contract to know what they can rely on.

Question. Write a data contract for the canonical active_user product and the check that enforces it in CI.

Input.

Contract element	Value
Dataset	analytics.active_user
Owner	growth-pod
Guaranteed columns	user_id, active_date, channel
Stability rule	user_id never null, never reused
Quality rule	no duplicate (user_id, active_date)
Freshness	< 3h

Code.

# contracts/active_user.v1.yaml — the canonical active-user product
dataset: analytics.active_user
owner: growth-pod
version: v1
description: One row per user per active day. Canonical MAU/DAU source.

columns:
  - name: user_id
    type: bigint
    nullable: false            # stability guarantee
    tests: [not_null, positive]
  - name: active_date
    type: date
    nullable: false
  - name: channel
    type: string
    accepted_values: [web, ios, android]

checks:
  - unique: [user_id, active_date]   # no double counting
  - freshness:
      column: loaded_at
      max_lag: 3h

policy:
  breaking_change: new_version_required   # v1 -> v2, 2-sprint deprecation

# Platform-provided CI gate: fail the PR if a change violates the contract.
import sys, yaml

def check_contract(contract_path: str, table_profile: dict) -> list[str]:
    """Return a list of violations; empty list == pass."""
    contract = yaml.safe_load(open(contract_path))
    violations = []

    declared = {c["name"] for c in contract["columns"]}
    present  = set(table_profile["columns"])

    # 1. No guaranteed column may disappear (breaking change).
    for col in contract["columns"]:
        if not col.get("nullable", True) and col["name"] not in present:
            violations.append(f"missing guaranteed column: {col['name']}")

    # 2. Uniqueness check must hold.
    for chk in contract.get("checks", []):
        if "unique" in chk and table_profile["dup_count"] > 0:
            violations.append(f"duplicate rows on {chk['unique']}")

    return violations

if __name__ == "__main__":
    issues = check_contract(sys.argv[1], load_profile(sys.argv[2]))
    if issues:
        print("CONTRACT VIOLATIONS:\n  " + "\n  ".join(issues))
        sys.exit(1)          # fails the build -> breaking change blocked
    print("contract OK")

Step-by-step explanation.

1. The contract declares not just the schema but the guarantees consumers rely on: user_id is non-null and never reused, and (user_id, active_date) is unique so nobody double-counts. These are the semantic promises that make the dataset a product rather than a table.
2. The accepted_values on channel is a semantic guardrail — a pipeline that starts emitting channel = "amp" fails the contract, catching a data-quality regression at the producer instead of in a downstream dashboard.
3. The CI gate is what turns the contract from documentation into enforcement. A PR that drops user_id or introduces duplicates fails the build; the producer cannot ship the breaking change silently. The platform provides the gate once; every contracted dataset reuses it.
4. The breaking_change: new_version_required policy encodes how change happens: you do not mutate v1 in place, you ship v2 with a deprecation window. This is how the platform lets producers evolve without breaking consumers overnight.
5. This is the platform enforcing standards without being a review bottleneck. The platform team does not manually review every dataset change — it ships the contract mechanism and the CI gate, and enforcement scales automatically across every domain.

Output.

Producer action	Contract gate result
Add a nullable column	pass (additive)
Drop user_id	fail (guaranteed column missing)
Introduce duplicate (user_id, active_date)	fail (uniqueness)
Emit unexpected channel value	fail (accepted_values)
Ship v2 with deprecation window	pass (policy-compliant)

Rule of thumb. Give the platform team a data-contract primitive and a CI gate, and it enforces standards at the speed of every domain's pipeline instead of the speed of a central review queue. Contracts are how governance scales: the platform builds the guardrail once, and correct behavior becomes the default path for every producer.

Senior interview question on the central platform team

A senior interviewer might ask: "Your central data platform team of five is drowning — it has a 40-ticket backlog, domains complain it is slow, and leadership is asking whether to just disband it and embed everyone. Make the case for the platform team, then describe exactly how you would transform it from a services desk into a product team, including what you would build, what you would stop doing, and how you would measure the change."

Solution Using a platform-as-product transformation with golden paths, showback funding, and adoption metrics

## The case for the platform team
Disbanding it re-creates the silo failure mode: 6 domains each
rebuild ingestion, orchestration, and governance -> O(domains)
duplication + divergent standards. The platform team is not the
problem; being a *services desk* is the problem. Fix the operating
model, not the existence of the team.

## Transformation plan (services desk -> product)

### Stop doing
- Stop fulfilling one-off pipeline requests. Every recurring
  request becomes a self-serve capability instead.
- Stop being the single approver on domain dataset changes;
  replace manual review with contract-test CI gates.

### Start building (golden paths)
1. Self-serve ingestion (config-driven source onboarding).
2. A standard DAG template + deploy CLI (self-serve orchestration).
3. Auto-cataloging on ingest + lineage.
4. Data-contract CI gate (governance without a review queue).
5. Shared semantic layer for `user` and `revenue`.

### Funding
- Central funding for the core substrate + showback so domains
  SEE their platform footprint (aligns incentives, no hard billing
  friction that pushes teams off-road).

-- The metric that proves the transformation: self-serve rate.
-- Requests resolved WITHOUT platform-team toil / total requests.
WITH reqs AS (
  SELECT
    date_trunc('month', created_at)                      AS month,
    count(*)                                             AS total,
    count(*) FILTER (WHERE resolution = 'self_serve')    AS self_serve
  FROM platform.requests
  GROUP BY 1
)
SELECT
  month,
  total,
  self_serve,
  round(100.0 * self_serve / nullif(total, 0), 1)        AS self_serve_pct
FROM reqs
ORDER BY month;

Step-by-step trace.

Layer	Before (services desk)	After (product)
Work intake	40-ticket backlog	self-serve + roadmap for shared work
Ingestion	platform builds each pipeline	domains self-serve via config
Governance	manual review bottleneck	contract-test CI gate
Funding	central, "nobody's line"	central core + showback
Success metric	tickets closed	self-serve rate + adoption + SLOs
Team identity	request desk	internal product team

After the transformation, the backlog stops growing because most requests become self-serve config changes domains make themselves; the platform team's roadmap becomes the queue only for genuinely shared work; governance is enforced by CI gates rather than a human review queue; and showback makes every domain's platform footprint visible so investment is justified by usage. The self-serve-rate metric climbs quarter over quarter, which is the objective proof that the team became a product.

Output:

Metric	Before	After
Open backlog	40 tickets, growing	shrinking; shared-only roadmap
Self-serve rate	~10%	70%+ target
Governance throughput	review-queue bound	CI-gate, per-pipeline
Domain lead time	weeks	days
Platform investment case	"cost center"	showback-justified

Why this works — concept by concept:

• Platform-as-product operating model — the transformation changes how the team works (product, roadmap, self-serve) rather than whether it exists. Disbanding treats the symptom; productizing removes the root cause (services-desk intake) while keeping the anti-duplication benefit.
• Golden-path self-serve — converting recurring requests into self-serve capabilities is the only mechanism that breaks the headcount-bound backlog; each productized capability permanently removes a class of tickets.
• Contract-test CI gate — governance moves from a human review queue (O(changes) toil) to an automated gate (O(1) platform build, enforced per pipeline), so standards scale with the org instead of bottlenecking it.
• Showback funding — visibility of usage aligns incentives and justifies investment without the hard-billing friction that drives teams off the paved road.
• Cost — five platform engineers building golden paths once, plus a showback pipeline and CI gates. The eliminated cost is O(domains) duplicated substrate plus a perpetually growing ticket queue. Net: the platform's cost is amortized across every domain that self-serves on it, and the self-serve-rate metric makes that leverage measurable.

Design
Topic — design
Design problems on self-serve data platforms

Practice →

ETL Topic — etl ETL problems on ingestion and orchestration paved roads

Practice →

---

3. Embedded data engineers

Embed data engineers in the domains for proximity and ownership — then solve the dotted-line problem before the silos form

The mental model in one line: embedded data engineers are DEs who sit inside product or business squads — reporting to the squad for delivery and accountability, working shoulder-to-shoulder with the engineers and PMs who own the domain — which buys deep domain context and fast iteration, but which creates the dotted-line problem: without a coordinating guild and a platform to build on, embedded engineers drift into silos, reinvent the same infrastructure, and diverge on standards until the org has as many data stacks as it has squads. Embedding is the highest-leverage structure for delivery speed and the highest-risk structure for coherence, and the entire art is capturing the speed while paying down the silo risk with a deliberate reporting and guild model.

Why embed — the benefits that make it worth the risk.

• Proximity to the domain. An embedded DE hears the product discussions, understands why an order can be in five states, and models the data correctly the first time. Domain context is the single biggest determinant of pipeline correctness, and embedding maximizes it.
• Fast iteration. No cross-team ticket to change a schema — the DE and the product engineer are in the same standup. Lead time from "we need this metric" to "it is in the dashboard" collapses.
• Clear product accountability. When the checkout dashboard is wrong, the checkout pod is accountable, not a distant central team. Ownership is unambiguous because the owner sits in the domain.
• Better prioritization. Embedded engineers work on what the domain actually needs, ranked by domain value, rather than on whatever bubbled to the top of a shared queue.

The dotted-line problem — the core tension of embedding.

• Solid line vs dotted line. The classic model: embedded DEs report solid line to their squad (day-to-day work, delivery, performance) and dotted line to a central data function or guild (standards, craft, career development, tooling). The solid line owns what they build; the dotted line owns how well they build it.
• Why the dotted line matters. Without it, an embedded DE is a data team of one, disconnected from peers, with no standards pressure and no craft community. They will make locally-reasonable choices that are globally incoherent — a different orchestrator, a different testing approach, a fork of the customer table.
• The failure of pure solid-line. If the DE reports only to the squad, the squad optimizes for its own delivery and treats shared standards as overhead. Multiply by six squads and you have six incompatible stacks — the silo failure mode.
• The failure of pure dotted-line. If the "embedded" DE actually reports to the central team and is merely seconded to the squad, they get treated as an outsider, deprioritized, and never gain real domain context — the worst of both worlds.

The guild model — how to make the dotted line real.

• What a guild is. A cross-cutting community of practice: all the embedded DEs (and AEs) across squads meet regularly, own shared standards, review each other's designs, and collectively steward the craft. It is the dotted line made concrete.
• What the guild owns. Coding standards, the golden-path adoption push, shared RFCs for cross-domain decisions, onboarding, and the career ladder for embedded engineers.
• Guild + platform together. The guild sets standards; the platform team provides the paved road that makes those standards the easy path. Guild without platform is a book club; platform without guild is a road nobody agrees to drive on.

Standards enforcement without central control.

• Contracts over review. Enforce shared standards through the platform's data-contract CI gates and shared DAG templates, not through a central team reviewing every PR. This scales; central review does not.
• Golden-path defaults. Make the standard the default by shipping templates and scaffolding. An embedded DE who scaffolds a new pipeline from the golden-path template inherits the standards for free.
• Career-path parity. Ensure embedded and platform tracks have equal promotion ceilings. If the only way to get promoted is to join the central team, your best embedded engineers leave the domains and the model collapses.

Common interview probes on embedded DEs.

• "Central vs embedded — where do DEs report?" — required answer: solid line to the domain, dotted line to a guild/platform for standards.
• "How do you stop embedded engineers from siloing?" — guild + platform paved road + contract-enforced standards + career parity.
• "What is the risk of embedding?" — standards drift, infra duplication, isolated engineers, divergent metrics.
• "When would you NOT embed?" — very early stage (too few people) or a capability that is genuinely shared and better centralized.

Worked example — the embedding RACI matrix

Detailed explanation. The cleanest way to settle "who owns what" between an embedded DE, the squad, the platform, and the guild is a RACI matrix (Responsible, Accountable, Consulted, Informed). It makes the solid-line/dotted-line split concrete for each activity and pre-empts the turf ambiguity that kills embedded models. Walk through building one.

• Actors. Embedded DE, squad lead, platform team, data guild.
• Activities. Domain pipeline delivery, tooling choice, standards, career development, incident response.

Question. Build the RACI for an embedded DE across the five activities that most often cause ownership disputes.

Input.

Activity	Embedded DE	Squad lead	Platform team	Data guild
Domain pipeline delivery	R	A	C	I
Tooling / infra choice	R	C	A	C
Coding + data standards	R	I	C	A
Career development / promo	C	R	I	A
Domain data incident	R	A	C	I

Code.

Reading the embedded-DE RACI
============================
R = Responsible (does the work)
A = Accountable (answerable; one per row)
C = Consulted   (two-way input)
I = Informed    (one-way notified)

Key resolutions this RACI encodes:
  - Pipeline delivery: DE does it (R), squad lead owns the
    outcome (A). This is the SOLID line -> the domain.
  - Tooling choice: platform is Accountable (A) so the org does
    not fragment; DE is Responsible for using it well. Domains
    do not each pick their own orchestrator.
  - Standards: the guild is Accountable (A). This is the DOTTED
    line -> craft coherence across squads.
  - Career/promo: shared A between squad lead (day-to-day) and
    guild (craft bar). Prevents "you must join the central team
    to get promoted."

Step-by-step explanation.

1. Pipeline delivery has the DE as Responsible and the squad lead as Accountable — this is the solid line encoded precisely. The domain owns the outcome; the embedded engineer does the work inside the domain. Nobody outside the squad is on the hook for domain delivery.
2. Tooling and infra choice flips the accountability to the platform team. This is the anti-silo control: individual squads do not get to pick their own orchestrator or warehouse, because that is exactly how six incompatible stacks arise. The DE is Responsible for using the shared tooling well, not for choosing new tooling.
3. Standards are the guild's accountability — the dotted line made operational. The DE follows and contributes to standards; the squad lead is only Informed, which stops a delivery-focused squad lead from overriding craft standards for short-term speed.
4. Career development is deliberately a shared accountability between the squad lead and the guild. The squad lead knows the day-to-day performance; the guild guards the craft bar and ensures the embedded path has the same ceiling as the platform path. This is the single most important row for retention.
5. Incident response mirrors delivery: the DE responds, the squad lead is accountable for the domain's data being right. The platform is Consulted because the incident might be in the shared substrate, in which case it escalates to platform on-call.

Output.

Dispute	RACI resolution
"Can our squad use its own scheduler?"	No — platform is Accountable for tooling
"Who fixes the wrong checkout metric?"	Embedded DE (R), squad lead (A)
"Who sets the testing standard?"	The guild (A)
"Can I get promoted staying embedded?"	Yes — guild + squad lead share promo accountability
"Substrate is down mid-incident"	Escalate to platform (Consulted -> on-call)

Rule of thumb. Settle embedded-team ownership with a RACI before the first turf dispute, not after. The two rows that matter most are tooling (platform Accountable, to stop fragmentation) and career development (guild + squad shared, to stop your best embedded engineers from leaving). If those two are ambiguous, the embedded model degrades into silos or attrition.

Worked example — a data-guild charter that operationalizes the dotted line

Detailed explanation. The guild is the mechanism that makes the dotted line real, but a guild without a charter is just a recurring meeting that decays into a status update. A guild charter defines its purpose, membership, decision rights, and rituals so it actually stewards standards. Walk through writing one.

• Purpose. Steward the craft and standards across all embedded data engineers.
• Decision rights. What the guild can decide vs recommend.
• Rituals. The concrete recurring activities that keep it alive.

Question. Write the data-guild charter that makes the dotted-line reporting relationship operational.

Input.

Charter element	Content
Purpose	shared standards, craft, and career coherence for embedded DEs/AEs
Membership	all embedded DEs + AEs; platform sends a liaison
Decision rights	owns standards + golden-path adoption; recommends on tooling
Rituals	weekly design review, RFC process, quarterly standards refresh
Career role	maintains the leveling rubric for embedded engineers

Code.

# Data Guild — Charter

## Purpose
Give every embedded data engineer a craft home: shared standards,
peer design review, and a career ladder that does not require
leaving the domain. This IS the dotted line.

## Membership
- All embedded DEs and analytics engineers across squads.
- A platform-team liaison (keeps the road and the standards aligned).
- Rotating facilitator (not a permanent "guild lead" -> shared ownership).

## Decision rights
- **Owns (can decide):** coding/data standards, testing conventions,
  the golden-path adoption bar, the RFC process, onboarding for new
  embedded engineers.
- **Recommends (advisory):** tooling/infra choices -> final call sits
  with the platform team (see RACI).
- **Escalates:** cross-domain data-product ownership disputes ->
  head of data.

## Rituals
- **Weekly design review**: any non-trivial pipeline or model design
  gets a 20-min peer review before it ships.
- **RFC process**: cross-domain decisions (a new shared entity, a
  standard change) go through a written RFC with guild sign-off.
- **Quarterly standards refresh**: prune, update, and re-publish
  standards so they do not rot.

## Career role
- Maintains the leveling rubric so an embedded DE can reach staff/
  principal without transferring to the platform team.

Step-by-step explanation.

1. The purpose statement names the guild as the dotted line explicitly. This is important: it tells embedded engineers that the guild is where their craft home and career coherence live, not a bureaucratic overhead meeting. The dotted line is a community, not a reporting box.
2. Membership includes a platform liaison, which keeps the guild's standards and the platform's paved road in sync. A guild that sets standards the platform does not support produces standards nobody can follow; the liaison closes that loop.
3. The decision rights are split precisely to match the RACI: the guild owns standards and craft (can decide) but only recommends on tooling (platform decides). This prevents the guild from becoming a shadow platform team while still giving it real authority over how work is done.
4. The rituals are what keep the guild from decaying. Weekly design review catches divergence before it ships; the RFC process gives cross-domain decisions a written, reviewable form; the quarterly refresh stops standards from rotting into ignored wiki pages.
5. The career-role clause is the retention mechanism. By maintaining a leveling rubric that lets embedded engineers reach staff/principal in place, the guild removes the incentive for the best engineers to abandon the domains — which is the failure mode that silently kills embedded models over 18 months.

Output.

Without a guild charter	With a guild charter
Embedded DEs isolated, no peers	craft community + peer review
Standards decided ad hoc per squad	guild-owned, refreshed quarterly
Cross-domain decisions litigated repeatedly	written RFC + sign-off
Promotion requires joining central team	in-place staff/principal ladder
Standards rot	quarterly refresh keeps them live

Rule of thumb. A guild without decision rights and rituals is a status meeting; give it clear ownership of standards, an RFC process, and a career ladder, and it becomes the dotted line that keeps embedded engineers coherent without a central command. The quarterly refresh and the in-place career ladder are the two clauses that decide whether the guild survives its first year.

Worked example — the reporting-line decision for a new domain

Detailed explanation. A concrete recurring decision: a new domain is spinning up and needs data engineering. Do you embed a DE, extend the platform, or start with the guild providing part-time support? The answer depends on domain maturity, data complexity, and shared-versus-specific needs. Walk through the decision for three new domains.

• Signals. Domain data complexity, delivery cadence, how much is genuinely shared vs domain-specific.
• Options. Full embed, platform-supported self-serve, or guild-supported part-time.
• Anti-pattern. Embedding a lone DE into a domain with no paved road to build on.

Question. Decide the reporting/support model for three new domains and justify each.

Input.

New domain	Data complexity	Cadence	Shared vs specific	Decision
Payments	high	fast	mostly specific	full embed (DE + AE)
Internal tools	low	slow	mostly shared	platform self-serve
Experimentation	high	fast	shared framework, specific data	embed AE + platform framework

Code.

Reporting-line decision tree (new domain)
=========================================

Q1. Is the domain's data complex AND its cadence fast?
      yes -> embedding is justified (proximity pays off)
      no  -> go to Q2

Q2. Is the need mostly SHARED (not domain-specific)?
      yes -> platform self-serve (do NOT embed a lone DE)
      no  -> lightweight guild support until it grows

Q3. If embedding: is there a paved road to build on?
      yes -> embed DE (+ AE if modelling-heavy)
      no  -> pave the minimal road FIRST, then embed
             (never drop a lone DE into a greenfield with no
              platform -> that is how silos are born)

Applied:
  Payments        -> Q1 yes -> embed DE + AE (complex, fast, specific)
  Internal tools  -> Q1 no, Q2 yes -> platform self-serve
  Experimentation -> Q1 yes, but framework is shared ->
                     embed 1 AE for domain models +
                     consume the platform's experimentation framework

Step-by-step explanation.

1. Payments is complex, fast-moving, and mostly domain-specific — the textbook case for full embedding. The proximity of an embedded DE and AE to the payments domain pays off immediately in correctness and speed, and there is little shared logic to centralize.
2. Internal tools is low-complexity, slow-cadence, and mostly shared needs. Embedding a dedicated DE here would be waste — the DE would be underutilized and isolated. Platform self-serve is correct: the internal-tools team uses the golden path for its modest needs.
3. Experimentation is the nuanced case: the data is domain-specific but the framework (assignment, exposure logging, stats) is shared. The right answer is a hybrid — embed an AE for the domain-specific models, but have them consume a platform-owned experimentation framework rather than build one.
4. The decision tree's Q3 encodes the most important anti-pattern check: never embed a lone DE into a greenfield domain with no paved road. Without a platform to build on, that DE will pave their own road, and you have manufactured a silo. Pave the minimal road first, then embed.
5. Reporting lines follow the decision: embedded engineers (payments, experimentation) get solid line to the domain and dotted line to the guild; the self-serve domain (internal tools) has no embedded engineer and simply consumes the platform. The structure matches the need rather than applying one template everywhere.

Output.

Domain	Model	Reporting
Payments	embed DE + AE	solid: payments; dotted: guild
Internal tools	platform self-serve	none embedded; consumes platform
Experimentation	embed AE + shared framework	solid: experimentation; dotted: guild

Rule of thumb. Do not apply one structural template to every domain. Embed where data is complex, fast-moving, and domain-specific; use platform self-serve where needs are shared and light; and never embed a lone engineer into a domain with no paved road to build on. Match the structure to the domain's complexity and cadence, and always pave before you embed.

Senior interview question on embedded data engineers

A senior interviewer might ask: "Your company embedded data engineers into six product squads a year ago. Delivery got faster, but now there are six orchestration setups, three definitions of revenue, embedded engineers say they feel isolated and two just quit for 'a real data team,' and leadership wants to yank everyone back to central. Design the reporting and coordination model that keeps the delivery speed but fixes the silos, isolation, and attrition — and explain what you would tell leadership."

Solution Using solid-line/dotted-line reporting, a chartered guild, a paved road, and career parity

Model: embedded delivery + guild coordination + platform road
=============================================================

  Embedded DE/AE in each squad
    - SOLID line -> squad (delivery, prioritization, accountability)
    - DOTTED line -> data guild (standards, craft, career)

  Data guild (all embedded engineers + platform liaison)
    - owns standards, RFCs, design review, leveling rubric
    - weekly design review; quarterly standards refresh

  Platform team
    - owns the paved road the standards ride on (one orchestrator,
      contracts, catalog, semantic layer)

  Fixes mapped to symptoms:
    6 orchestrators  -> platform owns tooling (RACI); migrate to one
    3 revenue defs   -> declare canonical `revenue` product + contract
    isolation        -> guild = craft community + peer review
    attrition        -> in-place staff/principal ladder (career parity)

## What I would tell leadership

Do NOT re-centralize. Yanking everyone back trades the silo
problem for the bottleneck problem you left behind a year ago,
and throws away the domain speed you paid for.

The problem is not embedding; it is embedding WITHOUT the two
things that make it work:
  1. A coordinating dotted line (a chartered guild).
  2. A shared paved road (a platform team) so engineers build on
     common rails instead of each paving their own.

Add those two. Keep the embedding. Declare the 2-3 highest-pain
cross-domain entities canonical with contracts. Give embedded
engineers a real career ladder so the best ones stay.

Step-by-step trace.

Symptom	Root cause	Structural fix
6 orchestration setups	no tooling owner	platform Accountable for tooling (RACI); consolidate
3 revenue definitions	no cross-domain semantic owner	canonical revenue product + contract
engineers isolated	no craft community	chartered guild + weekly design review
2 resignations	no in-place career path	staff/principal ladder for embedded track
leadership wants re-centralize	conflating "embedding" with "embedding done wrong"	keep embedding, add guild + platform

After the change, the six squads keep their embedded engineers and their delivery speed, but now build on one orchestrator and one paved road; revenue has a single contract-enforced definition; the guild gives isolated engineers a peer community and a design-review ritual; and the in-place career ladder removes the reason the two engineers left. Leadership gets the coherence they wanted without paying the bottleneck tax of re-centralization.

Output:

Metric	Before	After
Orchestration stacks	6	1 (platform-owned)
Revenue definitions	3	1 (canonical, contracted)
Embedded-engineer attrition	rising	stabilized (career parity)
Delivery speed	fast (kept)	fast (kept)
Standards coherence	drifting	guild-owned + contract-enforced

Why this works — concept by concept:

• Solid-line/dotted-line split — solid line to the squad preserves the delivery speed and accountability embedding bought; dotted line to the guild restores craft coherence. Keeping both is what distinguishes "embedding done right" from the silo failure mode.
• Chartered guild — the guild converts isolated engineers into a community with design review and shared standards, directly fixing both the isolation-driven attrition and the standards drift, without a central command structure.
• Platform paved road — one orchestrator and shared contracts give embedded engineers common rails, which is the only durable fix for infrastructure duplication; standards enforced by contract CI scale across all six squads.
• Career parity — an in-place staff/principal ladder removes the "leave for a real data team" incentive; retention of senior embedded engineers is what keeps domain context from resetting every 18 months.
• Cost — a guild (mostly time, not headcount), a platform team owning tooling, and a one-time orchestrator consolidation. The eliminated cost is O(squads) duplicated infrastructure, O(squads) divergent metric reconciliation, and the recurring cost of re-hiring and re-onboarding after attrition. Net: coordination cost is near-constant per squad instead of growing with fragmentation.

Design
Topic — design
Design problems on team ownership and reporting models

Practice →

SQL Topic — sql SQL problems on cross-domain metric consistency

Practice →

---

4. Analytics engineers and the dbt-era boundary

The analytics engineer is the dbt-era bridge — owning transformation and the semantic layer between the data engineer's pipelines and the analyst's dashboards

The mental model in one line: the analytics engineer is the role the modern data stack created to fill the gap between the data engineer (who builds ingestion and infrastructure) and the data analyst (who builds dashboards and answers questions) — the AE owns the transformation layer, turning raw ingested data into clean, tested, documented, business-ready models (typically in dbt) and often the semantic/metrics layer on top, applying software-engineering rigor (version control, testing, CI, modularity) to analytics work that used to live in untested SQL scripts and spreadsheet exports. The AE role exists because dbt and the ELT pattern moved transformation out of the pipeline and into the warehouse, and someone with one foot in engineering and one in analytics had to own that middle. Getting the DE↔AE↔analyst boundary clear is what stops the same model being built three times or falling through the cracks entirely.

What the analytics engineer actually does.

• Owns transformation. Takes raw, ingested tables and builds the staging, intermediate, and mart models that the business consumes. In the modern stack this is dbt: modular SQL models, refs, sources, and a DAG the AE owns end to end.
• Applies engineering rigor to analytics. Version control, code review, tests (uniqueness, not-null, referential integrity, freshness), CI, documentation, and modularity — the practices DEs take for granted, applied to the transformation layer that analysts previously wrote as one-off scripts.
• Owns the semantic/metrics layer. Increasingly the AE owns the shared definitions of metrics (revenue, active users, retention) so every downstream tool computes them the same way. This is the "single source of truth for metrics" that the org relies on.
• Bridges the vocabulary. The AE speaks SQL and dbt fluently (engineering) and understands the business questions (analytics), so they translate "we need to understand churn" into a tested, documented churn model the analyst can trust.

The DE ↔ AE ↔ analyst boundary — who owns which layer.

• Data engineer. Owns ingestion, the pipeline runtime, infrastructure, orchestration, and the raw/source layer landing in the warehouse. The DE gets trustworthy raw data into the warehouse reliably and at scale.
• Analytics engineer. Owns the transformation from raw to business-ready: staging, intermediate, and mart models; tests; documentation; and typically the semantic layer. The AE turns raw data into trusted, reusable models.
• Data analyst. Owns the last mile: exploration, dashboards, ad-hoc analysis, and turning trusted models into insight and recommendations for the business. The analyst answers questions using the AE's models.
• Data scientist. Owns modelling, experimentation analysis, and ML features — often consuming the AE's marts as clean inputs rather than wrangling raw data.

Where the boundary blurs (and how to settle it).

• DE vs AE on transformation. In some orgs DEs write transformations; in the modern stack AEs do. The clean split: DEs own the pipeline and the raw layer; AEs own everything from staging onward. If your DEs are writing dbt marts, you either do not have AEs or you have mislabeled them.
• AE vs analyst on modelling. Analysts sometimes build their own models when the AE is a bottleneck — which recreates the untested-SQL problem. The fix: AEs own reusable models; analysts do analysis on top. If analysts are maintaining production models, the AE layer is under-resourced.
• Who owns the semantic layer. The most-contested question. The strongest answer: the AE owns the semantic layer as code, in the same repo and CI as the dbt models, so metrics are versioned, tested, and consistent across every consuming tool. It is neither a BI-tool setting nor a DE responsibility.

The modern-data-stack ownership map.

• Ingestion (Fivetran/Airbyte/custom). DE / platform.
• Warehouse (Snowflake/BigQuery/Databricks). DE / platform.
• Transformation (dbt). Analytics engineer.
• Semantic/metrics layer (dbt metrics / semantic layer / metric store). Analytics engineer.
• BI / dashboards (Looker/Tableau/Mode). Data analyst, on the semantic layer.
• Reverse ETL / activation. DE/AE depending on org.

Common interview probes on analytics engineers.

• "What is an analytics engineer?" — required answer: the role that owns tested, documented transformation (dbt) between DE pipelines and analyst dashboards.
• "Who owns the semantic layer?" — strong answer: the AE, as versioned tested code alongside the dbt models.
• "DE vs AE vs analyst — where are the lines?" — DE owns raw + infra, AE owns transformation + semantics, analyst owns insight.
• "Why did the AE role emerge?" — dbt/ELT moved transformation into the warehouse; someone with engineering rigor and business context had to own it.

Worked example — the role-boundary matrix (DE vs AE vs DA vs DS)

Detailed explanation. The single artifact that settles most "who does what" disputes in a modern data team is a role-boundary matrix mapping each stack layer and each responsibility to a role. It stops the same model being built twice and stops layers falling through the cracks. Walk through building it for a team with all four roles.

• Layers. Ingestion, raw, transformation, semantic layer, dashboards, ML features.
• Roles. Data engineer, analytics engineer, data analyst, data scientist.
• Rule. Each layer has exactly one owning role; adjacent roles are consulted.

Question. Build the role-boundary matrix and identify the two layers most likely to be contested.

Input.

Layer / responsibility	Data engineer	Analytics engineer	Data analyst	Data scientist
Ingestion + pipeline runtime	Owner	—	—	—
Raw / source layer	Owner	Consulted	—	—
Transformation (staging → marts)	Consulted	Owner	Consulted	—
Semantic / metrics layer	—	Owner	Consulted	Consulted
Dashboards + ad-hoc analysis	—	Consulted	Owner	—
ML features + experimentation	Consulted	Consulted	—	Owner

Code.

Role-boundary rules (one owner per layer)
=========================================
DE   owns: ingestion, runtime, raw/source, infra, orchestration.
AE   owns: transformation (staging/intermediate/marts), tests,
           docs, and the semantic/metrics layer.
DA   owns: dashboards, exploration, ad-hoc analysis, insight.
DS   owns: ML features, model training, experiment analysis.

Contested layers (watch these):
  1. Transformation: DEs sometimes write marts. Rule -> if you have
     AEs, DEs stop at the raw layer; AEs own staging onward.
  2. Semantic layer: BI tool vs dbt vs DE. Rule -> AE owns it AS
     CODE (versioned, tested, in CI), not as a BI-tool setting.

Smell tests:
  - Analysts maintaining production models -> AE layer under-resourced.
  - DEs writing business marts -> AE role missing or mislabeled.
  - Metrics differ across dashboards -> no single semantic owner.

Step-by-step explanation.

1. The matrix assigns exactly one Owner per layer, which is the whole point — ambiguous ownership is what causes the same revenue model to be built by a DE, an AE, and an analyst independently. One owner per layer, adjacent roles Consulted.
2. The DE owns everything up to and including the raw/source layer: ingestion, runtime, infra, and landing trustworthy raw data. The AE is Consulted on the raw layer because they consume it, but they do not own the pipeline.
3. The AE owns transformation and the semantic layer — the two layers that define the modern AE role. The DE and analyst are Consulted (they produce inputs and consume outputs) but the AE is accountable for the models being tested, documented, and correct.
4. The two contested layers are transformation and the semantic layer. Transformation is contested because DEs historically wrote it; the rule is that with AEs present, DEs stop at raw. The semantic layer is contested because it can live in the BI tool, in dbt, or with DEs; the rule is AE-owned-as-code.
5. The smell tests turn the matrix into a diagnostic. Analysts maintaining production models means the AE layer is under-resourced; DEs writing business marts means the AE role is missing; metrics differing across dashboards means no single semantic owner. Each smell maps to a boundary the matrix would have prevented.

Output.

Symptom	Boundary violated	Fix
Same metric, 3 values	no single semantic owner	AE owns semantic layer as code
Analysts maintaining prod models	AE layer under-resourced	staff more AEs / clarify ownership
DEs writing marts	AE role missing/mislabeled	DEs stop at raw; hire/assign AEs
Marts untested	rigor not applied to transformation	AE brings tests + CI
ML features hand-built from raw	DS reinventing transformation	DS consumes AE marts

Rule of thumb. Draw a role-boundary matrix with exactly one owner per stack layer, and treat the two contested layers — transformation and the semantic layer — as AE-owned by default. When metrics disagree across dashboards, you almost always have no single semantic owner; assign it to the AE as versioned, tested code and the disagreement disappears.

Worked example — the layered-ownership map as text

Detailed explanation. A role-boundary matrix says who owns each layer; a layered-ownership map shows the flow — how data moves through the layers and where each role's responsibility begins and ends. Drawing it as an ASCII map makes the handoffs explicit and is a great whiteboard artifact. Walk through building it.

• Flow. Source → raw → staging → intermediate → marts → semantic layer → dashboards.
• Handoffs. The points where ownership passes from one role to the next.
• Contracts. What each handoff guarantees.

Question. Draw the layered-ownership map from source system to dashboard and annotate each handoff.

Input.

Layer	Owner	Handoff guarantee to next layer
Source → raw	DE	data landed, fresh, schema-detected
raw → staging	AE	typed, cleaned, renamed
staging → marts	AE	business entities, tested
marts → semantic	AE	metrics defined once
semantic → dashboards	DA	consistent numbers everywhere

Code.

Layered ownership map: source -> dashboard
==========================================

  [Source systems]          Postgres, Stripe, events
        |  DE owns: ingestion, runtime, freshness
        v
  [ raw / source layer ]    warehouse.raw.*        <-- DE handoff:
        |                                              landed + fresh
        |  AE owns from here down -------------------------------
        v
  [ staging ]               stg_* : typed, renamed, cleaned
        |
        v
  [ intermediate ]          int_* : joins, reusable logic
        |
        v
  [ marts ]                 dim_* / fct_* : business entities, TESTED
        |
        v
  [ semantic / metrics ]    metrics defined ONCE (revenue, MAU, churn)
        |  AE handoff: one definition per metric
        |  DA owns from here -----------------------------------
        v
  [ dashboards / analysis ] Looker/Tableau/Mode -> insight

Handoff contracts:
  DE -> AE : raw is fresh (< SLO) and schema is registered.
  AE -> DA : marts are tested; metrics have one definition.
  Break a contract -> the downstream layer breaks; contracts
  make the blame boundary unambiguous.

Step-by-step explanation.

1. The map makes the DE→AE handoff explicit: the DE owns everything down to the raw layer and guarantees it is landed and fresh. Everything below raw is the AE's territory. This single horizontal line resolves most DE-vs-AE disputes.
2. The staging → intermediate → marts progression is the AE's dbt DAG. Staging types and renames; intermediate holds reusable join logic; marts are the tested business entities (dim_customer, fct_orders). The AE owns this entire chain and its tests.
3. The marts → semantic handoff is where metrics get defined once. Instead of each dashboard computing revenue its own way, the semantic layer defines revenue a single time and every consumer references it. This is the AE's highest-leverage responsibility.
4. The semantic → dashboards handoff passes ownership to the analyst, with the guarantee that the numbers are consistent everywhere because they come from one definition. The analyst builds insight on trusted models rather than re-deriving metrics.
5. The handoff contracts make the blame boundary unambiguous. If a dashboard is wrong, you walk the map: is raw stale (DE), is a mart failing tests (AE), or is the dashboard mis-querying (DA)? The layered map turns "the data is wrong" into a specific, ownable failure.

Output.

Failure	Where on the map	Owner
Dashboard shows stale data	raw layer not fresh	DE
dim_customer has duplicates	marts failing tests	AE
Two dashboards disagree on revenue	no single semantic definition	AE
Dashboard filters wrong	dashboard query	DA
ML model fed raw instead of marts	DS bypassing the map	DS

Rule of thumb. Draw the layered-ownership map as a flow with explicit handoff contracts, and every "the data is wrong" incident becomes a walk down the map to a single owner. The two handoffs that carry the most weight are DE→AE (raw is fresh and registered) and AE→DA (marts are tested and metrics are defined once); enforce those two and the middle takes care of itself.

Worked example — a tested dbt model with a semantic definition

Detailed explanation. The clearest way to show what an AE owns is the artifact itself: a dbt model that transforms raw data into a tested business entity, plus the semantic definition of a metric built on it. This is the AE's day job made concrete. Walk through a fct_orders model and a revenue metric.

• Model. Transform raw.orders into a clean, tested fct_orders.
• Tests. Uniqueness, not-null, referential integrity.
• Metric. Define revenue once, on top of fct_orders.

Question. Write the dbt model, its tests, and the semantic metric definition an AE owns.

Input.

Artifact	Purpose
fct_orders.sql	clean, business-ready orders fact
fct_orders.yml	tests + docs (the contract)
revenue metric	single definition of revenue

Code.

-- models/marts/fct_orders.sql  (owned by the analytics engineer)
with source as (
    select * from {{ ref('stg_orders') }}      -- staging, not raw
),

cleaned as (
    select
        order_id,
        customer_id,
        status,
        total_cents / 100.0            as revenue_usd,   -- business unit
        ordered_at::date               as order_date
    from source
    where status != 'test'                              -- exclude test orders
)

select * from cleaned

# models/marts/fct_orders.yml — the tests ARE the contract
models:
  - name: fct_orders
    description: "One row per real customer order. Canonical orders fact."
    columns:
      - name: order_id
        tests: [unique, not_null]        # no duplicate orders
      - name: customer_id
        tests:
          - not_null
          - relationships:               # referential integrity
              to: ref('dim_customer')
              field: customer_id

# semantic layer: revenue defined ONCE for every consumer
metrics:
  - name: revenue
    label: "Revenue (USD)"
    model: ref('fct_orders')
    calculation_method: sum
    expression: revenue_usd
    dimensions: [order_date, status]

Step-by-step explanation.

1. The model reads from stg_orders (staging), not raw.orders. This encodes the layering: the DE lands raw, the AE's staging layer cleans and types it, and the mart builds business entities on staging. The AE never reaches into raw directly from a mart.
2. The transformation applies business logic the analyst should not have to re-derive: converting cents to dollars, excluding test orders, and casting to a date. Doing this once in the mart means every downstream consumer gets consistent revenue_usd, not fifteen slightly different conversions.
3. The .yml tests are the contract, enforced in dbt's CI. unique + not_null on order_id guarantees no double-counted orders; the relationships test guarantees every order references a real customer. A change that breaks these fails the build — the same contract-enforcement pattern the platform uses, applied at the model level.
4. The metrics block defines revenue exactly once, as sum(revenue_usd) over fct_orders. Every BI tool and every analyst references this metric rather than writing their own SUM, which is how the semantic layer guarantees consistent numbers everywhere.
5. This single pair of files is the AE role in miniature: engineering rigor (version control, tests, CI, modularity via ref) applied to analytics, producing a tested business entity and a single-definition metric that the analyst builds insight on and the DE never has to touch.

Output.

Consumer	What they get
Analyst in Looker	references revenue metric — consistent by construction
Another AE's model	ref('fct_orders') — reuses the tested fact
dbt CI	fails build if order_id dupes or FK breaks
Data scientist	clean fct_orders as an ML input, not raw
Auditor	docs + tests describe the contract

Rule of thumb. The AE's deliverable is a tested, documented, single-definition model — not a query. If your metrics live as ad-hoc SUMs in dashboards instead of as versioned semantic definitions on tested marts, you do not have an analytics-engineering layer; you have analysts re-deriving numbers and the inconsistency that follows. Define each metric once, test the model under it, and let CI enforce the contract.

Senior interview question on the analytics engineer boundary

A senior interviewer might ask: "Your company has data engineers and data analysts but no analytics engineers. Symptoms: DEs are swamped writing business transformations they do not have context for, analysts maintain a tangle of untested SQL that produces three different revenue numbers, and nobody owns the semantic layer. Make the case for introducing the analytics-engineering function, define its boundary with DEs and analysts, and describe how you would stand it up and who owns the semantic layer."

Solution Using a defined AE function, a role-boundary contract, and an AE-owned semantic layer

Where AE fits (the missing middle)
==================================

  Data Engineer      -> ingestion, runtime, raw/source layer, infra
        | handoff: raw is fresh + registered
  Analytics Engineer -> staging -> marts (dbt, TESTED) + semantic layer
        | handoff: marts tested; every metric defined once
  Data Analyst       -> dashboards, exploration, insight

Problem today: no AE -> the middle is done by whoever is free
  -> DEs write marts without domain context (swamped + wrong)
  -> analysts write untested SQL (3 revenue numbers)
  -> semantic layer is nobody's -> inconsistency by default

## Stand-up plan for the AE function

### Case
The transformation + semantic middle is currently orphaned.
DEs and analysts are both doing it badly because it is neither's
core role. An AE function owns it with engineering rigor.

### How to stand it up
1. Reskill/hire: promote SQL-strong analysts into AE roles or
   hire AEs; they need SQL + dbt + business context.
2. Move all business transformations OUT of DE pipelines and
   OUT of analyst scripts INTO a dbt project the AEs own.
3. Add tests (unique, not_null, relationships, freshness) to
   every mart -> the tests are the contract, enforced in CI.
4. Declare the semantic layer AE-owned, as code, in the dbt repo.
   Define `revenue` ONCE. All BI tools reference it.

### Boundary
- DE stops at raw/source. AE owns staging onward + semantics.
- Analysts do analysis on AE models; they do NOT maintain
  production models.

### Who owns the semantic layer
- The analytics engineer, as versioned + tested code alongside
  the dbt models. NOT a BI-tool setting, NOT the DE.

Step-by-step trace.

Concern	Before (no AE)	After (AE function)
Business transformation	DEs (no context) + analysts (no rigor)	AEs (context + rigor) in dbt
Testing	none on transformations	unique/not_null/relationships in CI
Revenue definition	3 different numbers	1 semantic definition
DE workload	swamped with marts	back to pipelines + infra
Analyst workload	maintaining SQL tangle	analysis on trusted models
Semantic layer owner	nobody	AE, as code

After introducing the AE function, business transformations move into a tested dbt project the AEs own; DEs return to pipelines and infrastructure where their skills fit; analysts stop maintaining production SQL and do analysis on trusted models; and the semantic layer gets a single owner, collapsing three revenue numbers into one contract-enforced definition. The boundary is unambiguous: DE owns raw and below, AE owns transformation and semantics, analyst owns insight.

Output:

Metric	Before	After
Revenue definitions	3	1 (semantic layer)
Tested transformations	0%	high (CI-enforced)
DE time on business logic	high	low (back to platform)
Analyst time maintaining models	high	low (analysis only)
Semantic-layer ownership	none	AE, versioned + tested

Why this works — concept by concept:

• The missing middle — the AE fills the transformation + semantic gap that neither DEs (lack domain context) nor analysts (lack engineering rigor) own well. Naming the role assigns the orphaned middle to someone whose job it is.
• Engineering rigor on analytics — version control, tests, CI, and modularity turn the untested-SQL tangle into a tested dbt project; the tests are the contract that stops silent regressions and the three-revenue-numbers problem.
• AE-owned semantic layer as code — defining each metric once, in the repo, under CI, is what makes numbers consistent across every tool; putting it in a BI-tool setting or on the DE recreates the inconsistency.
• Clean role boundary — DE stops at raw, AE owns the middle, analyst owns insight; the boundary frees DEs to return to infrastructure and analysts to return to analysis, so every role does what it is best at.
• Cost — reskilling analysts or hiring AEs, plus standing up a dbt project with CI. The eliminated cost is O(dashboards) re-derived metrics, swamped DEs doing low-context transformation, and the recurring firefighting of inconsistent numbers. Net: transformation cost is paid once per model instead of re-paid per consumer.

Dimensional Modeling
Topic — dimensional-modeling
Dimensional modeling problems on facts, dims, and marts

Practice →

SQL Topic — sql SQL problems on transformation and semantic-layer logic

Practice →

---

5. Scaling the org: topologies, data mesh and anti-patterns

Scale a data org with Team-Topologies mapping and deliberate data-mesh adoption — and fix the anti-patterns before they calcify

The mental model in one line: scaling a data team org past ~40 people is a topology problem — you map the data function onto the four Team-Topologies team types (stream-aligned, platform, enabling, complicated-subsystem), decide deliberately whether and how far to adopt data mesh (domain-owned data products with federated governance), size the org with sane headcount ratios (platform : embedded : analytics engineers), and actively remediate the anti-patterns (central bottleneck, data silos, ungoverned mesh, hero dependency) before they harden into permanent structure. There is no single right topology — there is the topology that matches your scale, your domain complexity, and your governance maturity, plus a discipline of catching anti-patterns while they are still cheap to fix.

Team Topologies mapped onto data.

• Stream-aligned teams. The domain/product squads with embedded DEs and AEs, aligned to a flow of business value (checkout, growth, logistics). They own their data products end to end and are the primary delivery teams.
• Platform teams. The central data platform team providing the paved road as an internal product, so stream-aligned teams self-serve. Reduces the cognitive load on domain teams.
• Enabling teams. A small, time-boxed team that helps stream-aligned teams adopt new capabilities (a new semantic layer, a migration to dbt), then steps back. Enabling teams coach; they do not do the work permanently.
• Complicated-subsystem teams. A specialist team for a genuinely hard shared subsystem (a real-time streaming platform, an ML feature store, a complex entity-resolution service) that needs deep expertise most domains lack.
• The interaction modes. Team Topologies also defines how teams interact: collaboration (temporary, close), X-as-a-Service (the platform model — clean interface, low coupling), and facilitating (enabling teams coaching). The platform-to-stream relationship should be X-as-a-Service; anything else means the platform is a bottleneck.

Data mesh — the four principles and the honest trade-offs.

• Domain ownership. Each domain owns its data as a product, end to end. Powerful at scale; premature below ~40–50 data people because you lack the domains and the platform maturity to support it.
• Data as a product. Domains publish discoverable, trustworthy, documented data products with SLAs — not raw dumps. This is the principle even non-mesh orgs should adopt.
• Self-serve data platform. Domains cannot own products without a strong platform to build on — mesh requires a mature platform team first. Mesh without a platform is just silos with a rebrand.
• Federated computational governance. Global standards (interoperability, security, canonical entities) enforced computationally (contracts, CI) while domains retain autonomy. This is the hardest principle and the one that separates a real mesh from an ungoverned sprawl.
• The honest trade-off. Data mesh trades central control for domain autonomy and scalability. It pays off at high scale with many domains and a mature platform; it is an expensive mistake at small scale, where it fragments a team that centralization would serve better.

Headcount ratios — sane starting points, not laws.

• Platform : embedded engineers. Roughly 1 platform engineer per 4–6 embedded engineers is a common starting ratio — enough platform to keep the road paved without over-centralizing.
• DE : AE. Often 1:1 to 2:1 embedded DE to AE in analytics-heavy domains; DE-heavy in infrastructure-heavy domains. The AE count grows as the transformation/semantic surface grows.
• Engineers : analysts. Highly variable; a company with many business stakeholders runs more analysts per engineer. The AE layer is what lets a few engineers support many analysts.
• The caveat. Ratios are diagnostics, not targets. A platform team that is 1:20 is under-invested (bottleneck incoming); 1:2 is over-invested (re-centralizing). Use the ratio to spot drift, not to set headcount mechanically.

The anti-pattern catalogue — and the fix for each.

• Central bottleneck. Everything queues behind the central team. Fix: self-serve paved road + embedding. (Section 2 and 3.)
• Data silos. Embedded teams diverge, no shared substrate. Fix: platform + guild + contracts. (Section 3.)
• Ungoverned mesh. Domain ownership without federated governance → a thousand undiscoverable data products. Fix: federated computational governance (catalog + contracts + interoperability standards) before scaling domains.
• Hero dependency. One person holds critical undocumented knowledge; the org grinds when they are out. Fix: documentation, on-call rotation, pairing, and bus-factor > 1 as a standard.
• Premature mesh. Adopting mesh at 20 people. Fix: centralize or hybrid until you have the domains and platform maturity mesh requires.
• Platform ivory tower. Platform builds what it finds interesting, not what domains need. Fix: platform-as-product with user research and adoption metrics.

Common interview probes on scaling.

• "How would you map a data org onto Team Topologies?" — stream-aligned domains, platform for the road, enabling teams to coach, complicated-subsystem for hard specialties.
• "When should we adopt data mesh?" — when you have many domains, a mature platform, and can afford federated governance; not before.
• "What are the ideal data-team ratios?" — ~1 platform per 4–6 embedded; DE:AE 1:1–2:1; use as diagnostics not targets.
• "How do you fix a data team hitting a scaling wall?" — diagnose the anti-pattern, map to topology, remediate structurally.

Worked example — mapping a data org onto Team Topologies

Detailed explanation. The most senior scaling artifact is a Team-Topologies map of the data function: which teams are stream-aligned, which is the platform, where an enabling team is needed, and which specialist subsystem warrants its own team — plus the interaction modes between them. Walk through mapping a 60-person data org.

• Teams. Six domain squads, one platform team, one streaming specialist team, plus a migration need.
• Types. Assign each to a topology type.
• Interactions. Define how they interact (X-as-a-Service, collaboration, facilitating).

Question. Map the 60-person data org onto Team Topologies and specify the interaction modes.

Input.

Team	Topology type	Interaction with others
6 domain squads (DE+AE embedded)	stream-aligned	consume platform X-as-a-Service
Data platform team	platform	X-as-a-Service to domains
Streaming/real-time team	complicated-subsystem	X-as-a-Service + collaboration when needed
dbt-migration team (temporary)	enabling	facilitating, then dissolves

Code.

Team-Topologies map: 60-person data org
========================================

  [ Enabling: dbt-migration squad ]  (time-boxed)
        : facilitating (coaches domains onto dbt, then dissolves)
        v
  [ Stream-aligned: 6 domain squads ]  <-- primary delivery
     checkout | growth | logistics | payments | support | catalog
        ^                     ^
        | X-as-a-Service      | X-as-a-Service (+collab when hard)
        |                     |
  [ Platform: data platform ] [ Complicated-subsystem: streaming ]
   paved road, catalog,        real-time / feature store —
   contracts, semantic layer   deep expertise domains lack

Interaction rules:
  - Platform -> domains: X-as-a-Service ONLY. If it turns into
    collaboration/ticketing, the platform is becoming a bottleneck.
  - Streaming team -> domains: X-as-a-Service, with short bursts of
    collaboration for a new hard use case.
  - Enabling team: facilitating, explicitly time-boxed, dissolves
    after the migration.

Step-by-step explanation.

1. The six domain squads are stream-aligned teams — the primary delivery units, each aligned to a flow of business value with embedded DEs and AEs. In Team Topologies, most teams should be stream-aligned; the others exist to support them.
2. The data platform team is a platform team whose entire job is to reduce the domains' cognitive load via X-as-a-Service. The critical rule: the interaction mode must be X-as-a-Service (clean self-serve interface), not collaboration. If domains have to collaborate closely with the platform for routine work, the platform has become a bottleneck.
3. The streaming/real-time team is a complicated-subsystem team — a genuinely hard specialty (exactly-once streaming, feature stores) that most domains cannot staff. It exists because forcing every domain to master streaming would be wasteful; centralizing the hard part is correct.
4. The dbt-migration team is an enabling team, explicitly time-boxed. It coaches domains onto dbt (facilitating interaction) and then dissolves. The anti-pattern is an enabling team that becomes permanent — that means it started doing the work instead of teaching it.
5. The interaction modes are the real content of the map. X-as-a-Service from platform to domains is the target state; collaboration should be temporary and intentional; facilitating is for enabling teams. Getting the interaction mode wrong (e.g. platform stuck in collaboration) is how a correctly-typed team still becomes a bottleneck.

Output.

Team	Type	Interaction	Health check
Domain squads	stream-aligned	consume platform as a service	shipping without waiting?
Platform	platform	X-as-a-Service	self-serve rate high?
Streaming	complicated-subsystem	X-as-a-Service + collab	domains not rebuilding streaming?
dbt-migration	enabling	facilitating (time-boxed)	on track to dissolve?

Rule of thumb. Map your data org onto the four Team-Topologies types, but spend your attention on the interaction modes, not the labels. The single most important rule is that platform-to-domain must be X-as-a-Service; the moment it becomes collaboration or ticketing, a correctly-structured platform team has silently become the bottleneck you built it to prevent.

Worked example — a headcount-ratio staffing model

Detailed explanation. When leadership asks "how many people do we need and in what mix," the answer is a staffing model built from ratios and domain count, not a gut number. It makes the ask defensible and surfaces where the org is under- or over-invested. Walk through building one for a company scaling from 6 to 10 domains.

• Inputs. Number of domains, complexity per domain, target platform ratio.
• Ratios. Platform : embedded, DE : AE, engineers : analysts.
• Output. A headcount plan with a rationale per line.

Question. Build a staffing model for a 10-domain org using sane ratios and flag any imbalance.

Input.

Role	Rule of thumb	10-domain sizing
Embedded DE	1–2 per domain	~12 (heavier domains get 2)
Analytics engineer	~1 per domain (more if analytics-heavy)	~10
Platform engineer	1 per 4–6 embedded	~4
Analysts	domain-driven	~15 (business-facing)
Specialist (streaming/ML)	1 shared team	~4

Code.

# Staffing model from ratios + domain count (defensible headcount ask)
domains          = 10
avg_de_per_dom   = 1.2       # some domains need 2
ae_per_domain    = 1.0
platform_ratio   = 5         # 1 platform eng per 5 embedded eng
analysts         = 15        # business-driven, not ratio-driven
specialist_team  = 4         # streaming + ML feature store

embedded_de = round(domains * avg_de_per_dom)      # 12
embedded_ae = round(domains * ae_per_domain)       # 10
embedded    = embedded_de + embedded_ae            # 22
platform    = round(embedded / platform_ratio)     # ~4

print(f"embedded DE : {embedded_de}")
print(f"analytics eng: {embedded_ae}")
print(f"platform    : {platform}   (ratio 1:{platform_ratio})")
print(f"analysts    : {analysts}")
print(f"specialists : {specialist_team}")
print(f"TOTAL       : {embedded_de+embedded_ae+platform+analysts+specialist_team}")

# Imbalance check
if platform < embedded / 8:
    print("WARN: platform under-invested -> bottleneck risk")
if platform > embedded / 3:
    print("WARN: platform over-invested -> re-centralizing risk")

Step-by-step explanation.

1. The model starts from domain count and complexity, not a target headcount. Ten domains at ~1.2 DEs each gives ~12 embedded DEs; heavier domains (payments, streaming-adjacent) justify two while lighter ones share. Sizing from domains makes the number defensible.
2. Analytics engineers are sized ~1 per domain, scaled up where the domain is analytics-heavy (growth, finance). The AE count tracks the transformation and semantic surface area, which is what lets a modest analyst count serve many stakeholders.
3. The platform team is derived from the embedded count via the 1:5 ratio — ~4 platform engineers for ~22 embedded. This keeps the road paved without over-centralizing. The ratio is the control that prevents both under- and over-investment.
4. Analysts are sized by business demand, not by a ratio to engineers, because analyst need is driven by the number of business questions and stakeholders, not by pipeline count. The AE layer is what keeps this number from exploding.
5. The imbalance checks turn the model into a diagnostic. If the platform falls below 1:8 of embedded, the code warns of bottleneck risk; above 1:3, it warns of re-centralizing. This is how the ratio is used correctly — to detect drift, not to mechanically set headcount.

Output.

Role	Count	Rationale
Embedded DE	12	~1.2 per domain
Analytics engineer	10	~1 per domain
Platform engineer	4	1:5 of embedded
Analysts	15	business demand
Specialists	4	shared streaming/ML
Total	45	defensible from ratios

Rule of thumb. Build the staffing ask from domain count and ratios, then use the ratios as a drift alarm, not a target. A platform team below ~1:8 of embedded engineers is a bottleneck forming; above ~1:3 it is re-centralizing. Size analysts from business demand and let the analytics-engineering layer be the multiplier that keeps that number sane.

Worked example — anti-pattern remediation

Detailed explanation. The final scaling skill is recognizing an anti-pattern from its symptoms and applying the structural fix rather than a band-aid. Each anti-pattern has a signature and a known remedy. Walk through diagnosing and fixing four common ones in a scaling org.

• Signatures. The observable symptoms of each anti-pattern.
• Root cause. The structural gap behind each.
• Fix. The remediation, plus what not to do.

Question. Diagnose four anti-patterns from their symptoms and prescribe the structural fix for each.

Input.

Anti-pattern	Signature symptom	Structural fix
Central bottleneck	everything behind one queue	self-serve road + embedding
Data silos	N stacks, divergent metrics	platform + guild + contracts
Ungoverned mesh	1000 undiscoverable products	federated computational governance
Hero dependency	org stalls when X is on leave	docs + rotation + bus factor > 1

Code.

Anti-pattern remediation runbook
================================

1) CENTRAL BOTTLENECK
   Symptom : lead time = weeks; central team backlog grows.
   Root    : services-desk platform, no self-serve.
   Fix     : paved-road self-serve + embed into domains.
   NOT     : add headcount to the queue (queue returns).

2) DATA SILOS
   Symptom : 6 orchestrators, 3 revenue definitions.
   Root    : embedding without platform/guild/contracts.
   Fix     : platform owns tooling; guild owns standards;
             canonical data products with contract-CI.
   NOT     : re-centralize everything (loses domain speed).

3) UNGOVERNED MESH
   Symptom : hundreds of data products, none discoverable,
             no interop, duplicated entities.
   Root    : domain ownership WITHOUT federated governance.
   Fix     : catalog + interop standards + contract-CI +
             canonical shared entities BEFORE more domains.
   NOT     : abandon mesh and re-centralize (throws away scale).

4) HERO DEPENDENCY
   Symptom : critical pipeline only one person understands.
   Root    : no docs, no rotation, bus factor = 1.
   Fix     : document, pair, on-call rotation, ownership in the
             catalog, bus factor > 1 as a standard.
   NOT     : give the hero a retention bonus and hope.

Step-by-step explanation.

1. The central bottleneck's signature is week-long lead times and a growing central backlog. The root is a services-desk operating model; the fix is a self-serve paved road plus embedding. The wrong move — adding headcount to the queue — treats the symptom, and the queue regrows to the new capacity.
2. Data silos show up as multiple orchestrators and divergent metric definitions. The root is embedding without the coordinating structures. The fix is platform-owned tooling, a guild for standards, and canonical data products with contract CI. The wrong move is full re-centralization, which sacrifices the domain speed embedding bought.
3. Ungoverned mesh is the failure specific to premature or under-governed mesh: hundreds of undiscoverable, non-interoperable data products. The root is domain ownership without federated computational governance. The fix is a catalog, interoperability standards, and contract CI before scaling more domains. The wrong move is abandoning mesh entirely, throwing away the scalability it offers.
4. Hero dependency is a bus-factor-of-one problem: a critical pipeline only one person understands. The root is missing documentation, rotation, and shared ownership. The fix is documentation, pairing, on-call rotation, and cataloged ownership so bus factor exceeds one. The wrong move is a retention bonus, which pays to preserve the risk instead of removing it.
5. Across all four, the pattern is identical: match the symptom to the anti-pattern, fix the structural root, and refuse the band-aid that treats the symptom. Anti-patterns caught early are cheap structural adjustments; caught late they are calcified reorgs.

Output.

Anti-pattern	Band-aid (wrong)	Structural fix (right)
Central bottleneck	more headcount	self-serve + embed
Data silos	re-centralize	platform + guild + contracts
Ungoverned mesh	abandon mesh	federated governance first
Hero dependency	retention bonus	docs + rotation + bus factor > 1

Rule of thumb. Every data-org anti-pattern has a signature symptom, a structural root, and a band-aid that will tempt you. Diagnose to the root, apply the structural fix, and refuse the band-aid — more headcount for a bottleneck, re-centralization for silos, or a retention bonus for a hero all treat symptoms while the root calcifies. Catch anti-patterns while they are still cheap structural adjustments.

Senior interview question on scaling a data team

A senior interviewer might ask: "Your data org has grown to 70 people organized as one big central team plus a few analysts scattered in business units. Delivery has slowed to a crawl, the central team is a bottleneck, business units are hiring shadow analysts and building their own pipelines, and the CTO is asking whether you should 'do data mesh.' Design the target topology, decide how far to go toward data mesh, give a staffing model, and sequence the transition without a big-bang reorg."

Solution Using a Team-Topologies target, deliberate partial mesh, a ratio-based staffing model, and an incremental transition

Target topology (Team Topologies)
=================================
  Stream-aligned : domain squads (embedded DE+AE), aligned to
                   business value; own their data products.
  Platform       : central data platform as X-as-a-Service (road).
  Enabling       : time-boxed team to coach domains onto the road.
  Complicated-   : streaming/ML feature-store specialists.
  subsystem

How far toward data mesh:
  - Adopt "data as a product" + "self-serve platform" NOW.
  - Adopt "domain ownership" INCREMENTALLY (domain by domain).
  - Adopt "federated computational governance" IN LOCKSTEP with
    domain ownership (catalog + contracts + interop standards).
  - This is a PARTIAL, deliberate mesh — not a big-bang.

## Staffing model (70 people, ~8 domains)
- Platform: 8   (1 : ~5 embedded)
- Embedded DE: 12, Analytics eng: 10   (~22 embedded)
- Specialists (streaming/ML): 5
- Analysts: 18 (business-driven, folded IN from shadow hires)
- Enabling: 3 (time-boxed) + leadership/leads: rest

## Transition (no big-bang; 3 quarters)
Q1 — Pave + legitimize
  - Carve platform team from central; ship self-serve road + catalog.
  - Bring shadow analysts into the org (fold, don't fight) with a
    dotted line to the guild.
Q2 — Embed + productize
  - Embed DE+AE into top-3 domains; migrate them onto the road.
  - Declare canonical entities (customer, revenue) with contract-CI.
  - Enabling team coaches each domain, then rotates on.
Q3 — Federate + govern
  - Extend domain ownership to remaining domains (partial mesh).
  - Turn on federated governance: interop standards, catalog
    coverage SLO, contract-CI everywhere. Enabling team dissolves.

Step-by-step trace.

Concern	Before	After
Structure	one central team + scattered analysts	stream-aligned domains + platform + enabling + specialists
Delivery	bottlenecked	self-serve road + embedded pods
Shadow IT	business units building pipelines	folded into domains on the road
Mesh stance	"should we do mesh?"	deliberate partial mesh, governance in lockstep
Governance	central + overloaded	federated computational (catalog + contracts)
Transition	risk of big-bang reorg	incremental, 3-quarter sequence

After the transition, the central bottleneck is replaced by stream-aligned domain squads building on a platform delivered X-as-a-Service; the shadow analysts are legitimized inside domains rather than fought; and data mesh is adopted deliberately and partially — "data as a product" and "self-serve platform" immediately, "domain ownership" and "federated governance" together, domain by domain, never as a big-bang. Governance scales computationally through the catalog and contract CI rather than through an overloaded central review.

Output:

Metric	Before	After
Delivery lead time	weeks	days
Shadow pipelines	proliferating	folded onto the road
Governance model	central, overloaded	federated + computational
Mesh adoption	undecided / risky	deliberate, incremental, governed
Reorg risk	big-bang	3-quarter incremental

Why this works — concept by concept:

• Team-Topologies target — mapping to stream-aligned + platform + enabling + complicated-subsystem gives each team a clear type and interaction mode; platform-as-X-as-a-Service is what breaks the bottleneck structurally.
• Deliberate partial mesh — adopting data-as-a-product and self-serve now, but domain ownership and federated governance in lockstep and incrementally, captures mesh's scalability while avoiding the ungoverned-mesh anti-pattern that premature adoption causes.
• Fold the shadow IT — legitimizing shadow analysts inside domains on the paved road converts a governance problem into embedded capacity, instead of fighting a losing battle against local demand.
• Ratio-based staffing — sizing platform at ~1:5 of embedded and analysts from business demand makes the headcount ask defensible and keeps the platform from becoming either a bottleneck or an ivory tower.
• Incremental transition — a 3-quarter, domain-by-domain sequence with a time-boxed enabling team avoids the big-bang reorg that stalls delivery for two quarters; each domain is migrated and governed before the next starts.
• Cost — a platform team, an enabling team (time-boxed), and the governance substrate (catalog + contract CI). The eliminated cost is the compounding O(domains²) reconciliation of an ungoverned mesh, the perpetual central-queue tax, and the shadow-IT duplication. Net: coordination and governance cost grow near-linearly with domains under federated computational governance, instead of exploding.

Design
Topic — design
Design problems on scaling data platforms and mesh

Practice →

Streaming
Topic — streaming
Streaming problems for the complicated-subsystem team

Practice →

---

Cheat sheet — data-team-structure recipes

• Three archetypes, one decision rule. Centralized = strong standards, bottleneck failure mode. Decentralized/embedded = fast + domain-accurate, silo failure mode. Hybrid (platform + embedded) = paved road plus domain speed, fails only if the platform re-centralizes. Diagnose your current failure mode first, then pick the archetype whose failure mode you can afford. Never copy another company's org chart.
• Conway's Law is the mechanism. Your pipelines, schemas, and ownership boundaries will mirror your communication structure. To get one canonical customer product, create one owning team plus a versioned, test-enforced contract and a real consumer channel. If you cannot name the single owner, you have a fork waiting to happen.
• Platform-as-product checklist. Users are engineers; golden path over bespoke; roadmap not backlog; self-serve is the goal; publish SLAs/SLOs with the query that measures them; run on-call for the substrate; state an explicit anti-goal ("we are NOT a request desk"). Measure success by adoption and self-serve rate, never tickets closed.
• Paved-road capability map. For each capability (ingestion, orchestration, catalog, quality, semantic layer) publish the golden path, the self-serve status, and an SLO. No golden path → domains silo; no SLO → domains cannot plan. Deliver governance as guardrails ON the road (auto-PII masking, contract CI), so the compliant path is the easy path.
• Data contract + CI gate. Give the platform a data-contract primitive (schema + semantics + quality) enforced by a CI gate that fails breaking-change PRs. This scales governance at the speed of every pipeline instead of a central review queue. Breaking changes ship as a new version with a deprecation window, never in place.
• Embedding: solid line + dotted line. Embedded DEs/AEs report solid line to the domain (delivery, accountability) and dotted line to a guild (standards, craft, career). Solid-line-only → silos; dotted-line-only → outsiders with no domain context. The two RACI rows that matter most: tooling (platform Accountable, stops fragmentation) and career (guild + squad shared, stops attrition).
• Guild charter, not a status meeting. A real guild owns standards, an RFC process, weekly design review, a quarterly standards refresh, and an in-place staff/principal ladder. The career ladder is the retention mechanism; the refresh is the anti-rot mechanism. Never embed a lone engineer into a domain with no paved road to build on.
• AE = the tested middle. The analytics engineer owns transformation (dbt staging → marts) and the semantic layer as versioned, tested code between DE pipelines and analyst dashboards. DE stops at raw; AE owns staging onward; analysts do analysis on AE models, not production SQL. If metrics disagree across dashboards, you have no single semantic owner.
• Role-boundary matrix. One owner per stack layer: DE = ingestion/raw/infra, AE = transformation + semantic layer, DA = dashboards/insight, DS = ML features. Contested layers default to AE (transformation and semantic layer). Smell tests: analysts maintaining prod models = AE under-resourced; DEs writing marts = AE role missing; three revenue numbers = no semantic owner.
• Team Topologies mapping. Stream-aligned domain squads (primary delivery), platform team (X-as-a-Service road), enabling team (time-boxed coaching, then dissolves), complicated-subsystem team (streaming/ML specialists). Spend your attention on interaction modes: platform → domain must be X-as-a-Service; the moment it becomes collaboration/ticketing, the platform is the bottleneck.
• Data mesh readiness test. Adopt "data as a product" and "self-serve platform" early; adopt "domain ownership" and "federated computational governance" together, incrementally, only when you have many domains and a mature platform. Mesh without a platform is silos with a rebrand; domain ownership without federated governance is the ungoverned-mesh anti-pattern.
• Headcount ratios (diagnostics, not targets). ~1 platform engineer per 4–6 embedded; DE:AE roughly 1:1 to 2:1; analysts sized by business demand with the AE layer as multiplier. Below ~1:8 platform-to-embedded = bottleneck forming; above ~1:3 = re-centralizing. Build the ask from domain count and ratios; use the ratio to spot drift.
• Anti-pattern → structural fix. Central bottleneck → self-serve + embed (not more headcount). Silos → platform + guild + contracts (not re-centralize). Ungoverned mesh → federated governance first (not abandon mesh). Hero dependency → docs + rotation + bus factor > 1 (not a retention bonus). Premature mesh → hybrid until mature. Always fix the root, refuse the band-aid.

Frequently asked questions

What is a data platform team?

A data platform team is the central group that owns the shared data substrate — ingestion, orchestration, storage, catalog, governance, and often the semantic layer — and provides it to the rest of the organization as a self-serve internal product rather than as a services desk. Its users are the domain engineers (embedded DEs and analytics engineers), and its job is to build a paved-road golden path so those teams can self-serve common needs (onboarding a source, scheduling a pipeline, discovering a dataset) instead of queuing behind a central team. A well-run platform team publishes SLAs/SLOs, runs on-call for the substrate, and measures itself by adoption and self-serve rate, not tickets closed. The defining test of a healthy platform team is whether it thinks "product" (build the road once) or "service" (do the work per request) — the latter becomes the org's bottleneck.

Central platform vs embedded data engineers — which is better?

Neither is better in isolation; the strongest data platform team structure at scale is the hybrid where a central platform team owns the shared substrate as a product and embedded data engineers build domain data products on top of it. A purely central data platform gives you strong standards and one warehouse but becomes a bottleneck as demand outpaces headcount and sits far from domain knowledge. Purely embedded data engineers give you fast iteration and deep domain context but drift into silos — divergent tools, duplicated infrastructure, and inconsistent metrics — because nobody owns the shared platform. The hybrid captures both: the platform paves the road (self-serve ingestion, contracts, catalog) and embedded engineers drive on it, reporting solid line to their domain for delivery and dotted line to a guild for standards. The centralized vs decentralized data team debate is really a question of what stage you are at — centralize early, hybridize as you scale.

What is an analytics engineer?

An analytics engineer is the modern-data-stack role that owns the transformation layer between the data engineer's pipelines and the data analyst's dashboards — turning raw, ingested data into clean, tested, documented, business-ready models (typically in dbt) and often owning the semantic/metrics layer on top. The role emerged because dbt and the ELT pattern moved transformation out of the pipeline and into the warehouse, and someone with software-engineering rigor (version control, testing, CI, modularity) and business context had to own that middle. In practice the analytics engineer writes modular SQL models, adds tests (uniqueness, not-null, referential integrity, freshness), documents the models, and defines each metric once in the semantic layer so every downstream tool computes it the same way. The clean boundary: the DE owns raw and below, the AE owns transformation and semantics, and the analyst does analysis on the AE's trusted models.

When should we adopt data mesh?

Adopt data mesh when you have many domains, a mature self-serve platform team, and the ability to enforce federated computational governance — typically past ~40–50 data people, not before. Data mesh has four principles: domain ownership of data as a product, data-as-a-product thinking, a self-serve data platform, and federated computational governance. The mistake is adopting "domain ownership" without the other three: mesh without a strong platform is just silos with a rebrand, and domain ownership without federated governance produces the ungoverned-mesh anti-pattern — hundreds of undiscoverable, non-interoperable data products and duplicated entities. The pragmatic path is a partial, deliberate mesh: adopt data-as-a-product and self-serve platform early, then extend domain ownership incrementally, domain by domain, with federated governance (catalog, interoperability standards, contract CI) turned on in lockstep. Data mesh trades central control for domain autonomy and scalability, which pays off only when your scale and platform maturity can support it.

DE vs AE vs analyst — what's the difference?

The three roles own three consecutive layers of the stack. The data engineer owns ingestion, the pipeline runtime, infrastructure, orchestration, and the raw/source layer landing in the warehouse — their job is trustworthy raw data at scale. The analytics engineer owns the transformation from raw to business-ready (staging → intermediate → marts in dbt), the tests and documentation on those models, and typically the semantic/metrics layer — their job is turning raw data into tested, reusable, single-definition models. The data analyst owns the last mile: exploration, dashboards, ad-hoc analysis, and turning trusted models into insight and recommendations. The clean rule is one owner per layer: DEs stop at raw, AEs own the middle, analysts own insight. Warning signs that the boundary has blurred: DEs writing business marts (AE role missing), analysts maintaining production models (AE layer under-resourced), or three different revenue numbers across dashboards (no single semantic owner).

What are the ideal data-team ratios?

There is no universal ratio, but sane starting diagnostics are roughly 1 platform engineer per 4–6 embedded engineers, an embedded DE:AE ratio of about 1:1 to 2:1 (more AE-heavy in analytics-intensive domains, more DE-heavy in infrastructure-heavy ones), and an engineers-to-analysts ratio driven by business demand rather than by a fixed number — with the analytics-engineering layer acting as the multiplier that lets a modest engineering team support many analysts. Treat these as drift alarms, not targets: a platform team below ~1:8 of embedded engineers is a bottleneck forming, while above ~1:3 it is re-centralizing and over-investing. Build the actual headcount ask from your domain count and complexity — roughly one to two embedded DEs and about one AE per domain, plus a shared specialist team for genuinely hard subsystems (streaming, ML feature stores) — then use the ratios to check the mix is balanced. The right data team org size is derived from domains and ratios, defended as a staffing model, never guessed.

Practice on PipeCode

• Drill the system design practice library → for the data-platform, ownership, and org-topology scenarios that leadership and senior interviews love.
• Rehearse the modelling foundations on the dimensional-modeling practice library → for the facts, dimensions, and marts an analytics engineer owns.
• Sharpen the pipeline axis with the ETL practice library → for the ingestion and orchestration paved-road patterns a central platform team provides.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the archetype, platform-as-product, and AE-boundary decisions against real graded inputs.

Lock in data-org design muscle memory

Docs describe org charts. PipeCode drills teach the decision — when centralization becomes a bottleneck, when embedding drifts into silos, where the analytics-engineer boundary sits, and when data mesh earns its place. Pipecode.ai is Leetcode for Data Engineering — design-first practice tuned for the org and architecture trade-offs senior data engineers and data leaders actually defend.

Practice design problems →
Practice SQL problems →