DEV Community: Sudeep Hazra

To SQL or to Python; there is another problem

Sudeep Hazra — Thu, 22 Oct 2026 18:23:27 +0000

A recent r/dataengineering discussion described a familiar disagreement. A team was moving millions of daily records into Databricks. Python already handled ingestion, but one engineer wanted the silver-layer transformations in Python as well. Other team members preferred SQL because more people could read and maintain it.

The discussion quickly became a vote for a language. SQL is easier to read. Python is easier to test. SQL is declarative. Python is proper engineering. One will scale; the other will not.

Most of those claims are too broad to guide a platform decision.

For structured Spark transformations, SQL and the DataFrame API often describe the same work. The more useful decision is about ownership: who must understand the transformation, what kind of change it receives, how it will be tested, and where its operational boundary belongs.

I would not standardize an entire data platform on one language. I would standardize the rules for choosing.

The execution engine removes one easy argument

Apache Spark's SQL and DataFrames guide makes an important point: SQL and the Dataset or DataFrame APIs use the same execution engine. Spark receives more information about structured data and the computation, then uses that information for optimization.

That does not guarantee identical performance for every pair of implementations. A Python user-defined function can block optimizations that a built-in SQL expression would preserve. A poorly written SQL query can create unnecessary shuffles. Different APIs can expose different features.

It does mean that "Python is faster" and "SQL scales better" are weak starting positions. If two implementations resolve to the same logical and physical plan, the syntax is not the main performance decision.

The team should compare evidence:

representative input
        ↓
SQL implementation + Python implementation
        ↓
logical and physical plans
        ↓
runtime, shuffle, spill, skew, cost
        ↓
maintenance and test review

The benchmark should use production-shaped data. A transformation that looks fine on ten thousand evenly distributed rows can behave very differently when one customer owns a third of the records.

Performance matters. It just has to be measured after the two options perform the same work.

Choose the language from the shape of the transformation

Databricks now documents this choice directly. Its Lakeflow guidance for SQL and Python says that both interfaces produce the same underlying dataflow graph and provide equivalent functionality for most processing. It recommends SQL for logic that can be expressed clearly in SQL, and Python when the pipeline needs programmatic control, an external library, or a Python-only feature.

That is a much better boundary than bronze, silver, and gold.

A layer name tells us how refined the data is supposed to be. It does not tell us whether the implementation needs loops, conditional generation, an external API, a custom library, or a readable set of joins and aggregations.

I would use a decision table like this:

Transformation shape	Default	Reason
Joins, filters, windows, aggregations	SQL	Direct expression of relational logic
Business-facing dimensional models	SQL	Wider reviewer and maintainer pool
Reusable DataFrame algorithm	Python	Functions and modules provide a clean boundary
Dynamic pipeline generation	Python	The logic needs programmatic control
External library or non-tabular processing	Python	SQL is no longer the natural interface
A pipeline containing both shapes	Both	Keep each part in the clearest interface

The last row is easy to overlook. Databricks supports mixing SQL and Python sources in one pipeline. A platform standard does not need to turn a mixed workload into an ideological contest.

The standard can say that relational transformations default to SQL, while Python is available when the transformation crosses a documented threshold. That gives the team consistency without making SQL imitate a programming language or making Python wrap every select, join, and groupBy.

A silver layer is not one kind of work

"Put the silver layer in Python" sounds precise because the layer has a name. In practice, silver often contains several different responsibilities:

parsing source records and handling malformed values
deduplicating events
applying CDC semantics
standardizing identifiers and units
joining reference data
enforcing quality expectations
preparing reusable domain entities

Some of that work is straightforward relational logic. Some of it needs stateful streaming behavior. Some of it depends on a library. Some of it encodes a business definition that an analyst or domain engineer must review.

One language across the layer can make the diagram tidy while making ownership worse.

Suppose the finance team must approve revenue adjustments. A SQL model that exposes the joins and conditions may be easier for that review, even if a data engineer could write the same transformation in PySpark. Now suppose the platform must create fifty similar ingestion flows from metadata. Python can generate those definitions without copying fifty SQL files.

The language changes because the work changes.

I would split the layer by stable responsibilities, then choose the interface for each responsibility:

source-specific parsing       -> Python when libraries or custom parsing are needed
generic row validation        -> SQL expectations or built-in expressions
relational conformance        -> SQL
shared algorithm              -> unit-tested Python module
business model                -> SQL owned with the domain

This also gives code review a useful question. The reviewer is no longer deciding whether they like SQL or Python. They are checking whether the implementation matches the kind of change the component is expected to receive.

Readability belongs to a named reader

Teams often say that one language is more readable without naming the reader.

Readable to whom?

The engineer maintaining ingestion may prefer Python because the surrounding framework, tests, and deployment code are already Python. The analytics engineer validating a customer metric may prefer SQL because the transformation is close to the query they use for reconciliation. The on-call engineer may prefer whichever version exposes input tables, output tables, and failure conditions without three layers of abstraction.

Readability is part of the operating model. I would identify at least four readers for an important transformation:

The author making the first change.
The reviewer checking correctness.
The on-call engineer diagnosing a failed run.
The next owner changing it six months later.

If only the first reader benefits from the chosen language, the team is optimizing for writing rather than ownership.

This is where clever frameworks become expensive. A metadata-driven Python abstraction can remove repeated code, but it can also hide the generated query, input dependencies, and quality rules. The abstraction earns its place when the repeated behavior is stable, the generated plan is observable, and the team can test one definition without running the entire framework.

If every transformation needs an exception in the metadata, the framework has stopped reducing complexity. It has merely moved the complexity into configuration.

Testing is different, but neither language gets a free pass

Python has a familiar unit-testing story. Databricks recommends separating transformation logic into plain PySpark functions so they can be tested locally with pytest; its local pipeline development guide also describes validation and workspace execution as complementary layers.

That is useful, especially for functions with branching logic. It does not make a mocked DataFrame test proof that the production pipeline is correct.

SQL can also be tested with small input relations, expected outputs, schema checks, and integration runs. The syntax is different, but the required evidence is similar:

representative happy-path rows
nulls and malformed values
duplicates and late records
corrections and deletions
schema changes
row-count and value reconciliation
a production execution plan

The team should define these checks once and require them from either interface.

I would add one more rule: test the published table contract separately from the implementation. If a Python transformation is replaced by SQL, consumers should not discover the change through missing columns or altered null behavior. The same applies in the other direction.

Write a small standard, not a language constitution

A workable standard can fit on one page:

Default to SQL for declarative relational transformations.

Use Python for programmatic control, shared algorithms,
external libraries, or features unavailable in SQL.

Mix them when one pipeline contains both kinds of work.

Require the same contract tests, reconciliation, lineage,
and operational metadata from both.

Benchmark performance claims with representative data
and inspect the execution plan.

Then record exceptions with a short decision note. State the transformation shape, intended maintainers, test approach, runtime evidence, and the condition that would trigger reconsideration.

This avoids two bad outcomes. The first is a Python-only platform where ordinary relational logic is hidden behind builders and helper functions. The second is a SQL-only platform where procedural logic leaks into generated statements and fragile templating.

For the Databricks migration in the Reddit discussion, I would keep ingestion in the Python framework that already works. I would default ordinary silver transformations to SQL when they are relational and need broad review. I would keep Python for custom parsing, reusable algorithms, dynamic definitions, and features that SQL does not cover. Both paths would publish the same ownership, test, lineage, and recovery information.

The language decision matters. The team identity attached to it does not.

Pick the interface that makes the transformation easiest to understand, verify, and operate by the people who own it. If that produces a mixed pipeline, that is a reasonable design, not a failure to standardize.

Data platform teams are becoming exception handlers

Sudeep Hazra — Thu, 22 Oct 2026 17:57:35 +0000

An r/dataengineering thread about losing interest in the field described a data platform engineer who knew the full route from source systems to business value. They maintained Snowflake, dbt, Airflow, ingestion code, an internal library, and a collection of BI tools. They were also pulled into dbt failures, model performance problems, and Snowflake troubleshooting.

That breadth sounds valuable. It is also how a platform engineer becomes the human exception handler for an entire data organization.

Every problem that does not have an obvious owner lands with the person who understands the whole system. They fix the pipeline, explain the metric, repair the framework, and help the next team use it. Because they solve the problem, the organization sends them the next one.

The platform becomes more important while its engineers have less time to improve it.

I would treat this as an architecture problem. Team capacity, support boundaries, and ownership are part of the platform design.

The platform sits at every boundary

A data platform connects systems that are owned by different groups:

source application
        ↓
ingestion service
        ↓
storage and processing
        ↓
domain models
        ↓
BI, ML, applications, and agents

When a number is wrong, any arrow could be responsible. The source may have changed its deletion behavior. Ingestion may have skipped a partition. A transformation may have duplicated records. A dashboard may apply another filter. The platform team has visibility across the route, so asking it for help is reasonable.

The problem begins when visibility is confused with ownership.

The platform team should own the shared capabilities it provides: execution environments, deployment paths, observability conventions, access patterns, templates, and supported interfaces. It should not silently inherit every pipeline's business rules, every source system's semantics, and every consumer's interpretation.

Without that distinction, the platform is a shared service in the diagram and a catch-all team in practice.

Support work has to return information to the platform

Google's SRE guidance calls repetitive, manual, service-related work that scales with growth toil. It also warns that excessive toil slows engineering progress, confuses the team's role, and can drive experienced engineers away.

A support request is not automatically toil. Diagnosing a new failure mode can teach the team something important. Helping an early adopter can reveal a missing interface. A production incident may deserve immediate collaboration across every boundary.

The work becomes toil when the same class of issue returns without improving the system.

I would require every recurring support case to take one of four exits:

Exit	Example
Platform fix	The shared retry behavior is wrong
Paved-road improvement	A template needs a default freshness check
Owner runbook	The domain team needs a documented backfill procedure
Explicit exception	A temporary legacy workflow needs named support and an end date

Closing a ticket is not enough. The ticket should leave behind a safer interface, better diagnostic evidence, clearer ownership, or a conscious decision to keep paying the support cost.

Google's chapter on operational overload recommends identifying the sources of stress, separating avoidable toil from legitimate operational work, and using service-level objectives to restore focus. The same approach works for a data platform. Count the interruptions. Group them by cause. Find the few categories consuming most of the time.

The first useful dashboard may be about the platform team rather than the platform:

support requests by producing team
repeated requests by failure class
hours spent on unplanned work
requests missing an owner or runbook
issues converted into platform improvements

If the team cannot see its operational load, it will plan as though every engineer is available for roadmap work. The roadmap will slip, support will feel like an interruption, and the people doing both will carry the discrepancy.

Define a support contract before adding a portal

Platform engineering is often presented as a self-service portal. The portal is the visible part. The important part is the contract behind it.

The CNCF Platforms White Paper describes an internal platform as a collection of capabilities shaped around user needs, with consistent experiences such as APIs, templates, documentation, and support. It also emphasizes that platform teams are responsible for the interfaces and user experience, while the underlying capability may be run by another provider or team.

That separation is useful for data platforms. A support contract can define three areas.

What the platform owns

the supported pipeline runtime and deployment path
authentication and access interfaces
shared monitoring and lineage integration
platform upgrades and compatibility windows
documented recovery controls offered by the platform
defects in templates, libraries, and managed services the team publishes

What the workload owner owns

source and business semantics
transformation correctness
data-quality rules for its domain
schema-change coordination with consumers
backfill approval and validation
first response to alerts for the workload

What requires collaboration

failures that cross several platform services
incidents where the cause is still unknown
capacity changes that affect shared infrastructure
migrations that change a supported interface
security or data-loss events

The collaborative area matters. A boundary should route work, not provide an excuse to throw a ticket back over the wall.

The contract should also name service levels. A production outage, a failing scheduled pipeline, a design consultation, and a request for a new connector do not need the same response time. When everything arrives through a direct message marked urgent, the platform team cannot make a sensible trade-off.

Require evidence at the support boundary

Many data incidents begin with a sentence like "the numbers look wrong." That is a valid symptom, but it is not enough information for efficient triage.

The support interface should collect a small diagnostic envelope:

affected dataset or job
business impact
first known bad time or partition
expected result and observed result
recent deployment or source change
run identifier and relevant logs
checks already performed
named workload owner

This is not bureaucracy for its own sake. It prevents the platform engineer from spending the first hour discovering which table, date, environment, and dashboard the request refers to.

The platform should make the evidence easy to obtain. A job page can expose the run identifier, input versions, row counts, quality results, owner, and runbook. A template can register the support channel and escalation policy when a new pipeline is created. An alert can link directly to the failed partition and recovery action.

Self-service is credible when it includes failure, not only creation.

The happy path may let a team create a pipeline in five minutes. If recovery still requires finding the one platform engineer who remembers a private command, the platform has automated provisioning while preserving dependency.

Route data meaning to the people who know it

Central platform teams often accumulate business logic because they are the only team with the skills or access to change the pipeline. That may be necessary during an early migration. It should not become invisible permanent ownership.

Google Cloud's overview of data mesh describes domain-oriented ownership and a self-service data platform as separate principles. The useful part is not the label. It is the recognition that the people closest to a domain understand its semantics, while a shared platform can remove the infrastructure burden.

Distributed ownership still costs something. Domain teams need training, access, time, and an escalation path. Giving a team a repository and declaring it the owner does not create operational capability.

I would transfer ownership in stages:

The platform team operates the workflow while documenting failure modes.
The domain team joins incident review and validates business outcomes.
The platform provides a runbook, dashboards, and a safe recovery action.
The domain team becomes first responder for its workload.
The platform remains responsible for shared-service defects and difficult cross-boundary failures.

This is slower than changing an owner field. It is also much more likely to survive the first incident.

For critical datasets, record ownership in the same place engineers see operational state. A catalog entry that names a business owner but does not connect to the runbook, alert route, and deployment repository is useful documentation, but it is not an operating model.

Protect enough capacity to improve the platform

Google's SRE model uses a 50% upper bound for operational work as a forcing function, not as a universal staffing formula. A data platform team can choose a different threshold. The important part is to choose and measure one.

If unplanned support consumes most of the team's week, it cannot build the self-service features expected to reduce support. That feedback loop is the dangerous part:

more unsupported workloads
        ↓
more manual help
        ↓
less platform engineering
        ↓
more missing capabilities
        ↓
more manual help

Breaking the loop may require temporarily slowing onboarding. It may require a support rotation so interruptions do not reach every engineer. It may require deleting a rarely used feature whose maintenance cost exceeds its value. It may require one quarter focused on observability and recovery rather than another set of integrations.

I would reserve capacity for the top recurring sources of support and review the balance each month. If the limit is exceeded, leadership should decide what work stops. The answer cannot be an invisible extension of the team's working day.

The platform should remove dependency, not collect it

A strong platform team will still be asked difficult questions. That is part of serving many workloads across shared infrastructure. The goal is not zero support.

The goal is for each support interaction to strengthen the system. A repeated request becomes an API, a template, an alert, a runbook, a training session, or a clearer boundary. Workload owners gain enough context to handle ordinary failures. Platform engineers keep responsibility for the capabilities only they can change.

For the engineer in the Reddit discussion, knowing the entire data route was an asset. The operating model turned that asset into a dependency.

I would fix the return path. Name who owns data meaning, workload operation, and shared infrastructure. Collect evidence at the support boundary. Measure recurring interruption. Protect enough engineering capacity to remove the next class of tickets.

A platform is successful when capable teams can use it without borrowing one of its engineers for every exception.

Your specification is not a code review

Sudeep Hazra — Mon, 12 Oct 2026 10:37:25 +0000

A comment in the September r/dataengineering discussion thread stopped me for a moment. A team had embraced spec-driven development so completely that people were proud of not reading the code their agents produced.

I like specifications. They force a team to describe the behaviour it wants before a coding session starts inventing behaviour on its behalf. GitHub's Spec Kit makes the idea concrete: define intent, plan the work, break it into tasks, implement, and converge on evidence.

A useful specification becomes dangerous when the team treats it as proof that the implementation is correct.

A spec tells us what should happen. The code still decides what can happen, including the error paths, permission boundaries, resource usage, rollback behaviour, and accidental side effects that never made it into the happy-path requirements.

If nobody can explain the resulting change, the team did not remove engineering work. It moved that work to production.

A good spec improves the review

Spec-driven development gives reviewers something better than a vague ticket. A useful spec should name the user-visible behaviour, constraints, acceptance criteria, and conditions that must remain true.

For example, "add bulk retry" is not much of a contract. A better spec would say:

Only failed items are eligible for retry.
A retry must not duplicate a successful write.
The caller may retry the same request after a timeout.
Authorization is checked again when the retry begins.
The operation records which items succeeded, failed, or were skipped.

That list makes the implementation easier to challenge. A reviewer can trace each condition into code and tests. It also exposes questions before code exists. What identifies the same request? How long is that identity retained? What happens when authorization changes between attempts?

The spec does not answer those questions unless someone puts the answers into it. Even then, the implementation may violate them.

GitHub's own agentic SDLC guide is more restrained than the popular interpretation of spec-driven development. It describes agents layered onto a conventional lifecycle, with small changes using the normal issue, pull request, test, and review path. Larger changes may use deeper specification and planning. Code still faces automated checks and maintainer review.

That model keeps the useful part of spec-driven development. The specification sharpens the review. It does not cancel it.

There are four different contracts

I would separate the work into four contracts:

Intent
What behaviour do we want?

Implementation
How does this change create that behaviour?

Verification
What evidence shows that it works and remains safe?

Operation
How will we detect, contain, and recover from failure?

A feature spec mostly owns intent. An implementation plan connects intent to the repository. Tests and analysis provide verification. Runbooks, telemetry, deployment controls, and rollback plans cover operation.

An agent can help produce every artifact in that chain. It cannot make the artifacts equivalent.

A passing unit test may prove that a retry helper returns the expected result for three inputs. It does not prove that two workers cannot process the same item concurrently. A correct implementation may still need a database constraint, idempotency key, or conditional write. Those decisions live below the level of the feature sentence.

This is why reviewing only the final agent summary is risky. The summary is another generated artifact. It can be clear, persuasive, and wrong in exactly the same direction as the code.

Review the repository state: the diff, tests, dependency changes, generated files, migrations, permissions, and runtime configuration. Narration is useful for navigation. It is not evidence.

Make the change small enough to understand

AI makes it cheap to generate a large change. It does not make a large change cheap to review.

A 2,000-line pull request that modifies 100 data models may contain correct code. It is still a poor unit of review if nobody can reason about its blast radius. The problem is not an arbitrary line limit. It is the number of independent claims the reviewer must verify at once.

I would slice work by behaviour and risk:

specification
    ↓
schema or interface change
    ↓
core behaviour
    ↓
integration path
    ↓
operational controls

Each slice should leave the repository in a valid state and produce its own evidence. A database migration can be reviewed before application logic depends on it. A new API can land behind a disabled route or feature flag. Observability can be added with the behaviour rather than appearing in a final "production readiness" pull request that nobody wants to delay.

Small changes also make agent mistakes easier to diagnose. If a test fails after one bounded edit, the search space is manageable. If an agent rewrites a subsystem, updates dependencies, changes infrastructure, and reformats adjacent files in one pass, the team has created a debugging project.

The agent is not the only source of this problem. Humans have always produced oversized pull requests. Faster generation simply removes the natural friction that used to slow us down.

Verification must come from outside the implementation

If the same prompt produces the code, the tests, and the explanation, all three may share the same blind spot.

That does not make agent-written tests useless. It means the acceptance evidence should not depend only on the implementation's own assumptions.

For the bulk retry example, I would require evidence at several levels:

Evidence	What it checks	What it can miss
Unit tests	Local state transitions and edge cases	Database and concurrency behaviour
Integration tests	Persistence, transactions, and authorization	Production scale and timing
Static analysis	Known unsafe patterns and policy violations	Business correctness
Load or fault tests	Contention, timeouts, and retry storms	Rare production combinations
Human review	Design fit, hidden coupling, and maintainability	Errors outside the reviewer's context

No row is sufficient by itself. Together they make a much stronger argument.

NIST's current DevSecOps reference model takes the same position: AI-generated output should pass through peer review, security validation, automated testing, and approval workflows. The model treats generated corrective actions as proposed inputs, not self-approving changes to software or configuration.

Calling the output a proposed input keeps accountability with the engineering process.

Review based on risk, not authorship

I would not create one ceremonial review policy for "AI code" and another for human code. The author does not determine the blast radius.

A one-line IAM wildcard can be more dangerous than a generated client with 500 lines. A documentation edit may need little more than link and rendering checks. A payment calculation deserves domain review whether a staff engineer, a junior developer, or an agent wrote it.

Classify the change instead:

Does it alter authentication, authorization, secrets, or network access?
Does it modify a schema, public API, or event contract?
Can it destroy, duplicate, or expose data?
Does it change infrastructure or deployment behaviour?
Can failure affect money, compliance, or customer access?
Is rollback tested and practical?

Higher-risk answers require stronger evidence and named reviewers. Low-risk work can move quickly with automation. This avoids turning review into theatre while preserving scrutiny where it matters.

GitHub's responsible-use guidance for Copilot agents explicitly presents generated changes, CI results, review activity, and session history so people can inspect work before merge. The workflow expects a human decision. Enabling automatic agent merge does not change who owns the outcome.

Keep a short evidence note with the pull request

I would ask every non-trivial agent-authored change to include a compact evidence note:

Behaviour changed:
- Failed items can be retried without duplicating successful writes.

Constraints preserved:
- Authorization is evaluated at retry time.
- The existing single-item endpoint is unchanged.

Evidence:
- Unit and integration suites passed.
- Concurrent retry test produced one committed write.
- Migration was applied and reversed in a clean environment.

Not verified:
- Retry behaviour above 500 requests per second.

Rollback:
- Disable `bulk_retry_enabled`; no schema rollback required.

This should be a set of claims a reviewer can inspect, not an essay about what the code probably does. The "not verified" section is especially useful because it stops missing evidence from being disguised as confidence.

The note also gives operations a head start. If the feature fails, the team already knows the intended boundary and the first containment action.

Specifications should make ownership clearer

Spec-driven development is worth adopting when it makes intent explicit, catches ambiguity early, and gives agents a bounded problem. It becomes dangerous when the organization uses the spec as permission to stop understanding its software.

My preferred workflow is simple:

reviewed intent
      ↓
small implementation slice
      ↓
independent verification
      ↓
risk-based human review
      ↓
controlled release and observation

Agents can accelerate every step. They can draft the spec, explore the repository, implement a slice, run tests, and assemble the evidence. That is useful work.

The engineering team still has to decide whether the evidence is sufficient. It still has to understand the failure modes and support the result after deployment.

A specification can tell an agent what to build. It cannot take the pager.

UTC is not a data contract. Period.

Sudeep Hazra — Mon, 12 Oct 2026 10:36:09 +0000

Someone in the October r/dataengineering discussion thread described losing three days to a timezone mismatch. The production dashboard looked correct for the first two hours each morning, then drifted by four or five hours.

That is painfully believable. Timestamp bugs often survive because every value looks like a valid date. The pipeline runs, rows arrive, and the dashboard renders. Only the meaning moved.

The usual prevention advice is "store everything in UTC." I would store event instants in UTC too, but that sentence is not a complete data contract.

UTC can identify when something happened. It cannot always tell us which local business day it belonged to, which timezone rules were applied, or whether 2026-11-01 01:30 referred to the first or second occurrence of that wall-clock time.

We need to model time according to the question the data must answer.

One timestamp can carry several meanings

Consider a restaurant booking recorded as 2026-11-01 01:30.

That value might mean:

an instant when the booking was created
a local reservation time chosen by a customer
a daily reporting boundary
a recurring schedule such as "every Sunday at 01:30"
text copied from a source that never supplied a timezone

Those are different data types even if the source puts all of them into a column called timestamp.

I would distinguish at least four concepts:

Concept	Example	Useful representation
Instant	Payment received	UTC timestamp
Local wall-clock time	Store opens at 09:00	Local date/time plus timezone name
Business date	Trading day or service date	Explicit date derived by an owned rule
Observed offset	Source sent `+05:30`	Offset retained as source evidence

An offset is not a timezone. +01:00 tells us the relationship to UTC for one value. It does not carry the historical and future daylight-saving rules of Europe/London or another named zone.

IANA published tzdata 2026e on September 30. Timezone rules are maintained data, not a law of nature embedded permanently in every runtime. IANA notes that downstream distributors may wait for a timezone database release before updating consumer software. Two services can therefore disagree even when both use a legitimate timezone name.

Database types do not rescue an ambiguous source

Type names are not consistent enough to infer intent by sight.

PostgreSQL documents that plain timestamp means timestamp without time zone. A value with a timezone indicator can be silently interpreted according to the target type and session settings. PostgreSQL's timestamptz represents an instant and displays it in the current session timezone. It does not preserve the original timezone name as a business attribute.

Snowflake has three timestamp variants with different behaviour. Its date and time documentation says:

TIMESTAMP_NTZ stores wall-clock time without timezone consideration.
TIMESTAMP_LTZ stores UTC internally but performs operations in the session timezone.
TIMESTAMP_TZ stores UTC with an associated offset, not the original timezone name.

Snowflake also warns that adding six months to a TIMESTAMP_TZ value can retain the original offset even when daylight-saving rules would give the named location a different offset in the new month. The zone name is no longer there to recalculate it.

None of these behaviours is inherently wrong. The mistake is expecting a database type to recover context the source did not provide.

If the source sends 2026-10-10 09:00:00, the ingestion layer needs a declared rule. Is that UTC? The source system's configured zone? The user's zone? The location attached to the record? Guessing from the server timezone turns deployment configuration into business logic.

Put time semantics into the contract

For an event, I would prefer a payload shaped like this:

{
  "event_id": "evt_7192",
  "occurred_at": "2026-10-10T03:30:00Z",
  "source_local_datetime": "2026-10-10T09:00:00",
  "source_timezone": "Asia/Kolkata",
  "source_utc_offset": "+05:30",
  "recorded_at": "2026-10-10T03:30:04Z"
}

Not every event needs every field. Keep the values when they answer different questions.

occurred_at supports global ordering and elapsed-time calculations. source_local_datetime preserves what the source or user saw. source_timezone allows the local value to be interpreted using named rules. source_utc_offset records what the producer applied at that moment. recorded_at helps measure ingestion delay.

When the event only represents a machine action and no local-calendar meaning exists, the UTC instant may be enough. When the event is a future appointment, store the local date/time and timezone name because the offset can change before the appointment occurs.

I would make the contract explicit about these questions:

Which fields are instants and which are wall-clock values?
Is a timezone name required, and where does it come from?
Is the source offset preserved?
Which timezone database version did the producer use?
What happens when a local time is ambiguous or does not exist?
Who owns the definition of business date?
Are late events assigned by occurrence time or arrival time?

"All timestamps are UTC" answers only part of the first question.

Business dates deserve their own rule

Many reporting errors happen after ingestion, when an instant becomes a date.

Imagine a payment at 2026-10-10T23:30:00Z. In London, Kolkata, and Los Angeles it can belong to different local dates. If finance reports by the merchant's trading day while support reports by the customer's local day, both teams may be correct and still disagree.

Do not let each dashboard rediscover the rule with a different call to DATE().

Model the derived date and its ownership:

occurred_at_utc
      +
merchant_timezone
      +
trading_day_cutoff
      ↓
merchant_business_date

The cutoff may not be midnight. Hotels, transport operators, markets, and overnight venues often define an operating day around a business boundary. That is domain logic, not timestamp formatting.

Keep the original instant so the date can be recomputed if the rule changes. If reports need to remain historically reproducible, version the rule or materialize the assigned business date with the rule version used.

Test the conversion path, not just the parser

A parser test proves that the string is syntactically valid. It does not prove that the pipeline preserved the same instant.

I would build a small set of timezone fixtures and run them through every important boundary:

source payload
    ↓
application parser
    ↓
message or file serialization
    ↓
warehouse column
    ↓
transformation model
    ↓
reporting query

The fixture set should include:

a normal UTC value
a non-whole-hour offset such as +05:30
the hour before and after a daylight-saving transition
an ambiguous local time during a fall-back transition
a nonexistent local time during a spring-forward transition
a historical date whose timezone rules differ from today
a future scheduled time
a source value with no timezone, if the contract permits one

For every fixture, record the expected instant, local rendering, business date, and rejection behaviour. These values should come from an agreed reference, not from whatever the current test runtime happens to return.

Snowflake's CONVERT_TIMEZONE documentation makes one source of accidental behaviour clear. A TIMESTAMP_NTZ has no timezone, so the two-argument form interprets it using the current session timezone. A warehouse task and an analyst session can therefore produce different answers if their session parameters differ.

Set session timezone configuration explicitly in pipelines. Then test it.

Add invariants that expose drift

The first useful alert is not "the timestamp column is non-null." A wrong timestamp is usually non-null.

I would monitor invariants such as:

the difference between occurred_at and recorded_at stays within an expected range
source offsets match the named timezone for the supplied local date/time
no new timezone names appear without contract approval
parsing failures and ambiguous local times remain visible
event counts by UTC hour and business hour remain within expected patterns
source and target daily totals reconcile across the timezone boundary

That last check catches the classic failure where no rows disappear but several move into the previous or next day.

Compare closed windows after an allowed lateness period. A live source count and a live warehouse count can differ because events are still in flight. Once the window closes, reconcile both total rows and the distribution around the day boundary.

If a pipeline begins drifting only after a particular hour, compare its session timezone, runtime timezone database, source offset, and conversion function. The shape of the error often points toward the layer that changed.

Treat timezone data like a dependency

Timezone rules change. That means the database version used by the operating system, language runtime, database engine, and warehouse is part of the pipeline's dependency set.

I would record those versions during deployment and expose them in diagnostics. When IANA publishes an update, test the zones relevant to the business before rolling it through every service. The goal is not to freeze old rules forever. It is to know when a conversion result changed and whether historical values need to remain reproducible.

For future schedules, recalculate derived UTC instants when the governing timezone data changes. For completed events, preserve the original instant and source evidence. Those are different migration policies because future intent and historical fact are different things.

Store the meaning, not only the value

UTC is the right representation for many event instants. I would still use it as the common ordering and transport form.

I would not ask it to carry local business meaning that was discarded earlier in the pipeline.

Name the concepts, preserve the source context that matters, define the conversion owner, and test the full route from producer to report. Then a timezone database update becomes a dependency change we can evaluate instead of a mystery that moves five hours of revenue into the wrong day.

A timestamp is easy to store. Time is part of the domain model.

Design semantic views around business questions, not agents

Sudeep Hazra — Mon, 05 Oct 2026 12:41:19 +0000

A question in the October r/dataengineering discussion thread caught my attention. A team was building Snowflake Semantic Views for business users and could not decide how to divide them. Should each topic have its own view, or should several views be connected to an agent?

That sounds like a Snowflake configuration question. It is really a modelling boundary question.

I would build semantic views around coherent business questions and stable metric definitions. I would let the agent route users to those views. The agent is an interface and orchestration boundary. It should not determine where the business model begins and ends.

This distinction matters because agents change quickly. Revenue, booking, inventory, and customer definitions usually need to outlive whichever chat interface happens to be fashionable this quarter.

A semantic view is a business contract

Snowflake defines a Semantic View as a schema-level object that maps business concepts to physical data. It contains logical tables, dimensions, facts, metrics, and the relationships between them. Cortex Analyst uses that metadata to generate SQL.

The useful part is not that the model sees friendlier column names. The view decides what a term means.

Consider a travel company asking for "booked revenue." That phrase needs decisions:

Does revenue use booking date, departure date, or payment date?
Are cancelled bookings excluded?
How are refunds represented?
Which currency conversion rate applies?
Is the metric additive across every dimension?

A description such as "total revenue from bookings" does not answer those questions. It merely gives the ambiguity a nicer label.

The semantic view should capture the accepted definition and the valid join paths. That makes it closer to a versioned business contract than a prompt accessory.

This is also why I would not start from the agent. An agent called travel_assistant may need bookings today, customer value tomorrow, and disruption operations next month. If the semantic model copies the agent's current scope, every new use case pushes unrelated concepts into the same view.

Avoid one view per physical table

Creating one semantic view for every warehouse table feels tidy. It preserves the physical architecture and gives each object a manageable size.

It also asks the consumer to reconstruct the business model.

If fact_booking, fact_payment, dim_route, and dim_customer become four separate semantic views, where does "revenue by destination" live? Which object owns the join between bookings and payments? Which view defines whether a refund reduces booked revenue or appears as a separate measure?

Snowflake's own getting-started guidance begins with business entities, their relationships, metrics, and dimensions. It then maps those concepts to physical data and recommends starting with a simple star schema. That sequence is important. The warehouse tables support the model; they do not define its audience.

A useful semantic view may include several logical tables when they form one understandable analytical subject. For example:

The boundary is not "all travel data." It is the set of concepts needed to answer booking performance questions with consistent grain and joins.

Avoid one giant enterprise view

The opposite design is one semantic view that contains every table, metric, synonym, instruction, and verified query in the warehouse.

This removes the routing decision by making every concept available everywhere. It also creates a model that few people can reason about.

Large views accumulate ambiguous names. status may refer to a booking, payment, flight, customer account, or support case. Relationships multiply, and valid join paths become harder to distinguish from technically possible ones. A change to one business area requires evaluating a shared object used by everyone else.

There is a governance problem too. Different domains may have different owners, release schedules, and access rules. Putting them into one object couples those decisions. The result may be convenient for the first demo and awkward for every production change after it (the usual bargain).

I would split the model when one or more of these conditions appear:

The questions serve different business decisions.
The core facts have different grains.
The metrics are owned and approved by different teams.
The relationships between the subjects are optional or ambiguous.
The access boundary is different.
A change requires a separate evaluation and release cycle.

For a travel platform, that might produce views such as booking_performance, customer_value, and service_disruption. These are examples of business subjects, not a universal naming scheme.

Some questions will cross those boundaries. If cross-domain questions are frequent and have agreed semantics, that is evidence for a composed view designed for that purpose. It is not a reason to merge everything in advance.

Let routing remain routing

Snowflake's Cortex Analyst REST API accepts an array of semantic models or views. For each query, Cortex Analyst chooses the most appropriate one from the list. A client can therefore expose several focused views without asking the user to choose a data source before every question.

That capability gives us a useful separation:

Routing answers, "Which model can handle this question?"

The semantic view answers, "What do these business terms mean, and which SQL relationships are allowed?"

Combining those responsibilities makes both harder to test. When an answer is wrong, the team cannot easily tell whether the router chose the wrong domain, the view defined the metric incorrectly, or SQL generation failed within an otherwise correct model.

There is one limit worth making explicit. Selecting from multiple views does not magically create a governed join across them. Snowflake's API documentation says Cortex Analyst chooses the most appropriate view for a query. If a single question genuinely needs concepts from two domains, model and test that combined analytical path rather than assuming the agent will invent it safely.

Verified queries are tests, not decoration

Descriptions and synonyms help, but a semantic view needs executable evidence.

Snowflake supports verified queries that pair a natural-language question with expected SQL. Its Cortex Analyst evaluation runs generated SQL and compares the result with the verified query. During an evaluation, selected queries are removed from the temporary view so they cannot guide the generation they are supposed to test.

That is a useful design. It turns a familiar business question into regression evidence instead of another prompt example that always looks correct in the demo.

I would keep two sets:

Guidance set
Questions that teach common language and approved patterns

Evaluation set
Questions held out to detect wrong SQL and regressions

The evaluation set should cover more than happy paths. Include common synonyms, ambiguous time periods, filters that previously failed, joins near the edge of the domain, and metrics that look similar but have different definitions.

For booking_performance, useful cases might include:

booked revenue for departures next month
payments received this month for bookings created earlier
cancelled passengers by route
refund amount without double-counting partial refunds
a question the view should decline because it belongs to service disruption

That last case matters. A reliable semantic boundary needs to say what it does not know.

Ownership matters more than view count

There is no correct number of semantic views. Three coherent views can be better than twelve, and twelve focused views can be better than one. Count is an outcome of the ownership model and question space.

For each view, I would record:

the business owner who approves metric meaning
the engineering owner who maintains mappings and relationships
the source models and expected grain
the questions it is designed to answer
the questions it must reject or route elsewhere
the verified-query guidance and evaluation sets
the access roles and release process

Then I would monitor how the model behaves in use. Snowflake exposes an administrator view containing Cortex Analyst requests across semantic models and views. That history can show questions that route poorly, common phrases missing from the model, and domains users try to combine.

Production usage should inform a change, not write directly into the semantic contract. A frequent query may reveal a missing synonym. It may also reveal that two departments use the same phrase for different metrics. Someone still needs to decide which interpretation is valid.

Start with one subject and make it boring

I would begin with one business subject that has an engaged owner, stable source data, and a small set of questions people already ask. Define the grain. Add the metrics and relationships needed for those questions. Build verified queries, keep some for evaluation, and measure the results.

Only then add another subject and routing.

The first success criterion is not that the agent can answer something impressive. It is that the same approved question produces the same correct result after the semantic view changes.

Design the views around business meaning. Let agents discover and route to them. When the interface changes, the definitions should remain useful. That is a much better test of a semantic layer than whether the first chatbot demo looked clever.

Your data engineer job description is trying to hire a team

Sudeep Hazra — Mon, 05 Oct 2026 12:18:58 +0000

A recent r/dataengineering discussion started with a frustrating result. A company interviewed 22 people for a data engineering role and could not find a match. After working to create data teams (specifically data engineering teams) for over a decade, I feel like this reflected a lot of the observations I also have had over these years.

The job needed someone with software engineering fundamentals who could work on a lakehouse and integrate it with the company's main application. The requested stack included Spark, AWS Glue, Lambda, Airflow, and dbt. Some applicants had analytics backgrounds, some came from software engineering without Spark experience, and some qualified data engineers wanted more money than the budget allowed.

The comments quickly became a debate about salary, training, and what the title "data engineer" is supposed to mean. That debate is useful because the hiring problem was not really a shortage of keywords. The role combined several kinds of ownership without saying which ones mattered most.

If a team needs one person to model business data, build distributed processing jobs, write application integrations, manage cloud infrastructure, operate production pipelines, and know its exact toolchain, it may be trying to hire a small team through one job description.

I would fix the role before adding another interview round.

A tool list does not describe the job

Spark, Glue, Lambda, Airflow, and dbt can appear in the same platform. They do not represent one coherent skill.

Someone may be excellent at dimensional modelling and dbt but have limited experience writing a service that exposes data to an application. Another engineer may understand distributed systems, IAM, deployment, and failure recovery but need a few weeks to learn Glue. A third may know every service name in the description because their last company used the same stack, yet struggle to explain what happens when a pipeline partially writes its output and retries.

The third candidate often passes the keyword screen. The first two may be more useful once the system reaches production.

AWS's Data Analytics Lens is a good reminder of how broad production data work becomes. Its design principles cover operational excellence, security, reliability, performance, cost, and governance of data and metadata changes. Knowing the SDK call that starts a Glue job is a small part of that responsibility.

I would define the role through the system it owns:

Then I would mark the boundaries this engineer owns directly, the boundaries they influence, and the boundaries another team operates.

That makes the requirement much clearer. "Own ingestion through serving for the customer lakehouse" describes a job. "Must know Spark, Glue, Lambda, Airflow, dbt, Terraform, Kafka, and Snowflake" describes a shopping trip.

Software engineering fundamentals need a concrete meaning

"Clean code" and "DRY" sound reasonable in a job description. They are also difficult to assess because two experienced engineers can disagree about both.

For a data platform, I would translate software engineering fundamentals into observable behaviours:

Can the candidate split a pipeline into components with clear inputs and outputs?
Can they make a write idempotent or explain why it cannot be?
Can they test transformation logic without deploying the whole platform?
Can they evolve a schema without surprising downstream consumers?
Can they trace a failed run, identify the damaged data, and describe recovery?
Can they review a change for security, cost, and operational impact?

These questions are closer to the work. They also give candidates from adjacent backgrounds a fair chance to show transferable judgement.

Google's SRE guidance treats a data pipeline as a service. The Data Processing Pipelines chapter recommends defining customer-facing objectives, detecting freshness and correctness failures, documenting the system, and building a development lifecycle that catches problems before production. That is software engineering applied to data. It is not a particular folder structure or an argument about whether a helper function should contain six lines or eight.

The interview should therefore test a change over time, not a snapshot of remembered syntax.

Give the candidate a small pipeline design. Then change one assumption:

The useful signal is how the candidate updates the design. Do they notice that corrections change the merge strategy? Do they separate freshness from correctness? Do they ask about deletion propagation, backups, and derived datasets? Do they add streaming infrastructure immediately, or first test whether frequent incremental batches meet the requirement?

That conversation tells me more than asking someone to name five Spark transformations.

Decide what must be known on day one

Every role has skills that cannot wait. The mistake is treating the entire current stack as one of them.

I would divide requirements into three groups:

Group	Meaning	Example
Required judgement	Hard to acquire during onboarding	Failure recovery, data correctness, security boundaries
Transferable implementation skill	Needed, but not tied to one vendor	Distributed processing, orchestration, SQL, testing
Local stack knowledge	Can be learned with access and support	Glue configuration, repository conventions, one dbt package

This does not mean tools never matter. A team migrating a complex Spark estate may reasonably need someone who has debugged Spark executors under load. A three-month delivery deadline may leave little time for cloud onboarding. Those are constraints, and the job description should say so.

The list becomes suspicious when every item is mandatory but none is identified as decisive.

For a mid-level lakehouse role, I would usually make production pipeline ownership and solid programming ability mandatory. I would require depth in one distributed processing environment, not necessarily the exact managed service. I would treat the local orchestration and transformation tools as learnable unless the person will own them alone from their first week.

If the team cannot support any learning, that is another requirement worth stating. It also means the role is more senior than the number of years in the description may suggest.

A better interview loop

I would keep the loop small and connect every stage to a decision.

The first conversation should establish scope. Ask the candidate to describe a data system they owned, where their responsibility started, where it ended, and what failure they were expected to handle. This prevents a familiar product name from standing in for actual ownership.

The technical exercise should be a short design problem using a domain anyone can understand. Orders, invoices, or device readings are enough. Provide volumes, latency, and recovery requirements. Let the candidate ask for missing constraints.

The code review should contain an ordinary pipeline bug: a non-idempotent retry, a timestamp interpreted in local time, an unbounded read, or a write that can expose partial output. Ask what they would change and how they would prove the fix.

The operational discussion should begin with evidence:

The job is green.
The dashboard is missing yesterday's revenue.
What do you inspect first?

A good answer separates orchestration status from data correctness. It moves through source arrival, processing boundaries, output counts, freshness, and reconciliation. The exact monitoring product is secondary.

Finally, include one learning question. Pick a tool the candidate has not used and provide a short piece of documentation. Ask them to explain how they would validate it in a proof of concept. This tests the skill the job will require repeatedly: learning the next service without pretending its logo is an architecture.

The role may still be too wide

Improving the interview cannot repair an incoherent operating model.

If one engineer is expected to own business definitions, application APIs, Spark performance, IAM, Terraform, on-call response, and stakeholder reporting, the team needs to choose. It can narrow the role, split the responsibilities, raise the seniority and compensation, or hire for potential and provide support.

There is no interview technique that creates a candidate who is simultaneously specialised in every layer, available within the current budget, and productive without onboarding. Sometimes the market is giving feedback about the design of the job.

My recommendation is to hire for the hardest judgement the team cannot teach quickly. Make the owned system explicit. Test engineering through realistic changes and failures. Treat vendor knowledge as a preference unless there is a concrete reason it must exist on day one.

The goal is not to find someone who has seen the same collection of tools. It is to find someone who can keep the data system correct when the collection changes. That is a rational and practical approach to a never-ending cycle.

Your Kafka cluster is healthy. But the event is still missing

Sudeep Hazra — Sat, 26 Sep 2026 08:42:01 +0000

A recent r/dataengineering question described a familiar setup: many microservices publish JSON events to Kafka, each project has an expected throughput, and the business wants an alert when those events stop arriving.

The natural response is to look for a Kafka monitoring tool. That is useful, but it solves only part of the problem.

A Kafka dashboard can tell you that the brokers are available, partitions have leaders, replicas are in sync, and consumers are not falling behind. Every panel can be green while an upstream service has stopped producing the one event the business needs.

The system is healthy. The data product is not.

That distinction changes what we need to monitor.

Kafka health and business health are different signals

Apache Kafka's monitoring guidance covers the platform well. It recommends watching message and byte rates, request latency, fetch rates, replica state, and consumer lag. Those metrics answer important operational questions:

Can producers reach the cluster?
Are brokers accepting requests?
Are replicas keeping up?
Are consumers processing records fast enough?
Is one partition behaving differently from the others?

They do not answer whether the expected invoice.created event arrived for a particular market during the last fifteen minutes.

Consumer lag is a good example. A lag of zero sounds healthy. It can also mean that no records were produced. If the source application failed before publishing, the consumer has nothing to read and therefore nothing to lag behind.

The same problem appears with throughput. An aggregate topic rate may look normal while one tenant, event type, or producer has gone silent. A busy topic can hide a very specific outage.

I would separate monitoring into four layers:

Kafka infrastructure
        ↓
Message flow
        ↓
Event contract
        ↓
Business expectation

Each layer needs different evidence and a different owner.

Layer 1: prove the platform can carry messages

The first layer is normal Kafka operations. Watch broker availability, request errors, under-replicated partitions, ISR changes, disk pressure, network saturation, produce latency, fetch latency, and authentication failures.

This is where JMX exporters, a managed Kafka metrics API, Prometheus, Grafana, or an observability platform fit. The product matters less than collecting the right broker and client signals.

Consumer lag belongs here too, although I would treat it as a flow signal rather than a complete service-level indicator. Kafka exposes records-lag, records-lag-max, fetch rates, commit latency, and time between polls. Those metrics can identify a slow or stuck consumer.

An alert such as this is useful:

consumer lag is increasing
AND
records consumed per second is below the normal range
FOR ten minutes

It tells us that records exist and the consumer is not keeping up. It still says nothing about records that were never published.

Layer 2: prove messages move through the expected path

The next layer measures flow at boundaries. For every important producer and consumer, record a small set of counters and timestamps:

events produced
events accepted by Kafka
events consumed
events processed successfully
events rejected or sent to a dead-letter path
timestamp of the newest event

The labels need care. service, event_type, environment, and perhaps region are usually manageable. Raw customer IDs, transaction IDs, and message IDs are not. Putting high-cardinality business identifiers into a metrics system is an efficient way to turn an observability improvement into a cost incident.

Keep per-message identifiers in logs or traces. Keep metrics dimensions bounded.

OpenTelemetry's messaging conventions provide common attributes for messaging systems, destinations, operations, consumer groups, and message context. The conventions are still marked as development in several areas, so I would pin the version used by the instrumentation instead of assuming the attribute names will never change.

Tracing is helpful when a business operation crosses several services. A correlation or conversation ID can connect the API request, producer span, Kafka operation, consumer span, and downstream write. That gives an engineer a route through the failure instead of a collection of unrelated charts.

Tracing every message may be too expensive at high volume. Sampling is reasonable for diagnostics. Counts and freshness metrics should remain complete because they drive alerts.

Layer 3: prove the event is usable

Arrival is not success.

A producer can publish malformed JSON at the expected rate. A serializer can omit a required field. A schema change can preserve valid syntax while changing the meaning of a value. The throughput chart will look excellent right up to the point where someone opens the downstream report.

Validate the contract where ownership is clearest. That may be in the producer before publish, in a schema registry, or at the consumer boundary. Record at least:

schema validation failures
unsupported schema versions
deserialization failures
required-field failures
duplicate or out-of-order events when those conditions matter
dead-letter volume and age

Do not collapse all of these into processing_error_total. The response to a broken schema is different from the response to a timeout writing to a database.

I would also resist putting the full payload into observability events. It creates a second, poorly governed copy of potentially sensitive data. Record the failure category, event type, schema version, and a safe lookup reference. Keep payload access inside the data platform's normal security boundary.

Layer 4: encode the business expectation

This is the part a Kafka dashboard cannot infer.

"Expected TPS" sounds like a threshold, but a single number is rarely enough. Traffic changes by hour, weekday, region, and business calendar. Some event types are continuous. Others arrive in bursts after a batch closes. A flat minimum can create noise during quiet periods and miss a partial outage during busy ones.

Represent the expectation explicitly:

event_type: invoice.created
producer: billing-service
environment: production
schedule: weekdays 08:00-20:00 Europe/London
minimum_events: 300
window: 5m
maximum_silence: 10m
owner: billing-platform
severity: high
runbook: https://example.internal/runbooks/invoice-events

This is configuration, not dashboard decoration. Put it in version control. Review changes. Give each expectation an owner.

The evaluation process can be simple:

expected event rules
        +
observed event counters and freshness
        ↓
evaluation job
        ↓
pass, warn, fail, or no-data
        ↓
alert with owner and runbook

The no-data state matters. Monitoring systems often treat a missing time series differently from a zero value. Your evaluator must decide what absence means for each rule. Prometheus alerting rules support duration-based conditions through for, which helps avoid paging on a brief gap. The business rule still has to define how long a gap is acceptable.

Reconcile counts instead of trusting one point

For important flows, compare counts across boundaries over the same business window:

producer accepted:  100,000
Kafka observed:      99,998
consumer processed:  99,990
target committed:    99,987

Those numbers do not prove which eight or thirteen records are missing, but they tell you where to investigate.

Use event IDs or business keys for periodic reconciliation when exact completeness matters. Counters are operational signals. Reconciliation is evidence.

The window must account for retries and late arrivals. Comparing two live counters at the current second will generate false differences because each stage observes the event at a different time. Close a window, allow a defined lateness period, then evaluate it. Streaming does not remove the need for accounting.

Alert the team that can act

The Kafka platform team owns broker capacity, replication, cluster access, and platform availability. The producer team owns whether it emits the promised event. The consumer team owns processing and downstream delivery. A data product owner may own the business expectation.

Sending every alert to the Kafka team recreates the old middleware support queue with newer software.

An alert should include:

what expectation failed
when the last valid event arrived
expected and observed counts
the affected producer, topic, event type, and consumer
whether Kafka itself is healthy
the owning team
the first diagnostic query or dashboard
the runbook

"Kafka events missing" is not enough. "invoice.created from billing-service has been silent for 12 minutes; brokers are healthy and other producers are active" is actionable.

Start with the important flows

I would not begin by buying a broad data-observability platform or instrumenting every event type. Start with the ten flows whose absence causes money, compliance, or customer impact.

For each one, define:

The event contract.
The expected schedule and volume.
The maximum acceptable silence.
The producer and consumer owners.
The reconciliation rule.
The response when the rule fails.

Then use the monitoring stack already in place. Add a new product only when the existing stack cannot express, evaluate, or route these rules without unreasonable work.

The useful dashboard is not the one with the most Kafka metrics. It is the one that can distinguish a broken broker, a slow consumer, an invalid event, and a producer that stopped doing its job.

Kafka health is necessary. The business still needs proof that the right event arrived.

Your agent does not need to write an essay to make a decision

Sudeep Hazra — Sat, 26 Sep 2026 08:15:32 +0000

Something interesting appeared in a recent LocalLLaMA project showcase. Several projects were exploring small models that do not generate prose. They take a state, a question, and a fixed set of choices, then return a probability for each choice.

That sounds less impressive than an autonomous agent narrating its plan. It may also be exactly what many agent loops need.

Consider the decisions inside a typical tool-using workflow:

Which tool should handle this request?
Is this command safe enough to run automatically?
Did the previous step succeed?
Should the workflow retry, stop, or ask a person?
Which retrieved document should be examined next?

These are closed-set decisions. Asking a generative model to produce a paragraph, parsing that paragraph, and hoping it stayed inside the allowed choices is a surprisingly elaborate way to select one value from an enum.

My current view is simple: use generative models for open-ended reasoning and content. Use constrained decision components when the answer space is already known.

Generation and decision are different jobs

A generative model is useful when the output cannot be enumerated in advance. Writing a migration plan, diagnosing an unfamiliar failure, explaining a trade-off, or proposing a patch all need flexible output.

Routing a ticket to one of twelve queues does not.

We often use the same large model for both jobs because the API is convenient. The model receives a prompt such as:

Choose one action: retry, stop, escalate.
Return JSON only.

The application then validates the response, repairs malformed JSON, rejects invented actions, and perhaps asks the same model to try again. We added a language generator, then built a fence around its language generation.

Structured output improves the interface, but it does not change the underlying job. The system still generates tokens to choose from a fixed set.

Projects such as Contrastive Language Models approach the problem differently. CLM encodes the current state and candidate actions, scores their relationship, and returns a distribution over the supplied actions. The repository describes uses such as tool routing, trajectory ranking, and retrieval shortlisting.

Decider takes another route. It reads a state and typed questions, then produces probabilities for fixed choices, yes-or-no decisions, or described score levels in one pass. Its core promise is deliberately narrow: no free-form decoding and no output outside the choices supplied by the application.

These projects are early and their benchmark claims need independent validation. The architecture is still worth paying attention to.

The application should own the choice set

In a production agent, the model should not decide which actions exist. The application should.

application state
        +
allowed actions
        ↓
decision component
        ↓
probabilities
        ↓
policy and threshold
        ↓
execute, ask, or stop

That boundary gives us several useful properties.

First, impossible actions stay impossible. If delete_customer is not in the candidate set, the decision component cannot select it. This is stronger than telling a model in a prompt not to mention deletion.

Second, the result is inspectable. A distribution such as this is easier to evaluate than a persuasive paragraph:

{
  "retry": 0.18,
  "stop": 0.07,
  "escalate": 0.75
}

Third, the application can apply policy after inference. A high score is evidence, not authority.

if decision == "retry" and probability >= 0.90 and retry_count < 2:
    retry()
else:
    request_review()

The model proposes. Code decides what that proposal is allowed to do.

Probability is useful only when it is calibrated

A number that looks like confidence is not automatically a trustworthy probability.

If a model assigns roughly 0.8 to one hundred decisions, we would like about eighty of them to be correct. That is calibration. A model can have good top-choice accuracy and still be overconfident, which makes threshold-based automation dangerous.

This is where a typed decision interface improves the engineering conversation. We can measure:

accuracy by decision type
false-allow and false-deny rates
calibration error
coverage at a chosen threshold
latency and cost per decision
performance when none of the supplied choices is correct

The last case is easy to miss. A closed set can be wrong. If the workflow offers retry, stop, and escalate, but the correct action is refresh_credentials, the model cannot repair the application's incomplete choice set.

Include an unknown or none_of_the_above option where appropriate. More importantly, evaluate whether it is selected when the input falls outside the known cases.

The threshold also depends on consequence. A 0.75 routing decision may be acceptable for choosing a knowledge-base category. It is not enough evidence to run a destructive infrastructure command.

Guardrails belong around the tool, not inside the prompt

The decision model is not a security boundary.

The OpenAI Agents SDK guardrail documentation makes a useful separation: tool input guardrails run before execution, tool output guardrails run after execution, and either can reject content or halt the run. The SDK also documents important coverage limits for different tool types.

The exact framework is less important than the placement. Authorization and safety checks must sit on the execution path.

model chooses candidate action
        ↓
schema validation
        ↓
authorization check
        ↓
risk policy
        ↓
human approval if required
        ↓
tool execution
        ↓
result validation

Do not ask the model, "Is the user allowed to delete this resource?" when the target system can answer that question from its own permission model. Do not treat a 0.99 probability as permission. Use identity, authorization, resource state, and explicit policy.

A decision component can help classify risk or select a route. It cannot grant authority the caller does not have.

Small decisions can make large agents easier to operate

Agent diagrams tend to focus on the large reasoning step. Production failures often happen in the small transitions around it.

Should the agent retry after a timeout? Was the tool response complete? Does the evidence support the conclusion? Is the next operation reversible? Those decisions determine whether an isolated model error becomes a repeated side effect.

Breaking them out creates observable control points:

request
   ↓
route decision
   ↓
reasoning or retrieval
   ↓
action decision
   ↓
policy check
   ↓
tool
   ↓
success decision
   ↓
continue or stop

Each decision can have its own dataset, threshold, fallback, and owner. That is less magical than one agent prompt. It is also much easier to test.

This direction fits the broader advice in Anthropic's Building effective agents: start with the simplest workable pattern, prefer predefined workflows when the path is known, and add agent autonomy when flexibility justifies the added cost and risk.

The interesting part is not replacing every generative call with a classifier. It is noticing which calls were never generative problems in the first place.

Evaluation needs real workflow data

A benchmark can tell us whether a model distinguishes choices in a prepared dataset. It cannot tell us whether those choices represent the messy states in our system.

I would build the evaluation set from production-shaped examples:

normal cases that should pass automatically
ambiguous cases that should request review
rare failures that previously caused incidents
adversarial input that tries to steer the decision
stale or incomplete context
cases where the correct choice is absent
repeated attempts after a tool failure

Label them with the decision that the workflow should take, not the explanation we hope the model writes.

Then compare at least three baselines:

Deterministic rules.
A general generative model with structured output.
A specialized or constrained decision model.

Rules may win. For example, retry_count >= 2 does not need inference. A model becomes useful when the state contains language or evidence that code cannot classify reliably with a small, stable rule set.

Run shadow evaluations before granting automation. Record the model's decision and probability without executing it. Compare that output with the actual operator or workflow decision. This shows where a threshold would automate safely and where it would merely automate confidence.

Do not create a second platform by accident

There is a predictable failure mode here. A team sees specialized decision models, creates a separate serving stack, a feature store, a training pipeline, an evaluation service, and a new control plane before proving that one decision benefits from any of it.

Start with one bounded choice that occurs frequently enough to matter.

Tool routing is a reasonable candidate. Define the allowed tools, create a representative test set, compare rules and models, and measure error cost. Serve a smaller model only if it produces a material latency, cost, privacy, or accuracy benefit.

For low-volume workflows, a general model with strict structured output may remain the simplest operational choice. Fewer services can be worth a little extra inference cost.

The recommendation changes when decision calls dominate the loop. If an agent makes dozens of small choices per task, generation latency and token cost accumulate. A fast decision component can then reduce both, while giving the application a consistent probability interface.

Keep the agent for the parts that need an agent

Open-ended work still benefits from capable generative models. A decision model will not investigate an unfamiliar production failure, compare architecture options, or write a useful migration plan.

It can decide which diagnostic to run next from an approved list. It can estimate whether the evidence supports another step. It can route uncertainty to a person before the workflow turns a weak guess into an action.

That is enough.

An agent does not become less intelligent because some decisions move into smaller, constrained components. The system becomes clearer about where intelligence is needed, where policy belongs, and where ordinary code is still the better tool.

If the answer must be one of five choices, make the system choose one of five choices. Save the essay for the part that needs an explanation.

Before You Add an Event Bus, Name the Failure

Sudeep Hazra — Mon, 21 Sep 2026 11:53:00 +0000

A discussion about event-driven architecture caught my attention because the replies disagreed on something more useful than Kafka versus a database. People were describing different problems with the same word: event.

One team needs to run a slow task after a transaction. Another needs to tell several independent systems that a business fact has changed. A third needs an audit history it can replay. All three may produce a message, but they have different requirements for ownership, ordering, and recovery.

I would start by naming the failure that the current design cannot handle. If that failure is a web request waiting on slow work, a database-backed job may be enough. If independent consumers need a durable record of a business change, publishing an event may be worth the extra machinery. That gives the team a reason for the extra component.

A slow operation is not automatically an event

Imagine an application that accepts an order and generates a PDF receipt. The user needs to know whether the order was accepted. They do not need to hold the connection open while the PDF renderer runs.

The first design I would test is simple:

The order and job are written in one database transaction. If the transaction rolls back, neither exists. If the renderer fails, the job remains visible for a retry or investigation. The application can show the user an order state and a separate receipt state. There is no broker to operate merely because work happens later.

That job still needs engineering. Give it a stable ID, an attempt count, a timeout, and a clear terminal failure state. Decide how a worker claims a job and how another worker takes over after a crash. PostgreSQL documents SKIP LOCKED as useful for avoiding contention among consumers of a queue-like table, while warning that it gives an inconsistent view of rows. That is a tool for workers, not a way to make arbitrary reporting queries faster.

A database queue has limits. Heavy fan-out, many unrelated consumers, a large backlog, or retention far beyond the operational life of a job can make the table and its cleanup awkward. Those are measurable reasons to consider messaging infrastructure. “We want async” is not yet one.

Publish facts when other owners need them

Now add a finance service, a customer notification service, and a fulfillment system. Each has its own release schedule and its own idea of what to do when an order is paid. The order service should not need to call all three during the payment request and wait for their health before it can record payment.

OrderPaid can be a useful event here. It says something that has happened, with an order ID, event ID, and version of the contract. The consumers decide what that fact means for their own systems. This is a better reason for an event than the mere existence of multiple functions in one application.

The awkward part is the boundary between the order database and the broker. This code is unsafe:

The process can die between those lines. Reversing the lines creates the opposite problem: a consumer may act on an event for a payment that never committed. AWS describes the transactional outbox pattern for this dual-write problem. Store the business update and an outbox record in the same transaction; a separate publisher sends committed records to the broker.

The outbox improves the handoff. It does not make the rest of the system atomic. The publisher can send an event and crash before marking it delivered. The consumer must cope with that event arriving again. A stable event ID and a consumer-side record of processed IDs are more useful than a diagram that promises “exactly once” without saying where the guarantee ends.

For example, a notification consumer can store OrderPaid:event-123 before or with the state change that schedules an email. If it sees the same event again, it can skip the duplicate scheduling step. A payment consumer has a higher bar: it must establish whether an external charge already succeeded before retrying. Its idempotency boundary is the payment provider and the local record together, not only the broker.

Decide what the user must know now

Asynchrony changes the product contract. Suppose checkout returns “payment complete” immediately, but fulfillment might reject the order ten minutes later because stock ran out. That may be a valid business process, but the UI and support team need to know what “complete” meant.

For each step, I would write down three states:

Question	Example answer
What is committed before the response?	Payment authorization and order record.
What can finish later?	Receipt generation and loyalty points.
What happens if later work fails?	Retry, visible pending state, then support action.

If the next operation must succeed before the user can proceed, a synchronous API call may be easier to reason about. An event can still be emitted afterward for observers. If the user can wait, an accepted response plus a status endpoint can make the delay explicit. AWS's asynchronous communication guidance describes that claim-check approach: acknowledge the request, return an identifier, and let the client retrieve the later result.

The delay has a user-facing cost: the text on a screen, the retry a user may press, and the question a support engineer has to answer.

Delivery guarantees do not finish the design

The team needs to decide what duplicates or ordering would do to the business operation before selecting a broker. A standard Amazon SQS queue, for example, delivers messages at least once. The same message may be received again. Its standard queue type also gives best-effort ordering.

Those properties are fine for many tasks. A consumer updating a current projection can ignore an event it has already processed. A consumer applying account balance deltas cannot casually apply the same delta twice. It may also need per-account ordering. Picking a FIFO queue can help with ordering and broker deduplication, but it does not remove the need to reason about retries across external systems and the full business workflow.

I would ask the following before choosing the transport:

Is this a command for one worker, or a fact several owners may consume?
What state is committed when the message is created?
Can the same message be processed twice without damage?
Must messages be ordered globally, per entity, or not at all?
How long must a consumer be able to recover after an outage?
Who notices a stuck consumer, and how is the backlog cleared?

The answers narrow the choice considerably. A one-owner receipt job does not need the same infrastructure as a company-wide change stream. A durable business event deserves more thought than a callback disguised as a topic.

Operate the failure path before scaling the happy path

Events make producers and consumers independent in useful ways. They also make a transaction harder to follow across time. An API request may be complete while the business process is still pending in another service. A dashboard that shows only broker throughput cannot tell you whether orders are actually progressing.

I would give each event a correlation ID, a producer timestamp, a contract version, and an owning service. Then I would measure the delay from publication to the consumer's business outcome, not merely the time until a message is read. A dead-letter queue is a holding area for work that failed repeatedly. It is not a recovery plan until someone owns inspection, correction, and replay.

Changes to the event contract need similar care. If consumers deploy independently, a producer cannot assume everyone upgrades on the same day. Add fields in a compatible way, document their meaning, and test old consumers against new payloads. When the meaning of a fact changes, a new event version may be clearer than silently reusing an old name.

This is the cost side of the architecture decision. If a system has one application and one database, a job table may give the team all the separation it needs. If several autonomous systems depend on the same business fact and must recover independently, the broker, outbox, consumer idempotency, and operational tooling earn their place.

My rule is to write down the failure and the recovery path first. Once those are concrete, the decision between a worker, an API call, and an event is usually much less mysterious.

What Stays in Secrets Manager After Workload Identity?

Sudeep Hazra — Mon, 21 Sep 2026 11:41:03 +0000

I liked the question in a recent r/devops discussion: after moving workloads to identity federation, how many secrets are left?

The answer depends on where each credential crosses a trust boundary.

GitHub Actions can exchange an OIDC token for short lived cloud credentials. An application on AWS can use an IAM role instead of carrying an access key. AWS even supports temporary credentials for workloads outside AWS, provided you operate the certificate infrastructure needed to establish that trust.

Those changes remove a class of long lived credentials. They do not cause a payment provider, an older database, or a webhook sender to accept your cloud identity. Some secrets remain because the other system controls its own authentication method.

My recommendation is to migrate the paths that support federation, then give the remaining credentials more deliberate ownership. The useful measure is how much risk and work remain per credential, not the number of entries in a vault.

Draw the boundary before deleting anything

Consider a service that writes to S3, calls an external billing API, receives signed webhooks, and connects to an older SQL Server.

The S3 access key is a good candidate for removal. Give the workload an identity and a narrowly scoped role. The billing API key still exists because that API expects a key. The webhook signing secret still exists because the receiver must verify what the sender signed. The database password might remain until that particular server version, driver, and deployment can use another supported authentication path.

I would draw that as four separate edges:

The last line needs investigation, not an assumption. Some SQL Server deployments can use Microsoft Entra authentication; an older on premises deployment may have different constraints. Check the exact version and connection path before promising a passwordless migration.

There is another subtle point here. A workload role can authorize a service to read a secret from a manager. It does not remove the secret stored there. It improves how the service obtains that value and who may read it. The provider credential still needs scoping, rotation, revocation, and an owner.

A report saying “we use identity for all workloads” tells me little about the secrets inventory. I still want to know how each destination authenticates the caller.

The leftovers are often harder than the keys you removed

Cloud access keys are painful when they spread through CI settings and configuration files, but cloud platforms give us mature replacements. The remaining secrets may have awkward lifecycles.

One vendor lets you create a replacement key while the old key stays valid. Another gives you one active credential, so rotation requires a coordinated cutover. A webhook endpoint may accept more than one signing secret during a transition, or it may not. An appliance may require a manual change through a browser. A shared password may have several consumers that nobody documented.

These differences determine the operating procedure. A scheduled rotation button is useful only when the destination also changes and every consumer picks up the new value. AWS Secrets Manager's rotation documentation is explicit about updating both the stored secret and the database or service. Changing the vault entry alone is a reliable way to schedule an outage.

For each remaining credential, I would record:

Question	Why it matters
Who issues and revokes it?	Determines the actual recovery path.
Which workload uses it?	Limits access and exposes accidental sharing.
Can old and new values overlap?	Determines whether rotation can avoid downtime.
How does the workload reload it?	A new vault value is useless to a process holding the old one.
What happens if it leaks?	Sets urgency and containment steps.
Who can test the replacement?	Makes rotation an operation rather than a calendar reminder.

That inventory is more informative than “37 secrets remaining.” Two of those 37 might be broad production credentials with no tested revocation path. They deserve attention before twenty low privilege integration tokens.

Make migration reversible until the new path is proven

I would migrate one workload edge at a time. Start with a well understood cloud API call. Give the workload a role with only the permissions it needs, then run a real operation through the new identity. Check which principal appears in the audit trail. Check that the workload refreshes credentials before expiry. Run the job long enough to exercise a refresh, not merely its startup path.

Then remove the old key from the application's configuration and rerun the job. Only after that should you revoke the old key. Finally, search the places where copies tend to survive: CI secrets, deployment templates, local environment files, support scripts, and scheduled jobs. The old key may be unused by the main service but still active elsewhere.

For GitHub Actions, restrict which repository, branch, or environment can assume the cloud role. OIDC removes a stored cloud key, but a broad trust policy can still authorize the wrong workflow. The token exchange is only as narrow as the trust conditions and permissions on the role.

I would also test failure behavior. If federation is unavailable, does the job fail with a clear error? Does someone have a documented recovery path? Secretless authentication can still have a dependency on an identity provider, metadata endpoint, certificate authority, or token service. Those dependencies deserve monitoring.

Keep the secrets manager, but give it a smaller job

AWS describes Secrets Manager as a place for database credentials, application credentials, OAuth tokens, API keys, and other secrets. After federation, that list may shrink. The service still has a useful job: controlling access to values that cannot be replaced by an identity handshake.

I would organize the remaining inventory by system boundary and owner rather than by a generic prod/secrets folder. A billing key should be readable only by the billing workload. A webhook signing secret should be readable only by the verifier and rotation procedure. A database credential should have permissions that match the application's actual queries, with separate credentials for migration or administration.

Set rotation intervals according to the system's capability and exposure. Do not promise automatic rotation for a vendor that offers no safe API or overlapping credentials. For those cases, write a cutover runbook and rehearse it. A manual procedure that has been tested is better than an “automated” rotation that only changes one side of the connection.

There is a cost decision too. If a team has a small number of static credentials, it may not need another vault product or a custom broker. Reuse the secrets service already in the platform, provided it gives the team access control, audit history, and a workable recovery path. Additional infrastructure should solve a specific remaining problem.

What I would measure

The percentage of credentials replaced is easy to celebrate and easy to misread. I would track four operational results instead:

Which long lived cloud keys have actually been revoked?
Which remaining secrets have a named owner and known consumers?
Which credentials can be rotated without an outage, and when was that last tested?
How quickly can a leaked credential be identified and disabled?

Those answers tell us whether the migration changed production risk. They also expose the next improvement: a vendor integration to replace, a database auth path to modernize, or a shared key to split by workload.

Workload identity is worth adopting where it fits. It removes credentials that never needed to be stored in the first place. The interesting work begins after that, when the remaining list is short enough to read and inconvenient enough to deserve proper attention.

Your Orchestrator Is an Operating Model, Not a Feature List

Sudeep Hazra — Wed, 16 Sep 2026 19:20:11 +0000

Data orchestration became interesting again, although I am not sure it ever stopped being complicated.

In July, Prefect announced that it was acquiring Dagster Labs. The announcement says Dagster will keep its name, support, and open-source license, with no migration required for customers. In September, Kestra released version 2.0 with a rewritten engine, separate control and data planes, remote workers, and flows that agents can call as tools.

Naturally, the familiar question returned: which orchestrator should we use?

The Reddit discussion includes almost every answer you would expect. Stay with Dagster. Use Airflow because it is established. Choose a managed service. Try something smaller for a lean team. A few vendors also arrived to recommend their own products, because some laws of distributed systems are social.

I think the product comparison is starting one step too late.

An orchestrator encodes how a team owns work. The syntax and user interface matter, but the bigger decision is who defines workflows, where code runs, how failures are recovered, and which boundaries the platform enforces.

The DAG is the easy part

Most tools can express this:

Extract
   |
   v
Transform
   |
   v
Validate
   |
   v
Publish

A demo usually ends when those four boxes turn green. Production starts when the second box runs for five hours, the third box discovers bad data after the publish step, and someone needs to backfill Tuesday without rerunning Wednesday.

That is where orchestrators differ in ways a feature matrix struggles to capture.

Does the platform think in tasks, assets, events, or deployments? Does it store enough state to reconstruct what happened? Can one team update a workflow without gaining access to every credential used by the worker? Can workloads run inside the network where the data already lives? How are upgrades tested, rolled back, and supported?

The answers shape team behaviour. They also determine who gets paged.

Start with the ownership boundary

Before comparing Airflow, Dagster, Prefect, Kestra, or a cloud-native scheduler, I would draw the boundary:

Platform team owns
  - control plane
  - identity and secrets
  - worker environments
  - deployment path
  - shared monitoring

Data teams own
  - workflow definitions
  - transformation code
  - data-quality rules
  - schedules and dependencies
  - domain runbooks

This is only one model, but it makes the questions concrete. If every data team needs the platform team to install a Python dependency, self-service is mostly a logo. If every workflow author can run arbitrary code with shared production credentials, self-service has gone too far in the other direction.

The orchestrator should make the desired boundary easier to enforce.

Kestra 2.0 is interesting in this context because its release notes describe workers connecting to the control plane through an outbound gRPC stream. User code runs in the data plane, and workers can sit in another region, cloud, or outbound-only network. That is more than an engine detail. It supports an operating model where a central team manages orchestration while execution stays near the workload.

It also comes with migration work. The 2.0 release removes or changes several constructs, requires an upgrade through 1.3.x, and changes the default behaviour for unmatched worker routing from waiting to failing. Those details belong in the decision because an orchestrator is a long-lived operational dependency, not a library we casually swap on Friday afternoon.

Decide what the system is organizing

The task-versus-asset distinction is another operating choice.

A task-oriented model asks, "What should run next?" It maps naturally to jobs, scripts, APIs, and infrastructure operations. Teams with mixed workloads often find this easy to reason about.

An asset-oriented model asks, "What data should exist, and what does it depend on?" That can make partitions, lineage, freshness, and backfills easier to express for data-heavy platforms.

Neither model is universally better. The useful question is what engineers spend their time debugging.

If incidents usually sound like "this job did not run," task state may be the natural center. If they sound like "the customer dimension is missing yesterday's partition," asset state may be more useful. If the same platform also runs infrastructure automation, ML training, and API workflows, a strongly data-specific abstraction can become awkward.

Pick the model that matches the failure language of the team.

An acquisition is a signal, not a migration plan

Prefect's acquisition of Dagster creates legitimate questions about long-term product direction. It does not, by itself, make a running Dagster deployment unsafe.

The public announcement commits to continued support, the existing name, and the existing open-source license. That is the fact available today. Future convergence is a possibility, not a documented migration requirement.

I would respond with boring engineering work:

pin the current version and test upgrades in a representative environment;
document the APIs, metadata, and deployment assumptions that create lock-in;
keep transformation logic outside orchestration definitions where practical;
export workflow and run metadata needed for audit or migration;
define the event that would trigger a reevaluation.

That last item is useful. "We feel nervous" is difficult to act on. "Security fixes fall outside our required window" or "the supported deployment no longer fits our network boundary" is a decision trigger.

The same discipline applies to every orchestrator. Project ownership can change. Cloud pricing can change. A managed feature can move tiers. Even a stable open-source project can become difficult for a small team to operate.

An exit path is part of the architecture.

Run a failure-focused proof of concept

Happy-path evaluations flatter every product. I would test a small workflow and then deliberately make it unpleasant.

Use one hourly ingestion, one daily aggregation, and one data-quality rule. Add a partitioned backfill. Run a task longer than the worker lifetime. Rotate its credential. Remove network access halfway through a run. Deploy an incompatible workflow change and roll it back.

Then score the products on the work the team will actually own:

Decision area	Question to answer
Deployment	Can we upgrade and roll back without improvisation?
Isolation	Can workloads use separate identities and dependencies?
Recovery	Can an operator restart the correct unit without duplicating data?
Backfills	Can historical work run without starving current schedules?
Observability	Can we identify the failed data, code version, and owner quickly?
Self-service	Can a team ship safely without platform tickets?
Cost	What grows with workflow count, run count, retention, and worker size?
Exit	How much domain logic is trapped in the orchestrator?

Two days of deliberate failure will tell you more than two weeks of clicking through feature pages.

My recommendation changes with the starting point

For an organization already running Airflow reliably, I would keep it unless there is a measured problem. A newer orchestration model may look cleaner, but migration creates parallel operations, retraining, rewritten workflows, and a new failure surface. Familiarity is not glamorous. It is still an asset.

For a new data platform, I would compare one task-oriented and one asset-oriented option using a real pipeline. If the team is small, include the cost of operating the control plane, not just the speed of writing the first workflow. A managed service may be cheaper than months of occasional maintenance even when its invoice is higher.

For workloads spread across restricted networks or multiple clouds, I would test the control-plane and worker boundary first. Kestra 2.0 now makes an explicit architectural claim in that area. The PoC should verify identity, connectivity, failure isolation, and upgrade behaviour rather than accepting the diagram.

For a team whose warehouse or data platform already provides adequate scheduling, I would start there. Another orchestrator must earn its database, deployment, alerting, and support burden.

The orchestrator decision is not Airflow versus Dagster versus Prefect versus Kestra. It is centralized versus federated ownership, tasks versus assets, shared execution versus isolated workers, and product convenience versus operational control.

Choose those boundaries first. The product shortlist usually becomes much smaller after that.

Your Data Engineering Roadmap Is Probably Too Long

Sudeep Hazra — Wed, 16 Sep 2026 19:19:04 +0000

A data engineering roadmap crossed my feed recently. It is thoughtful, detailed, and 500+ lines long. The associated Reddit discussion had a predictable reaction: useful reference, terrifying learning plan.

That distinction matters.

A map can show every road in a country. It does not mean you need to drive every road before you are allowed to leave home.

The Data Engineering Roadmap 2026 covers software engineering, Python, SQL, operational databases, warehouses, lakehouse storage, orchestration, observability, security, and cost. As an inventory of the field, that is helpful. As a checklist for becoming employable, it is too easy to read it as an entrance exam that never ends.

Several engineers in the Reddit discussion made the same point in different ways. Learn SQL, Python, modelling, and the fundamentals of transformation. Add tools when the problem asks for them. The disagreement was mostly about how much product knowledge someone needs before starting.

My view is simpler: learn one complete data system before collecting ten disconnected technologies.

Tools are the visible part of the job

Job descriptions make data engineering look like a shopping list:

Python + SQL + Spark + Kafka + Airflow
+ dbt + Snowflake + Databricks + AWS
+ whatever was added to the platform last Tuesday

The list is visible because product names are easy to search for. The harder skills hide underneath it.

Can you tell whether a source is producing inserts, updates, or replacements? Can you make a load safe to retry? What happens when a column changes type? How do you backfill three months without corrupting today's incremental run? Who gets alerted when the pipeline finishes successfully but loads zero rows?

Those questions survive tool changes.

Airflow can become Dagster. Redshift can become Snowflake. Spark can become a warehouse query or a small Python process because the data was never large enough to justify a cluster. The names move around. Ordering, idempotency, schema evolution, reconciliation, and operational ownership remain.

That is why a roadmap built around products creates a strange learning pattern. A person can finish six courses and still not know how to recover a failed pipeline. They have seen every component, but they have not owned a system.

Build one pipeline with consequences

I would structure the learning path around one modest project:

PostgreSQL source
       |
       v
Incremental extraction
       |
       v
Object storage as Parquet
       |
       v
Warehouse tables
       |
       v
Quality checks and reconciliation

The business story can be ordinary: orders, payments, support tickets, or device events. Ordinary is good. The learning comes from making the pipeline dependable, not from inventing a futuristic use case.

Start with a full load. Record the source row count, loaded row count, start time, end time, and status. Run it twice and prove that the second run does not duplicate data.

Then add incremental processing. Choose a watermark and explain why it is safe. If you use updated_at, decide what happens when records arrive late or a source clock is wrong. If you use a monotonically increasing ID, decide how updates are detected. There is no magic watermark, only a trade-off you can defend.

Next, break the schema on purpose. Add a nullable column. Rename another one. Change a numeric field into text. The pipeline should either handle the change or fail with enough evidence for someone to diagnose it. A silent partial load does not count as resilience (it counts as tomorrow's incident).

Finally, add a backfill that can run alongside the current schedule. Keep historical and incremental state separate. Reconcile totals after the backfill, then document how to restart it.

This single project teaches more useful engineering than a folder of disconnected tutorials because every new decision affects something already running.

Learn in layers, not brands

The order I would use is deliberately boring.

1. Data correctness

Begin with SQL, data modelling, transactions, indexes, query plans, and the difference between an operational schema and an analytical one. Learn how nulls, duplicates, time zones, and changing business definitions damage results.

Python matters here, but mainly as software. Use functions, tests, type hints, logging, dependency management, and a command-line entry point. A notebook is useful for exploration. It is a poor substitute for a repeatable job.

2. Pipeline behaviour

Add scheduling, retries, idempotency, checkpoints, and backfills. Learn the difference between a task completing and the data being correct. Store enough run metadata to explain what happened without reading raw logs for an hour.

This is also the right time to learn an orchestrator. Pick one. The goal is to understand dependencies, scheduling, state, retries, and failure recovery. You do not need equal fluency in Airflow, Dagster, Prefect, Kestra, and every managed cloud service.

3. Scale and distribution

Only after the single-node version becomes limiting would I add Spark, Kafka, or a lakehouse table format. Otherwise, it is difficult to separate the complexity of the problem from the complexity of the platform.

When Spark enters the project, measure why. Is the input too large for memory? Is parallelism shortening a batch window? Does the team already run the platform? "Spark appears in many job descriptions" is a career reason to understand it, but it is not an architecture reason to deploy it.

4. Production ownership

Security, cost, deployment, observability, and support are not advanced electives. They are what turns a pipeline into a service.

Use workload identity instead of static credentials. Set a budget alert. Define freshness and correctness checks. Write a runbook for the two failures most likely to happen. Package the project so another engineer can run it.

That final step changes the portfolio from "I followed a tutorial" to "I can own a data workload."

Breadth still matters, just later

There is a reasonable counterargument. Engineers often join environments with a fixed stack, and hiring filters still look for product names. A consultant may need to compare several platforms quickly. Breadth has career value.

The mistake is treating breadth and depth as competing destinations. They are a sequence.

Build depth with one stack. Then map nearby tools onto concepts you already understand:

Concept	First implementation	What to compare later
Orchestration	One scheduler	State model, retries, deployment, access control
Storage	One warehouse and object store	Cost, isolation, schema evolution, query patterns
Transformation	SQL and one processing engine	Pushdown, testing, lineage, scale
Ingestion	Batch first	CDC semantics, ordering, replay, source impact
Operations	Logs and run metadata	Metrics, tracing, alert routing, SLOs

Now a second tool is not another syllabus. It is a comparison against known constraints.

A roadmap should produce decisions

The best learning project leaves behind evidence of judgment:

why the pipeline is batch rather than streaming;
how retries avoid duplicates;
what happens when the schema changes;
how historical loads differ from current ingestion;
which metric tells you the data is late;
when the current design would stop being enough.

An interviewer can discuss those decisions. A future teammate can review them. More importantly, you can reuse the reasoning when the product names change.

The long roadmap is still useful. Keep it as an atlas. Use it to notice gaps and choose the next area to explore.

For the actual journey, pick one source, one destination, and one pipeline with consequences. Make it correct. Make it restartable. Make it understandable by someone else.

Then add the next tool because the system needs it, not because the roadmap had another box.