DEV Community: Sudeep Hazra

Your Kafka cluster is healthy. But the event is still missing

Sudeep Hazra — Sat, 26 Sep 2026 08:42:01 +0000

A recent r/dataengineering question described a familiar setup: many microservices publish JSON events to Kafka, each project has an expected throughput, and the business wants an alert when those events stop arriving.

The natural response is to look for a Kafka monitoring tool. That is useful, but it solves only part of the problem.

A Kafka dashboard can tell you that the brokers are available, partitions have leaders, replicas are in sync, and consumers are not falling behind. Every panel can be green while an upstream service has stopped producing the one event the business needs.

The system is healthy. The data product is not.

That distinction changes what we need to monitor.

Kafka health and business health are different signals

Apache Kafka's monitoring guidance covers the platform well. It recommends watching message and byte rates, request latency, fetch rates, replica state, and consumer lag. Those metrics answer important operational questions:

Can producers reach the cluster?
Are brokers accepting requests?
Are replicas keeping up?
Are consumers processing records fast enough?
Is one partition behaving differently from the others?

They do not answer whether the expected invoice.created event arrived for a particular market during the last fifteen minutes.

Consumer lag is a good example. A lag of zero sounds healthy. It can also mean that no records were produced. If the source application failed before publishing, the consumer has nothing to read and therefore nothing to lag behind.

The same problem appears with throughput. An aggregate topic rate may look normal while one tenant, event type, or producer has gone silent. A busy topic can hide a very specific outage.

I would separate monitoring into four layers:

Kafka infrastructure
        ↓
Message flow
        ↓
Event contract
        ↓
Business expectation

Each layer needs different evidence and a different owner.

Layer 1: prove the platform can carry messages

The first layer is normal Kafka operations. Watch broker availability, request errors, under-replicated partitions, ISR changes, disk pressure, network saturation, produce latency, fetch latency, and authentication failures.

This is where JMX exporters, a managed Kafka metrics API, Prometheus, Grafana, or an observability platform fit. The product matters less than collecting the right broker and client signals.

Consumer lag belongs here too, although I would treat it as a flow signal rather than a complete service-level indicator. Kafka exposes records-lag, records-lag-max, fetch rates, commit latency, and time between polls. Those metrics can identify a slow or stuck consumer.

An alert such as this is useful:

consumer lag is increasing
AND
records consumed per second is below the normal range
FOR ten minutes

It tells us that records exist and the consumer is not keeping up. It still says nothing about records that were never published.

Layer 2: prove messages move through the expected path

The next layer measures flow at boundaries. For every important producer and consumer, record a small set of counters and timestamps:

events produced
events accepted by Kafka
events consumed
events processed successfully
events rejected or sent to a dead-letter path
timestamp of the newest event

The labels need care. service, event_type, environment, and perhaps region are usually manageable. Raw customer IDs, transaction IDs, and message IDs are not. Putting high-cardinality business identifiers into a metrics system is an efficient way to turn an observability improvement into a cost incident.

Keep per-message identifiers in logs or traces. Keep metrics dimensions bounded.

OpenTelemetry's messaging conventions provide common attributes for messaging systems, destinations, operations, consumer groups, and message context. The conventions are still marked as development in several areas, so I would pin the version used by the instrumentation instead of assuming the attribute names will never change.

Tracing is helpful when a business operation crosses several services. A correlation or conversation ID can connect the API request, producer span, Kafka operation, consumer span, and downstream write. That gives an engineer a route through the failure instead of a collection of unrelated charts.

Tracing every message may be too expensive at high volume. Sampling is reasonable for diagnostics. Counts and freshness metrics should remain complete because they drive alerts.

Layer 3: prove the event is usable

Arrival is not success.

A producer can publish malformed JSON at the expected rate. A serializer can omit a required field. A schema change can preserve valid syntax while changing the meaning of a value. The throughput chart will look excellent right up to the point where someone opens the downstream report.

Validate the contract where ownership is clearest. That may be in the producer before publish, in a schema registry, or at the consumer boundary. Record at least:

schema validation failures
unsupported schema versions
deserialization failures
required-field failures
duplicate or out-of-order events when those conditions matter
dead-letter volume and age

Do not collapse all of these into processing_error_total. The response to a broken schema is different from the response to a timeout writing to a database.

I would also resist putting the full payload into observability events. It creates a second, poorly governed copy of potentially sensitive data. Record the failure category, event type, schema version, and a safe lookup reference. Keep payload access inside the data platform's normal security boundary.

Layer 4: encode the business expectation

This is the part a Kafka dashboard cannot infer.

"Expected TPS" sounds like a threshold, but a single number is rarely enough. Traffic changes by hour, weekday, region, and business calendar. Some event types are continuous. Others arrive in bursts after a batch closes. A flat minimum can create noise during quiet periods and miss a partial outage during busy ones.

Represent the expectation explicitly:

event_type: invoice.created
producer: billing-service
environment: production
schedule: weekdays 08:00-20:00 Europe/London
minimum_events: 300
window: 5m
maximum_silence: 10m
owner: billing-platform
severity: high
runbook: https://example.internal/runbooks/invoice-events

This is configuration, not dashboard decoration. Put it in version control. Review changes. Give each expectation an owner.

The evaluation process can be simple:

expected event rules
        +
observed event counters and freshness
        ↓
evaluation job
        ↓
pass, warn, fail, or no-data
        ↓
alert with owner and runbook

The no-data state matters. Monitoring systems often treat a missing time series differently from a zero value. Your evaluator must decide what absence means for each rule. Prometheus alerting rules support duration-based conditions through for, which helps avoid paging on a brief gap. The business rule still has to define how long a gap is acceptable.

Reconcile counts instead of trusting one point

For important flows, compare counts across boundaries over the same business window:

producer accepted:  100,000
Kafka observed:      99,998
consumer processed:  99,990
target committed:    99,987

Those numbers do not prove which eight or thirteen records are missing, but they tell you where to investigate.

Use event IDs or business keys for periodic reconciliation when exact completeness matters. Counters are operational signals. Reconciliation is evidence.

The window must account for retries and late arrivals. Comparing two live counters at the current second will generate false differences because each stage observes the event at a different time. Close a window, allow a defined lateness period, then evaluate it. Streaming does not remove the need for accounting.

Alert the team that can act

The Kafka platform team owns broker capacity, replication, cluster access, and platform availability. The producer team owns whether it emits the promised event. The consumer team owns processing and downstream delivery. A data product owner may own the business expectation.

Sending every alert to the Kafka team recreates the old middleware support queue with newer software.

An alert should include:

what expectation failed
when the last valid event arrived
expected and observed counts
the affected producer, topic, event type, and consumer
whether Kafka itself is healthy
the owning team
the first diagnostic query or dashboard
the runbook

"Kafka events missing" is not enough. "invoice.created from billing-service has been silent for 12 minutes; brokers are healthy and other producers are active" is actionable.

Start with the important flows

I would not begin by buying a broad data-observability platform or instrumenting every event type. Start with the ten flows whose absence causes money, compliance, or customer impact.

For each one, define:

The event contract.
The expected schedule and volume.
The maximum acceptable silence.
The producer and consumer owners.
The reconciliation rule.
The response when the rule fails.

Then use the monitoring stack already in place. Add a new product only when the existing stack cannot express, evaluate, or route these rules without unreasonable work.

The useful dashboard is not the one with the most Kafka metrics. It is the one that can distinguish a broken broker, a slow consumer, an invalid event, and a producer that stopped doing its job.

Kafka health is necessary. The business still needs proof that the right event arrived.

Your agent does not need to write an essay to make a decision

Sudeep Hazra — Sat, 26 Sep 2026 08:15:32 +0000

Something interesting appeared in a recent LocalLLaMA project showcase. Several projects were exploring small models that do not generate prose. They take a state, a question, and a fixed set of choices, then return a probability for each choice.

That sounds less impressive than an autonomous agent narrating its plan. It may also be exactly what many agent loops need.

Consider the decisions inside a typical tool-using workflow:

Which tool should handle this request?
Is this command safe enough to run automatically?
Did the previous step succeed?
Should the workflow retry, stop, or ask a person?
Which retrieved document should be examined next?

These are closed-set decisions. Asking a generative model to produce a paragraph, parsing that paragraph, and hoping it stayed inside the allowed choices is a surprisingly elaborate way to select one value from an enum.

My current view is simple: use generative models for open-ended reasoning and content. Use constrained decision components when the answer space is already known.

Generation and decision are different jobs

A generative model is useful when the output cannot be enumerated in advance. Writing a migration plan, diagnosing an unfamiliar failure, explaining a trade-off, or proposing a patch all need flexible output.

Routing a ticket to one of twelve queues does not.

We often use the same large model for both jobs because the API is convenient. The model receives a prompt such as:

Choose one action: retry, stop, escalate.
Return JSON only.

The application then validates the response, repairs malformed JSON, rejects invented actions, and perhaps asks the same model to try again. We added a language generator, then built a fence around its language generation.

Structured output improves the interface, but it does not change the underlying job. The system still generates tokens to choose from a fixed set.

Projects such as Contrastive Language Models approach the problem differently. CLM encodes the current state and candidate actions, scores their relationship, and returns a distribution over the supplied actions. The repository describes uses such as tool routing, trajectory ranking, and retrieval shortlisting.

Decider takes another route. It reads a state and typed questions, then produces probabilities for fixed choices, yes-or-no decisions, or described score levels in one pass. Its core promise is deliberately narrow: no free-form decoding and no output outside the choices supplied by the application.

These projects are early and their benchmark claims need independent validation. The architecture is still worth paying attention to.

The application should own the choice set

In a production agent, the model should not decide which actions exist. The application should.

application state
        +
allowed actions
        ↓
decision component
        ↓
probabilities
        ↓
policy and threshold
        ↓
execute, ask, or stop

That boundary gives us several useful properties.

First, impossible actions stay impossible. If delete_customer is not in the candidate set, the decision component cannot select it. This is stronger than telling a model in a prompt not to mention deletion.

Second, the result is inspectable. A distribution such as this is easier to evaluate than a persuasive paragraph:

{
  "retry": 0.18,
  "stop": 0.07,
  "escalate": 0.75
}

Third, the application can apply policy after inference. A high score is evidence, not authority.

if decision == "retry" and probability >= 0.90 and retry_count < 2:
    retry()
else:
    request_review()

The model proposes. Code decides what that proposal is allowed to do.

Probability is useful only when it is calibrated

A number that looks like confidence is not automatically a trustworthy probability.

If a model assigns roughly 0.8 to one hundred decisions, we would like about eighty of them to be correct. That is calibration. A model can have good top-choice accuracy and still be overconfident, which makes threshold-based automation dangerous.

This is where a typed decision interface improves the engineering conversation. We can measure:

accuracy by decision type
false-allow and false-deny rates
calibration error
coverage at a chosen threshold
latency and cost per decision
performance when none of the supplied choices is correct

The last case is easy to miss. A closed set can be wrong. If the workflow offers retry, stop, and escalate, but the correct action is refresh_credentials, the model cannot repair the application's incomplete choice set.

Include an unknown or none_of_the_above option where appropriate. More importantly, evaluate whether it is selected when the input falls outside the known cases.

The threshold also depends on consequence. A 0.75 routing decision may be acceptable for choosing a knowledge-base category. It is not enough evidence to run a destructive infrastructure command.

Guardrails belong around the tool, not inside the prompt

The decision model is not a security boundary.

The OpenAI Agents SDK guardrail documentation makes a useful separation: tool input guardrails run before execution, tool output guardrails run after execution, and either can reject content or halt the run. The SDK also documents important coverage limits for different tool types.

The exact framework is less important than the placement. Authorization and safety checks must sit on the execution path.

model chooses candidate action
        ↓
schema validation
        ↓
authorization check
        ↓
risk policy
        ↓
human approval if required
        ↓
tool execution
        ↓
result validation

Do not ask the model, "Is the user allowed to delete this resource?" when the target system can answer that question from its own permission model. Do not treat a 0.99 probability as permission. Use identity, authorization, resource state, and explicit policy.

A decision component can help classify risk or select a route. It cannot grant authority the caller does not have.

Small decisions can make large agents easier to operate

Agent diagrams tend to focus on the large reasoning step. Production failures often happen in the small transitions around it.

Should the agent retry after a timeout? Was the tool response complete? Does the evidence support the conclusion? Is the next operation reversible? Those decisions determine whether an isolated model error becomes a repeated side effect.

Breaking them out creates observable control points:

request
   ↓
route decision
   ↓
reasoning or retrieval
   ↓
action decision
   ↓
policy check
   ↓
tool
   ↓
success decision
   ↓
continue or stop

Each decision can have its own dataset, threshold, fallback, and owner. That is less magical than one agent prompt. It is also much easier to test.

This direction fits the broader advice in Anthropic's Building effective agents: start with the simplest workable pattern, prefer predefined workflows when the path is known, and add agent autonomy when flexibility justifies the added cost and risk.

The interesting part is not replacing every generative call with a classifier. It is noticing which calls were never generative problems in the first place.

Evaluation needs real workflow data

A benchmark can tell us whether a model distinguishes choices in a prepared dataset. It cannot tell us whether those choices represent the messy states in our system.

I would build the evaluation set from production-shaped examples:

normal cases that should pass automatically
ambiguous cases that should request review
rare failures that previously caused incidents
adversarial input that tries to steer the decision
stale or incomplete context
cases where the correct choice is absent
repeated attempts after a tool failure

Label them with the decision that the workflow should take, not the explanation we hope the model writes.

Then compare at least three baselines:

Deterministic rules.
A general generative model with structured output.
A specialized or constrained decision model.

Rules may win. For example, retry_count >= 2 does not need inference. A model becomes useful when the state contains language or evidence that code cannot classify reliably with a small, stable rule set.

Run shadow evaluations before granting automation. Record the model's decision and probability without executing it. Compare that output with the actual operator or workflow decision. This shows where a threshold would automate safely and where it would merely automate confidence.

Do not create a second platform by accident

There is a predictable failure mode here. A team sees specialized decision models, creates a separate serving stack, a feature store, a training pipeline, an evaluation service, and a new control plane before proving that one decision benefits from any of it.

Start with one bounded choice that occurs frequently enough to matter.

Tool routing is a reasonable candidate. Define the allowed tools, create a representative test set, compare rules and models, and measure error cost. Serve a smaller model only if it produces a material latency, cost, privacy, or accuracy benefit.

For low-volume workflows, a general model with strict structured output may remain the simplest operational choice. Fewer services can be worth a little extra inference cost.

The recommendation changes when decision calls dominate the loop. If an agent makes dozens of small choices per task, generation latency and token cost accumulate. A fast decision component can then reduce both, while giving the application a consistent probability interface.

Keep the agent for the parts that need an agent

Open-ended work still benefits from capable generative models. A decision model will not investigate an unfamiliar production failure, compare architecture options, or write a useful migration plan.

It can decide which diagnostic to run next from an approved list. It can estimate whether the evidence supports another step. It can route uncertainty to a person before the workflow turns a weak guess into an action.

That is enough.

An agent does not become less intelligent because some decisions move into smaller, constrained components. The system becomes clearer about where intelligence is needed, where policy belongs, and where ordinary code is still the better tool.

If the answer must be one of five choices, make the system choose one of five choices. Save the essay for the part that needs an explanation.

Before You Add an Event Bus, Name the Failure

Sudeep Hazra — Mon, 21 Sep 2026 11:53:00 +0000

A discussion about event-driven architecture caught my attention because the replies disagreed on something more useful than Kafka versus a database. People were describing different problems with the same word: event.

One team needs to run a slow task after a transaction. Another needs to tell several independent systems that a business fact has changed. A third needs an audit history it can replay. All three may produce a message, but they have different requirements for ownership, ordering, and recovery.

I would start by naming the failure that the current design cannot handle. If that failure is a web request waiting on slow work, a database-backed job may be enough. If independent consumers need a durable record of a business change, publishing an event may be worth the extra machinery. That gives the team a reason for the extra component.

A slow operation is not automatically an event

Imagine an application that accepts an order and generates a PDF receipt. The user needs to know whether the order was accepted. They do not need to hold the connection open while the PDF renderer runs.

The first design I would test is simple:

The order and job are written in one database transaction. If the transaction rolls back, neither exists. If the renderer fails, the job remains visible for a retry or investigation. The application can show the user an order state and a separate receipt state. There is no broker to operate merely because work happens later.

That job still needs engineering. Give it a stable ID, an attempt count, a timeout, and a clear terminal failure state. Decide how a worker claims a job and how another worker takes over after a crash. PostgreSQL documents SKIP LOCKED as useful for avoiding contention among consumers of a queue-like table, while warning that it gives an inconsistent view of rows. That is a tool for workers, not a way to make arbitrary reporting queries faster.

A database queue has limits. Heavy fan-out, many unrelated consumers, a large backlog, or retention far beyond the operational life of a job can make the table and its cleanup awkward. Those are measurable reasons to consider messaging infrastructure. “We want async” is not yet one.

Publish facts when other owners need them

Now add a finance service, a customer notification service, and a fulfillment system. Each has its own release schedule and its own idea of what to do when an order is paid. The order service should not need to call all three during the payment request and wait for their health before it can record payment.

OrderPaid can be a useful event here. It says something that has happened, with an order ID, event ID, and version of the contract. The consumers decide what that fact means for their own systems. This is a better reason for an event than the mere existence of multiple functions in one application.

The awkward part is the boundary between the order database and the broker. This code is unsafe:

The process can die between those lines. Reversing the lines creates the opposite problem: a consumer may act on an event for a payment that never committed. AWS describes the transactional outbox pattern for this dual-write problem. Store the business update and an outbox record in the same transaction; a separate publisher sends committed records to the broker.

The outbox improves the handoff. It does not make the rest of the system atomic. The publisher can send an event and crash before marking it delivered. The consumer must cope with that event arriving again. A stable event ID and a consumer-side record of processed IDs are more useful than a diagram that promises “exactly once” without saying where the guarantee ends.

For example, a notification consumer can store OrderPaid:event-123 before or with the state change that schedules an email. If it sees the same event again, it can skip the duplicate scheduling step. A payment consumer has a higher bar: it must establish whether an external charge already succeeded before retrying. Its idempotency boundary is the payment provider and the local record together, not only the broker.

Decide what the user must know now

Asynchrony changes the product contract. Suppose checkout returns “payment complete” immediately, but fulfillment might reject the order ten minutes later because stock ran out. That may be a valid business process, but the UI and support team need to know what “complete” meant.

For each step, I would write down three states:

Question	Example answer
What is committed before the response?	Payment authorization and order record.
What can finish later?	Receipt generation and loyalty points.
What happens if later work fails?	Retry, visible pending state, then support action.

If the next operation must succeed before the user can proceed, a synchronous API call may be easier to reason about. An event can still be emitted afterward for observers. If the user can wait, an accepted response plus a status endpoint can make the delay explicit. AWS's asynchronous communication guidance describes that claim-check approach: acknowledge the request, return an identifier, and let the client retrieve the later result.

The delay has a user-facing cost: the text on a screen, the retry a user may press, and the question a support engineer has to answer.

Delivery guarantees do not finish the design

The team needs to decide what duplicates or ordering would do to the business operation before selecting a broker. A standard Amazon SQS queue, for example, delivers messages at least once. The same message may be received again. Its standard queue type also gives best-effort ordering.

Those properties are fine for many tasks. A consumer updating a current projection can ignore an event it has already processed. A consumer applying account balance deltas cannot casually apply the same delta twice. It may also need per-account ordering. Picking a FIFO queue can help with ordering and broker deduplication, but it does not remove the need to reason about retries across external systems and the full business workflow.

I would ask the following before choosing the transport:

Is this a command for one worker, or a fact several owners may consume?
What state is committed when the message is created?
Can the same message be processed twice without damage?
Must messages be ordered globally, per entity, or not at all?
How long must a consumer be able to recover after an outage?
Who notices a stuck consumer, and how is the backlog cleared?

The answers narrow the choice considerably. A one-owner receipt job does not need the same infrastructure as a company-wide change stream. A durable business event deserves more thought than a callback disguised as a topic.

Operate the failure path before scaling the happy path

Events make producers and consumers independent in useful ways. They also make a transaction harder to follow across time. An API request may be complete while the business process is still pending in another service. A dashboard that shows only broker throughput cannot tell you whether orders are actually progressing.

I would give each event a correlation ID, a producer timestamp, a contract version, and an owning service. Then I would measure the delay from publication to the consumer's business outcome, not merely the time until a message is read. A dead-letter queue is a holding area for work that failed repeatedly. It is not a recovery plan until someone owns inspection, correction, and replay.

Changes to the event contract need similar care. If consumers deploy independently, a producer cannot assume everyone upgrades on the same day. Add fields in a compatible way, document their meaning, and test old consumers against new payloads. When the meaning of a fact changes, a new event version may be clearer than silently reusing an old name.

This is the cost side of the architecture decision. If a system has one application and one database, a job table may give the team all the separation it needs. If several autonomous systems depend on the same business fact and must recover independently, the broker, outbox, consumer idempotency, and operational tooling earn their place.

My rule is to write down the failure and the recovery path first. Once those are concrete, the decision between a worker, an API call, and an event is usually much less mysterious.

What Stays in Secrets Manager After Workload Identity?

Sudeep Hazra — Mon, 21 Sep 2026 11:41:03 +0000

I liked the question in a recent r/devops discussion: after moving workloads to identity federation, how many secrets are left?

The answer depends on where each credential crosses a trust boundary.

GitHub Actions can exchange an OIDC token for short lived cloud credentials. An application on AWS can use an IAM role instead of carrying an access key. AWS even supports temporary credentials for workloads outside AWS, provided you operate the certificate infrastructure needed to establish that trust.

Those changes remove a class of long lived credentials. They do not cause a payment provider, an older database, or a webhook sender to accept your cloud identity. Some secrets remain because the other system controls its own authentication method.

My recommendation is to migrate the paths that support federation, then give the remaining credentials more deliberate ownership. The useful measure is how much risk and work remain per credential, not the number of entries in a vault.

Draw the boundary before deleting anything

Consider a service that writes to S3, calls an external billing API, receives signed webhooks, and connects to an older SQL Server.

The S3 access key is a good candidate for removal. Give the workload an identity and a narrowly scoped role. The billing API key still exists because that API expects a key. The webhook signing secret still exists because the receiver must verify what the sender signed. The database password might remain until that particular server version, driver, and deployment can use another supported authentication path.

I would draw that as four separate edges:

The last line needs investigation, not an assumption. Some SQL Server deployments can use Microsoft Entra authentication; an older on premises deployment may have different constraints. Check the exact version and connection path before promising a passwordless migration.

There is another subtle point here. A workload role can authorize a service to read a secret from a manager. It does not remove the secret stored there. It improves how the service obtains that value and who may read it. The provider credential still needs scoping, rotation, revocation, and an owner.

A report saying “we use identity for all workloads” tells me little about the secrets inventory. I still want to know how each destination authenticates the caller.

The leftovers are often harder than the keys you removed

Cloud access keys are painful when they spread through CI settings and configuration files, but cloud platforms give us mature replacements. The remaining secrets may have awkward lifecycles.

One vendor lets you create a replacement key while the old key stays valid. Another gives you one active credential, so rotation requires a coordinated cutover. A webhook endpoint may accept more than one signing secret during a transition, or it may not. An appliance may require a manual change through a browser. A shared password may have several consumers that nobody documented.

These differences determine the operating procedure. A scheduled rotation button is useful only when the destination also changes and every consumer picks up the new value. AWS Secrets Manager's rotation documentation is explicit about updating both the stored secret and the database or service. Changing the vault entry alone is a reliable way to schedule an outage.

For each remaining credential, I would record:

Question	Why it matters
Who issues and revokes it?	Determines the actual recovery path.
Which workload uses it?	Limits access and exposes accidental sharing.
Can old and new values overlap?	Determines whether rotation can avoid downtime.
How does the workload reload it?	A new vault value is useless to a process holding the old one.
What happens if it leaks?	Sets urgency and containment steps.
Who can test the replacement?	Makes rotation an operation rather than a calendar reminder.

That inventory is more informative than “37 secrets remaining.” Two of those 37 might be broad production credentials with no tested revocation path. They deserve attention before twenty low privilege integration tokens.

Make migration reversible until the new path is proven

I would migrate one workload edge at a time. Start with a well understood cloud API call. Give the workload a role with only the permissions it needs, then run a real operation through the new identity. Check which principal appears in the audit trail. Check that the workload refreshes credentials before expiry. Run the job long enough to exercise a refresh, not merely its startup path.

Then remove the old key from the application's configuration and rerun the job. Only after that should you revoke the old key. Finally, search the places where copies tend to survive: CI secrets, deployment templates, local environment files, support scripts, and scheduled jobs. The old key may be unused by the main service but still active elsewhere.

For GitHub Actions, restrict which repository, branch, or environment can assume the cloud role. OIDC removes a stored cloud key, but a broad trust policy can still authorize the wrong workflow. The token exchange is only as narrow as the trust conditions and permissions on the role.

I would also test failure behavior. If federation is unavailable, does the job fail with a clear error? Does someone have a documented recovery path? Secretless authentication can still have a dependency on an identity provider, metadata endpoint, certificate authority, or token service. Those dependencies deserve monitoring.

Keep the secrets manager, but give it a smaller job

AWS describes Secrets Manager as a place for database credentials, application credentials, OAuth tokens, API keys, and other secrets. After federation, that list may shrink. The service still has a useful job: controlling access to values that cannot be replaced by an identity handshake.

I would organize the remaining inventory by system boundary and owner rather than by a generic prod/secrets folder. A billing key should be readable only by the billing workload. A webhook signing secret should be readable only by the verifier and rotation procedure. A database credential should have permissions that match the application's actual queries, with separate credentials for migration or administration.

Set rotation intervals according to the system's capability and exposure. Do not promise automatic rotation for a vendor that offers no safe API or overlapping credentials. For those cases, write a cutover runbook and rehearse it. A manual procedure that has been tested is better than an “automated” rotation that only changes one side of the connection.

There is a cost decision too. If a team has a small number of static credentials, it may not need another vault product or a custom broker. Reuse the secrets service already in the platform, provided it gives the team access control, audit history, and a workable recovery path. Additional infrastructure should solve a specific remaining problem.

What I would measure

The percentage of credentials replaced is easy to celebrate and easy to misread. I would track four operational results instead:

Which long lived cloud keys have actually been revoked?
Which remaining secrets have a named owner and known consumers?
Which credentials can be rotated without an outage, and when was that last tested?
How quickly can a leaked credential be identified and disabled?

Those answers tell us whether the migration changed production risk. They also expose the next improvement: a vendor integration to replace, a database auth path to modernize, or a shared key to split by workload.

Workload identity is worth adopting where it fits. It removes credentials that never needed to be stored in the first place. The interesting work begins after that, when the remaining list is short enough to read and inconvenient enough to deserve proper attention.

Your Orchestrator Is an Operating Model, Not a Feature List

Sudeep Hazra — Wed, 16 Sep 2026 19:20:11 +0000

Data orchestration became interesting again, although I am not sure it ever stopped being complicated.

In July, Prefect announced that it was acquiring Dagster Labs. The announcement says Dagster will keep its name, support, and open-source license, with no migration required for customers. In September, Kestra released version 2.0 with a rewritten engine, separate control and data planes, remote workers, and flows that agents can call as tools.

Naturally, the familiar question returned: which orchestrator should we use?

The Reddit discussion includes almost every answer you would expect. Stay with Dagster. Use Airflow because it is established. Choose a managed service. Try something smaller for a lean team. A few vendors also arrived to recommend their own products, because some laws of distributed systems are social.

I think the product comparison is starting one step too late.

An orchestrator encodes how a team owns work. The syntax and user interface matter, but the bigger decision is who defines workflows, where code runs, how failures are recovered, and which boundaries the platform enforces.

The DAG is the easy part

Most tools can express this:

Extract
   |
   v
Transform
   |
   v
Validate
   |
   v
Publish

A demo usually ends when those four boxes turn green. Production starts when the second box runs for five hours, the third box discovers bad data after the publish step, and someone needs to backfill Tuesday without rerunning Wednesday.

That is where orchestrators differ in ways a feature matrix struggles to capture.

Does the platform think in tasks, assets, events, or deployments? Does it store enough state to reconstruct what happened? Can one team update a workflow without gaining access to every credential used by the worker? Can workloads run inside the network where the data already lives? How are upgrades tested, rolled back, and supported?

The answers shape team behaviour. They also determine who gets paged.

Start with the ownership boundary

Before comparing Airflow, Dagster, Prefect, Kestra, or a cloud-native scheduler, I would draw the boundary:

Platform team owns
  - control plane
  - identity and secrets
  - worker environments
  - deployment path
  - shared monitoring

Data teams own
  - workflow definitions
  - transformation code
  - data-quality rules
  - schedules and dependencies
  - domain runbooks

This is only one model, but it makes the questions concrete. If every data team needs the platform team to install a Python dependency, self-service is mostly a logo. If every workflow author can run arbitrary code with shared production credentials, self-service has gone too far in the other direction.

The orchestrator should make the desired boundary easier to enforce.

Kestra 2.0 is interesting in this context because its release notes describe workers connecting to the control plane through an outbound gRPC stream. User code runs in the data plane, and workers can sit in another region, cloud, or outbound-only network. That is more than an engine detail. It supports an operating model where a central team manages orchestration while execution stays near the workload.

It also comes with migration work. The 2.0 release removes or changes several constructs, requires an upgrade through 1.3.x, and changes the default behaviour for unmatched worker routing from waiting to failing. Those details belong in the decision because an orchestrator is a long-lived operational dependency, not a library we casually swap on Friday afternoon.

Decide what the system is organizing

The task-versus-asset distinction is another operating choice.

A task-oriented model asks, "What should run next?" It maps naturally to jobs, scripts, APIs, and infrastructure operations. Teams with mixed workloads often find this easy to reason about.

An asset-oriented model asks, "What data should exist, and what does it depend on?" That can make partitions, lineage, freshness, and backfills easier to express for data-heavy platforms.

Neither model is universally better. The useful question is what engineers spend their time debugging.

If incidents usually sound like "this job did not run," task state may be the natural center. If they sound like "the customer dimension is missing yesterday's partition," asset state may be more useful. If the same platform also runs infrastructure automation, ML training, and API workflows, a strongly data-specific abstraction can become awkward.

Pick the model that matches the failure language of the team.

An acquisition is a signal, not a migration plan

Prefect's acquisition of Dagster creates legitimate questions about long-term product direction. It does not, by itself, make a running Dagster deployment unsafe.

The public announcement commits to continued support, the existing name, and the existing open-source license. That is the fact available today. Future convergence is a possibility, not a documented migration requirement.

I would respond with boring engineering work:

pin the current version and test upgrades in a representative environment;
document the APIs, metadata, and deployment assumptions that create lock-in;
keep transformation logic outside orchestration definitions where practical;
export workflow and run metadata needed for audit or migration;
define the event that would trigger a reevaluation.

That last item is useful. "We feel nervous" is difficult to act on. "Security fixes fall outside our required window" or "the supported deployment no longer fits our network boundary" is a decision trigger.

The same discipline applies to every orchestrator. Project ownership can change. Cloud pricing can change. A managed feature can move tiers. Even a stable open-source project can become difficult for a small team to operate.

An exit path is part of the architecture.

Run a failure-focused proof of concept

Happy-path evaluations flatter every product. I would test a small workflow and then deliberately make it unpleasant.

Use one hourly ingestion, one daily aggregation, and one data-quality rule. Add a partitioned backfill. Run a task longer than the worker lifetime. Rotate its credential. Remove network access halfway through a run. Deploy an incompatible workflow change and roll it back.

Then score the products on the work the team will actually own:

Decision area	Question to answer
Deployment	Can we upgrade and roll back without improvisation?
Isolation	Can workloads use separate identities and dependencies?
Recovery	Can an operator restart the correct unit without duplicating data?
Backfills	Can historical work run without starving current schedules?
Observability	Can we identify the failed data, code version, and owner quickly?
Self-service	Can a team ship safely without platform tickets?
Cost	What grows with workflow count, run count, retention, and worker size?
Exit	How much domain logic is trapped in the orchestrator?

Two days of deliberate failure will tell you more than two weeks of clicking through feature pages.

My recommendation changes with the starting point

For an organization already running Airflow reliably, I would keep it unless there is a measured problem. A newer orchestration model may look cleaner, but migration creates parallel operations, retraining, rewritten workflows, and a new failure surface. Familiarity is not glamorous. It is still an asset.

For a new data platform, I would compare one task-oriented and one asset-oriented option using a real pipeline. If the team is small, include the cost of operating the control plane, not just the speed of writing the first workflow. A managed service may be cheaper than months of occasional maintenance even when its invoice is higher.

For workloads spread across restricted networks or multiple clouds, I would test the control-plane and worker boundary first. Kestra 2.0 now makes an explicit architectural claim in that area. The PoC should verify identity, connectivity, failure isolation, and upgrade behaviour rather than accepting the diagram.

For a team whose warehouse or data platform already provides adequate scheduling, I would start there. Another orchestrator must earn its database, deployment, alerting, and support burden.

The orchestrator decision is not Airflow versus Dagster versus Prefect versus Kestra. It is centralized versus federated ownership, tasks versus assets, shared execution versus isolated workers, and product convenience versus operational control.

Choose those boundaries first. The product shortlist usually becomes much smaller after that.

Your Data Engineering Roadmap Is Probably Too Long

Sudeep Hazra — Wed, 16 Sep 2026 19:19:04 +0000

A data engineering roadmap crossed my feed recently. It is thoughtful, detailed, and 500+ lines long. The associated Reddit discussion had a predictable reaction: useful reference, terrifying learning plan.

That distinction matters.

A map can show every road in a country. It does not mean you need to drive every road before you are allowed to leave home.

The Data Engineering Roadmap 2026 covers software engineering, Python, SQL, operational databases, warehouses, lakehouse storage, orchestration, observability, security, and cost. As an inventory of the field, that is helpful. As a checklist for becoming employable, it is too easy to read it as an entrance exam that never ends.

Several engineers in the Reddit discussion made the same point in different ways. Learn SQL, Python, modelling, and the fundamentals of transformation. Add tools when the problem asks for them. The disagreement was mostly about how much product knowledge someone needs before starting.

My view is simpler: learn one complete data system before collecting ten disconnected technologies.

Tools are the visible part of the job

Job descriptions make data engineering look like a shopping list:

Python + SQL + Spark + Kafka + Airflow
+ dbt + Snowflake + Databricks + AWS
+ whatever was added to the platform last Tuesday

The list is visible because product names are easy to search for. The harder skills hide underneath it.

Can you tell whether a source is producing inserts, updates, or replacements? Can you make a load safe to retry? What happens when a column changes type? How do you backfill three months without corrupting today's incremental run? Who gets alerted when the pipeline finishes successfully but loads zero rows?

Those questions survive tool changes.

Airflow can become Dagster. Redshift can become Snowflake. Spark can become a warehouse query or a small Python process because the data was never large enough to justify a cluster. The names move around. Ordering, idempotency, schema evolution, reconciliation, and operational ownership remain.

That is why a roadmap built around products creates a strange learning pattern. A person can finish six courses and still not know how to recover a failed pipeline. They have seen every component, but they have not owned a system.

Build one pipeline with consequences

I would structure the learning path around one modest project:

PostgreSQL source
       |
       v
Incremental extraction
       |
       v
Object storage as Parquet
       |
       v
Warehouse tables
       |
       v
Quality checks and reconciliation

The business story can be ordinary: orders, payments, support tickets, or device events. Ordinary is good. The learning comes from making the pipeline dependable, not from inventing a futuristic use case.

Start with a full load. Record the source row count, loaded row count, start time, end time, and status. Run it twice and prove that the second run does not duplicate data.

Then add incremental processing. Choose a watermark and explain why it is safe. If you use updated_at, decide what happens when records arrive late or a source clock is wrong. If you use a monotonically increasing ID, decide how updates are detected. There is no magic watermark, only a trade-off you can defend.

Next, break the schema on purpose. Add a nullable column. Rename another one. Change a numeric field into text. The pipeline should either handle the change or fail with enough evidence for someone to diagnose it. A silent partial load does not count as resilience (it counts as tomorrow's incident).

Finally, add a backfill that can run alongside the current schedule. Keep historical and incremental state separate. Reconcile totals after the backfill, then document how to restart it.

This single project teaches more useful engineering than a folder of disconnected tutorials because every new decision affects something already running.

Learn in layers, not brands

The order I would use is deliberately boring.

1. Data correctness

Begin with SQL, data modelling, transactions, indexes, query plans, and the difference between an operational schema and an analytical one. Learn how nulls, duplicates, time zones, and changing business definitions damage results.

Python matters here, but mainly as software. Use functions, tests, type hints, logging, dependency management, and a command-line entry point. A notebook is useful for exploration. It is a poor substitute for a repeatable job.

2. Pipeline behaviour

Add scheduling, retries, idempotency, checkpoints, and backfills. Learn the difference between a task completing and the data being correct. Store enough run metadata to explain what happened without reading raw logs for an hour.

This is also the right time to learn an orchestrator. Pick one. The goal is to understand dependencies, scheduling, state, retries, and failure recovery. You do not need equal fluency in Airflow, Dagster, Prefect, Kestra, and every managed cloud service.

3. Scale and distribution

Only after the single-node version becomes limiting would I add Spark, Kafka, or a lakehouse table format. Otherwise, it is difficult to separate the complexity of the problem from the complexity of the platform.

When Spark enters the project, measure why. Is the input too large for memory? Is parallelism shortening a batch window? Does the team already run the platform? "Spark appears in many job descriptions" is a career reason to understand it, but it is not an architecture reason to deploy it.

4. Production ownership

Security, cost, deployment, observability, and support are not advanced electives. They are what turns a pipeline into a service.

Use workload identity instead of static credentials. Set a budget alert. Define freshness and correctness checks. Write a runbook for the two failures most likely to happen. Package the project so another engineer can run it.

That final step changes the portfolio from "I followed a tutorial" to "I can own a data workload."

Breadth still matters, just later

There is a reasonable counterargument. Engineers often join environments with a fixed stack, and hiring filters still look for product names. A consultant may need to compare several platforms quickly. Breadth has career value.

The mistake is treating breadth and depth as competing destinations. They are a sequence.

Build depth with one stack. Then map nearby tools onto concepts you already understand:

Concept	First implementation	What to compare later
Orchestration	One scheduler	State model, retries, deployment, access control
Storage	One warehouse and object store	Cost, isolation, schema evolution, query patterns
Transformation	SQL and one processing engine	Pushdown, testing, lineage, scale
Ingestion	Batch first	CDC semantics, ordering, replay, source impact
Operations	Logs and run metadata	Metrics, tracing, alert routing, SLOs

Now a second tool is not another syllabus. It is a comparison against known constraints.

A roadmap should produce decisions

The best learning project leaves behind evidence of judgment:

why the pipeline is batch rather than streaming;
how retries avoid duplicates;
what happens when the schema changes;
how historical loads differ from current ingestion;
which metric tells you the data is late;
when the current design would stop being enough.

An interviewer can discuss those decisions. A future teammate can review them. More importantly, you can reuse the reasoning when the product names change.

The long roadmap is still useful. Keep it as an atlas. Use it to notice gaps and choose the next area to explore.

For the actual journey, pick one source, one destination, and one pipeline with consequences. Make it correct. Make it restartable. Make it understandable by someone else.

Then add the next tool because the system needs it, not because the roadmap had another box.

AI Security Scanning Needs Evidence, Not Just More Agents

Sudeep Hazra — Sun, 13 Sep 2026 08:47:20 +0000

Google’s Mantis caught my attention because it points to a problem most AI security demos quietly walk around: finding a vulnerability is not the same as proving one exists.

That distinction matters. Security teams already live with noisy scanners, half-useful alerts, and findings that require someone experienced to separate a real exploit path from a theoretical complaint. Adding an LLM can improve that workflow. It can also make the scanner more fluent while being wrong in more elaborate ways.

That is not progress. That is just a better-written interruption.

The interesting part of Mantis is not the label “agentic vulnerability scanning.” Everyone is attaching agentic to things now. Apparently software is not allowed to have a normal workflow anymore. The useful part is the structure around grounding: repository context, history, threat models, reviewer stages, critic stages, sandboxed reproduction, and patches that are tied back to evidence.

That shape makes sense because vulnerability detection is not one job. It is several jobs that often get collapsed into one vague “scan the code” box.

The reproduction step is the part I would care about most.

If a system can show a working crash, a failing test, an exploit path, or a concrete data-flow issue, the review conversation changes. The security team is no longer reading model confidence. They are reading evidence. That does not remove human judgment, but it gives the human something useful to judge.

Most teams do not suffer because they have too few alerts. They suffer because the alerts arrive without enough context. Someone has to open the repository, understand the service boundary, trace input handling, inspect validation, check the framework behavior, look at previous fixes, decide whether the path is reachable, and then argue with the scanner’s output. That is expensive work.

AI can help with that work, but only if the system is designed as a workflow rather than a magic box.

For example, a scanner that says:

Possible SQL injection in UserController.java
is not enough. A better system should be able to say:

This route accepts user input here.
The value reaches this query builder here.
This sanitizer does not cover this pattern.
This is a minimal reproducer.
This is the failing test.
This is the proposed fix.
This is the residual uncertainty.

That last line matters. Residual uncertainty is not weakness. It is honesty. A security tool that pretends every finding is equally certain creates bad incentives. Engineers start ignoring it, security teams start tuning it down, and eventually the tool becomes background noise.

The useful promise of an agentic scanner is that it can break the work into stages and let each stage challenge the previous one. One agent may identify suspicious flows, whereas another may criticize the finding, and another may try to reproduce it. Another may propose a patch. A final stage may check whether the patch changes behavior outside the intended area.

That is much closer to how a careful human review works.

It also changes how I would think about model selection. Not every stage needs the biggest model available. Classification, deduplication, clustering similar findings, and summarizing repository structure are not the same as tracing a subtle authorization bypass through five layers of application code. A practical system should spend reasoning budget where reasoning is actually needed.

That is a very SRE-ish way to think about AI, and I mean that as a compliment. The model is not magic. It is one component in a workflow with cost, latency, permissions, state, logs, and failure modes.

The cost piece is easy to ignore in a research announcement and painful to ignore in a real engineering organization. If every pull request triggers a deep multi-agent investigation across a large monorepo, the bill will get interesting very quickly. The system needs triage. It needs cheap filters, expensive analysis only when justified, and clear rules for what runs synchronously in CI versus what runs asynchronously in a security pipeline.

There is also a timing question. Some checks belong directly in the developer loop. They should run fast, fail clearly, and produce a result while the developer still remembers what they changed. Other checks are better as background analysis. A deep investigation across service boundaries may be valuable, but it probably should not block every commit unless the organization is prepared for that operational cost.

This is where the design should separate developer feedback from security investigation.

Developer feedback needs speed and clarity. Security investigation needs depth and evidence. Trying to make one workflow satisfy both usually produces a system that is too slow for developers and too shallow for security teams. That is a familiar failure mode. We have seen it with static analysis, dependency scanning, data-quality checks, and policy-as-code.

AI does not remove that trade-off. It just makes it easier to hide for a while.

I would probably separate the workflow like this:

This keeps the expensive reasoning stages closer to the findings that deserve them.

There is still plenty of operational work hiding behind the nice diagram. The scanner needs sandboxing. It needs restricted network access. It needs deterministic handoffs between stages. It needs audit logs. It needs a way to prevent an aggressive false-positive filter from suppressing weak-looking findings that are actually real. It also needs ownership, because once a tool starts filing security bugs or proposing patches, someone has to decide what “good enough” means.

This is where security automation often becomes uncomfortable. A tool that only reports findings is easy to ignore. A tool that opens patches is harder to ignore, but also more dangerous. The patch might fix the immediate issue while changing behavior somewhere else. It might silence a test instead of fixing the cause. It might introduce a different vulnerability. It might be correct technically but wrong for the product’s authorization model.

Authorization bugs are a good example. They rarely live in one obvious line of code. The check may depend on route configuration, middleware behavior, tenant context, cached permissions, database filters, and assumptions in the UI. A scanner that sees only the controller method may miss the real boundary. A model with repository context may do better, but only if the system gives it the right evidence and then forces it to prove the path.

The same is true for deserialization, SSRF, file access, and dependency confusion. The interesting question is often not “does this function look suspicious?” It is “can untrusted input actually reach this dangerous capability under realistic conditions?” That requires reachability analysis, test construction, environment modeling, and sometimes domain knowledge about how the application is deployed.

This is where agents can be useful, but also where they can become overconfident. A fluent explanation of a possible exploit is not the same as exploitability. I would rather have a scanner that says “I cannot prove this yet” than one that files a confident but ungrounded critical finding.

So I would not give an AI scanner unlimited write access to a production codebase. I would start with evidence generation, then move to suggested patches behind review, then allow more automation only for narrow, well-tested classes of changes.

Something like:

Phase 1: identify and explain
Phase 2: reproduce with tests
Phase 3: suggest patches
Phase 4: auto-fix low-risk patterns
Phase 5: measure escaped defects and false negatives

The important part is that automation earns trust through measured behavior. Not vibes. Not a launch post. Measured behavior.

The metrics should reflect that. I would track more than “number of vulnerabilities found.” That metric is too easy to game. A useful program would track confirmed true positives, false-positive rate by category, time from finding to reproduction, time from reproduction to patch, developer review burden, escaped vulnerabilities, and whether suggested fixes survived regression testing.

Those numbers tell you whether the system is improving the security process or merely producing activity.

There is also a governance concern. If the scanner learns from internal code, writes summaries to disk, executes reproducers in sandboxes, and uses multiple models, the organization needs to know where sensitive data goes. Security tooling often has broad repository access by design. That makes data handling, retention, model-provider boundaries, and access logs part of the architecture, not an appendix.

I would treat an AI security scanner almost like a privileged internal service:

That is more work than running a command-line scanner. But if the system is going to reason over sensitive code and propose security patches, the extra discipline is not optional.

The bigger issue is that AI security tooling can easily become another alert generator. Teams do not need more findings. They need better evidence, better prioritization, and a shorter path from suspicion to validated fix.

That is where agentic workflows may actually help. Not because an agent can read a lot of files, although that helps. Not because it can write a patch, although that is useful. The real value is in connecting the steps that human reviewers already perform manually:

understand context
form a hypothesis
challenge it
reproduce it
and only then act

There is also a cultural side to this. Developers will trust the system faster if it explains itself in the language of the codebase. Security teams will trust it sooner if there is reproducible evidence. Engineering leaders will trust it faster if it reduces mean time to validated fix without flooding teams with noise. Those are different success metrics, and a serious platform needs to satisfy all three.

My current view is simple: AI belongs in security scanning when it is treated as an evidence-generation system.

The agent can suggest. The harness has to prove. That is enough for the first serious version.

Only after that, let us talk about autonomy.

| References:

Inspired by InfoQ’s coverage of Google Mantis: https://www.infoq.com/news/2026/09/google-mantis-vulnerability-scan/
Related trend context: InfoQ’s September 2026 AI, ML, and data engineering coverage highlighted agentic testing, security, and production-readiness themes.