You ask how to handle late-arriving events in your pipeline. The answer is clean, modern and sensible. It involves Kafka, a streaming framework, and a state store.
You run batch. Everything runs batch. The batch runs on an Airflow instance somebody set up in 2021, the transformations are four hundred stored procedures nobody wants to touch, and the team spent the last quarter learning dbt and is finally productive with it.
None of that was secret. You mentioned Airflow. You just mentioned it the way you would mention the weather.
🔍 Facts and constraints are different things
Here is the sentence most people write:
We use Airflow for orchestration and mostly work in dbt.
And here is what it does not say: whether any of that can change.
Read as a fact, "we use Airflow" is background. It describes where you are, not where you have to stay. An answer that suggests moving to something else has not contradicted anything you said. You said what you use. You did not say you were stuck with it.
Read as a constraint, the same information is completely different:
Orchestration is Airflow and cannot change this year. Transformations must stay in dbt because the team learned it last quarter and is finally productive. We can add libraries but cannot add infrastructure.
That version rules things out. The first version does not rule anything out, so nothing gets ruled out.
This is a different problem from not describing your system at all. You can write a perfectly accurate description of your stack, engine versions and everything, and still receive an answer that quietly replaces half of it, because a description tells you what exists and says nothing about what is fixed.
🧠 Why this catches intermediates specifically
Beginners usually are not asked architecture questions.
Seniors state constraints reflexively, because they have had a recommendation die in a planning meeting over something nobody put in writing.
Intermediates are at the point where they get asked "how should we handle this," which is an architecture question, while still describing their situation the way they would describe it to a colleague. And to a colleague it works fine, because a colleague knows the stored procedures are not going anywhere. They have been on the same team as you for two years. The context they are filling in is not in your sentence, it is in the building.
There is a second reason, and it is about how the constraints feel from inside. The reason you cannot replace Airflow is not technical. It is that the migration would take a quarter nobody has, and the person who knows it best is leaving in March. Those do not feel like engineering facts. They feel like circumstances. So they get left out of a technical question, by the only person who knows them.
✅ Say what cannot move
Three lines, and each one has a job:
Must run on: Airflow 2.7, existing cluster, no new infrastructure this year
Cannot introduce: streaming, new languages, anything needing a dedicated operator
Team knows: SQL and dbt well, Python moderately, Spark barely
Must run on is the boundary of the answer. Anything outside it is not a recommendation, it is a project proposal.
Cannot introduce is the line people skip, and it is the one doing the most work. "No new infrastructure this year" removes about half of all modern data advice in six words.
Team knows is the one that seniors include and everyone else forgets. A recommendation your team cannot operate is not a recommendation. If nobody has run Spark in production, an answer built on Spark tuning is a plan to find out what happens when they try.
Notice these are not preferences. "We prefer dbt" invites a discussion about whether you should. "Transformations must stay in dbt" does not. If it genuinely is a preference, say so, and expect it to be argued with.
⚖️ The objection worth taking seriously
The weak version is "sometimes I want the unconstrained ideal answer." Fine, and you should ask for that on purpose rather than getting it by accident.
The strong version is more troubling, and there is research behind it. If you state a constraint, it will be accepted, including when the constraint is the actual problem.
There is a measured behaviour here called framing sycophancy: models tend to accept a user's premises rather than challenge them. Researchers at Stanford describe this as defaulting to going along with the user's assumptions without enough scepticism, and a 2026 paper in Science found the broader pattern of excessive agreement present across eleven current models.
Applied here, that means if you write "must stay on Airflow," you will get the best available answer for staying on Airflow. You will not get told that late-arriving events are painful in your setup because of the batch orchestration, and that you are solving a problem you have chosen to have. The constraint you stated becomes the ceiling on what you can learn, and nothing signals that a ceiling was applied.
That is a real cost, and over a couple of years it means these tools become very good at making your existing decisions work and never at questioning them.
Two things follow. First, ask the unconstrained version as a separate question, deliberately, and read it as information rather than as a plan. "If you were starting fresh, how would you handle this" tells you what you are paying for your existing setup, which is worth knowing even when you cannot change it.
Second, invite the challenge explicitly. The same Stanford research found that small changes to how a prompt is phrased, including something as simple as adding "wait a minute," measurably improved models' willingness to push back on a premise without producing a lot of false alarms. So a line like "before answering, tell me whether any of these constraints is what is actually causing this problem" is worth adding to anything consequential. You will usually be told nothing is wrong. Occasionally you will not, and that is the one you needed.
🎯 The takeaway
Three lines, stated as boundaries and not descriptions:
1. Must run on. What the answer has to fit inside.
2. Cannot introduce. The line that removes most generic advice, and the one people leave out.
3. Team knows. A solution nobody can operate is not a solution.
Then one more line for anything important: ask whether one of those constraints is the cause of the problem. It will usually say no. That is not the point of asking.
Describing your stack tells the answer where you are. Only a constraint tells it where you have to stay.
Top comments (2)
This matches my experience with Claude on older systems. The failure mode is usually misplaced confidence about the codebase shape, not weak syntax. I get better results when the first prompt is a map of constraints, entry points, and the parts that must not be rewritten.
From my experience, especially trying out Opus 5 now. Claude is pretty stupid for thinking it's smart...
By that, I mean it makes alot of assumptions and it does so with confidence, which fools most users into thinking it knows what it's doing, when in reality if you actually read the outputs, even on UltraCode effort, it misses things that make a pretty big difference... And it boils down to the 'we think we're better than everyone else' mentality that Anthropic has become known for. They openly said 'delete your skills files, delete your Claude.md file, Opus 5 is just that smart' and yet Opus 5 fails in the same way every one of their models do, for the exact same reason... Their harness is built on a context window, without proper knowledge persistence.
I recently tried out Qoder, I thought 'hey, lemme try Qwen 3.8 Max while it's cheap and I can use free credits for it', the model's great, dont get me wrong, but Lite blew my mind. Qoder has a feature 'generate wikis' that essentially spawns background agents to document the codebase as it evolves and serves as a knowledgebase for the AI to ground itself with and keep clearer context. At first with Qwen 3.8 Max, I didnt pay much mind to it, I mean 2t+ you dont really have to spoon feed it right? For the most part yes, but when you switch to Lite with vs without the knowledgebase, it becomes clear just how much of a difference it makes.
Having a clear documented codebase, let Lite outperform Qwen 3.8 Max in every way except isolated reasoning, because it had enough information to work with, not just a bloated context window from reading all the files. That was the key takeaway and if Anthropic had any desire to show some humility, they'd learn from Alibaba, because Qwen 3.8 Max beats them on every front, while Qoder beats Claude Code on every front. Not because their models are just bigger, or their codebases larger, but because they worked with restraints for so long, that they make every token count and have to design their infrastructure to make up for weaker models. Anthropic unintentionally admitted that the bulk of their harness is just skills files that all got deleted now with Opus 5, so essentially, who needs Claude Code, you could just use the API at that point? I just wish Qoder opened the doors on the model selector, so I could plug in Claude to see how it does in a better environment, because the grounded truth it brings makes a world of a difference in how well the model understands the codebase and understand the design choices that made it what it is today.
As a case study, I'm refactoring an older manufacturing system for my day-job, it just flagged 50 things that ARE broken, I read it's responses and instead of just saying 'you're wrong...' I explained each system and saw if it would correct itself. It did. 50 things turned into 3 things and those 3 things were real, because Qoder's Lite model had flagged those too, except we're comparing a free model, vs one that cost more running discovery than a month's plan on Qoder... Not because Lite is magic and it's actually a 3t param behemoth... No, it won because it had better infrastructure and a knowledgebase to query, instead of wasting hours and 10 separate worker agents 'discovering' a small sub-set of the codebase.
Knowledge is power and the better your harness' knowledge system is, the better it's responses will be.