DEV Community: Bry

Serverless vs Containers: A Manager's Guide

Bry — Tue, 06 Oct 2026 15:00:00 +0000

Key Points

Serverless reduces operational burden — your team writes code, not infrastructure. Containers give you more control, but someone has to own that control.
The cost winner depends on your traffic pattern, not your team's preference. Low or spiky traffic favors serverless. High, steady traffic favors containers.
If your application needs persistent connections or long-running processes, serverless cannot do the job. Containers win by default.
Most mature engineering organizations use both. Serverless handles events and background jobs; containers run the core API.
Start with serverless. Migrate to containers when you hit its limits — not before.

Introduction

Every engineering team reaches a moment where the architecture question becomes real: do we go serverless or do we run containers? The answer your engineers give will sound technical. The decision you make should be business-driven. I have sat in that room — first as the engineer making the case, later as the person holding the budget — and the framing that matters is almost never the one the engineers lead with. This is not a question of which technology is more sophisticated. It is a question of which model reduces your operational burden, matches your cost structure, and fits the expertise you have on the team today. The wrong choice does not break your product — but it does burn money and time that you cannot get back.

The Core Question

Engineering teams often frame this debate around technical elegance. That is the wrong frame for a leadership conversation.

The right questions are: How much of your team's time do you want spent managing infrastructure versus building product? What does your traffic look like — steady and predictable, or bursty and unpredictable? And what does your cost structure look like at the scale you expect to reach in 12 to 18 months?

Those three questions will do more to guide your decision than any benchmark comparison.

The Analogy That Makes This Clear

Think of serverless as hiring contract workers. When work arrives, they show up — instantly, in whatever number the job requires. When work stops, they go home and you stop paying. The tradeoff: each new contractor takes a moment to get oriented before they start producing (this is the "cold start" delay engineers worry about). For most workloads, that delay is imperceptible. For some, it matters.

Containers are more like full-time staff. They are always at their desks, warmed up, ready to respond the moment a request arrives. Performance is consistent and predictable. The tradeoff: you pay their salary whether they are busy or not. If your servers run at 10% capacity most of the day, you are paying full-time wages for part-time work.

Neither model is universally better. The right one depends on what your workload actually looks like.

What This Means for Your Team

The operational difference between these two approaches is significant — and it falls on your engineering team, not on you.

With serverless, your team does not manage servers, operating systems, or capacity planning. The cloud provider handles all of that. The result is a smaller operational burden, faster deployment cycles, and less specialized infrastructure expertise required. A team of three engineers can run a meaningful serverless application without a dedicated DevOps hire. The tradeoff is that debugging and monitoring are harder. There is no persistent process to inspect, no server to SSH into. Your engineers need to think differently about troubleshooting.

With containers, your team owns the runtime environment — which means they control everything, including the things that go wrong. This is powerful for complex applications with specific runtime requirements. It also means someone on your team needs to understand container orchestration: the tooling that runs, scales, and connects your containers across a fleet of servers. That is a real skill that takes time to develop and retain. In practice, I have seen teams underestimate this consistently — they provision containers, get them running, and then discover six months later that nobody owns the upgrade cycle or the on-call rotation for the orchestration layer. If your team already has the expertise, containers unlock significant capability. If they do not, you are taking on an infrastructure learning curve while trying to ship product.

A 2024 platform engineering survey found that organizations running container orchestration required an average of 2.5 dedicated platform engineers per 100 application developers. Serverless architectures required 0.5. That difference has a salary cost.

The Cost Model Without the Math

Both pricing models are simple in principle. Serverless charges per request and per millisecond of execution time. If no one is using your application, you pay nothing. Containers charge per running hour — the server is on whether requests are flowing or not.

The cost winner is almost entirely determined by your traffic pattern:

Traffic Level	Serverless	Containers
Low / spiky (under 1M requests/month)	Cheaper — you pay only for actual use	Wasteful — servers sit idle most of the time
Medium (1M–100M requests/month)	Competitive	Competitive
High / steady (over 100M requests/month)	Can get expensive	Cheaper — cost amortized across constant load

A useful real-world reference point: a nightly batch job that runs two hours per day costs roughly a third less on serverless than on a dedicated container. A high-volume production API serving 500 requests per second costs roughly half as much on containers. The inflection point is utilization — how much of your provisioned capacity you are actually using at any given moment.

One important nuance: serverless has hidden costs that do not appear in the headline pricing. API gateways, data transfer, and third-party integrations all add to the bill. Factor these in before drawing conclusions from a back-of-napkin estimate. I recommend building a 90-day cost model that includes those line items before you take either option to a budget conversation — the headline number is almost always optimistic.

What Serverless Cannot Do

Serverless is not the right tool for every job, and knowing its hard limits will save you from an expensive architectural mistake.

Long-running processes. Most serverless platforms impose a maximum execution time of 15 minutes. If your application does anything that runs longer — large data processing jobs, video transcoding, complex ML inference — serverless will cut it off. Containers have no such limit.

Persistent connections. Serverless functions do not hold open connections between requests. If your application relies on WebSockets (live chat, real-time dashboards, multiplayer features) or long-lived TCP connections (certain financial data feeds, streaming protocols), serverless cannot support them. This is an architectural constraint, not a configuration option. I have seen this surface mid-project when a team built their real-time notification feature on serverless, got it working in testing, and then watched it fail immediately under a live user load that expected sustained connections — a costly rebuild two months before launch.

In-memory session state. Serverless functions are stateless by design. They do not remember anything between requests. Applications that rely on keeping data in memory across a user's session need a different approach — either an external cache or a stateful server, which points toward containers.

If your application requires any of these capabilities, containers are not just a preference. They are a requirement.

How Most Organizations Actually Do This

The serverless-versus-containers debate often presents as a binary choice. In practice, most mature engineering organizations use both — deliberately.

The common pattern: serverless handles everything event-driven and asynchronous. File uploads trigger a serverless function that resizes images and updates a database. Webhooks from payment processors invoke serverless logic that updates order status. Scheduled jobs run as serverless functions on a timer. None of this requires a persistent server.

Meanwhile, the core API — the service that handles user-facing requests in real time — runs in containers. It has predictable traffic, consistent performance requirements, and stateful session logic that makes serverless a poor fit.

This hybrid approach is not a compromise. It is a rational use of each tool where it performs best. If you are starting fresh, there is no rule that says you must choose one and abandon the other. I default to this split on greenfield projects: serverless for anything async and event-driven, containers for the core API — and revisit the boundaries only when cost data or a hard technical limit gives me a reason to.

Decision Flowchart

Use this to orient your conversation with your engineering team. It does not replace a proper architectural review, but it will tell you which direction to lean.

Adoption Roadmap

If you are moving toward either architecture deliberately — rather than inheriting whatever a previous team built — a phased approach reduces risk.

Phase 1 — Evaluate (Weeks 1–4). Map your existing workloads. Identify which are event-driven and short-lived versus long-running and stateful. Check your current monthly request volume and traffic distribution. Audit your team's infrastructure skills. This phase produces a recommendation, not a commitment.

Phase 2 — Pilot (Weeks 5–10). Choose one low-risk, non-critical workload and run it on the target architecture. A background job or an internal webhook handler is ideal. Measure actual cost against your estimate. Have your team document what was harder than expected.

Phase 3 — Expand (Months 3–6). Move additional workloads based on what you learned in the pilot. Establish internal patterns and templates so that each new workload does not require a new architectural conversation. Define your monitoring and alerting standards for the chosen model.

Phase 4 — Standardize (Month 6+). Codify the decision criteria your team uses. Document which workload types go serverless and which go containers. New engineers joining the team should be able to read one internal document and understand why things are built the way they are. That is the sign that the architecture is settled.

Questions to Ask Your Engineering Team

Before your team commits to an architecture, make sure you can get clear answers to these seven questions. If the answers are vague or contradictory, you need more time in Phase 1 before moving forward.

What does our traffic actually look like? Ask for a graph of requests per hour over the last 30 days. The shape of that graph — spiky or flat — is the single most important input to this decision.
Does our application need to hold open connections, and for how long? This determines whether serverless is technically viable, full stop. If the answer is yes, the discussion is over — you need containers.
Who on the team owns infrastructure, and what is their current skill set? If the answer is "nobody, really," that tells you something important about the operational risk of adopting containers. In my experience, this question surfaces the most organizational debt — teams that say they "use Kubernetes" but have one person who set it up two years ago and nobody who can troubleshoot it at 2 a.m.
What is our expected request volume in 12 months? Cost models diverge significantly at scale. An architecture that looks cheaper today may reverse at 50 million requests per month. Make sure the projection includes growth.
What is our tolerance for cold start latency? Serverless functions can take a fraction of a second to initialize when they have been idle. For most web APIs, this is invisible. For latency-sensitive applications — real-time bidding, financial transactions, gaming — it may be unacceptable.
Do any of our processes run longer than 15 minutes? ETL jobs, report generation, large file processing. If the answer is yes for any workload, that workload needs containers. It does not necessarily mean everything needs containers.
What does our current infrastructure bill look like, and what is it buying us? Understanding your baseline makes it possible to evaluate whether a new architecture is actually cheaper or just feels cheaper because it is new.

Conclusion

Serverless and containers are not rivals — they are tools with different strengths, suited to different conditions. The teams I have seen make this decision well are the ones who start from their actual workload and traffic data, not from what is trending at the time. Start with serverless if your team is small, your traffic is variable, and you want to minimize operational overhead. Move to containers — or add them alongside — when the data tells you to, not when someone in a conference room makes an argument that sounds good. The seven questions above are not a formality: if your engineers cannot answer them cleanly, you are not ready to commit to either architecture, and that is the most important thing to know before you approve a roadmap.

Junior Engineer Pipeline Collapse: Who Trains the Next Generation When AI Does the Grunt Work

Bry — Wed, 30 Sep 2026 15:00:00 +0000

Key Points

Entry-level hiring has fallen sharply and fast: roughly 65% down at major tech companies and 76% at early-stage startups, with new software engineering postings dropping 15% in just the first two months of this year.
The logic driving it is straightforwardly rational at the individual-firm level. One senior engineer with an AI coding assistant now ships what used to take a senior plus a junior, and when one person does the work of one and a half, nobody needs the extra half.
Junior work has always been how engineers learn the specific, nameable judgment that became the valuable thing. Cutting junior hiring isn't just saving money short-term. It's quietly cutting the only pipeline that reliably produces the senior judgment firms will need five to ten years from now.
We now open with the pipeline problem specifically because every other Delivery Practice including RFPs, fixed-price risk, client trust, insourcing, maintenance economics, assumes a supply of experienced engineers that this collapse is putting in real doubt for firms that don't deliberately intervene.

Introduction

A senior engineer said something to a hiring committee I sat on this year that was blunt enough to stick with me: why hire a junior for $90,000 a year when the AI coding assistant that makes any one of us more productive costs $10 a month? Nobody on the committee had a good rebuttal in the moment, because at the level of that single hiring decision, he wasn't wrong. The math really did favor not hiring.

The data backs up how widespread that individual logic has become. Entry-level hiring has fallen roughly 65% at major tech companies and 76% at early-stage startups this year, and new software engineering postings dropped 15% in just the first two months, hitting entry-level roles hardest of all. Unemployment for recent computer science and computer engineering graduates has climbed well above the general population rate, 6-7% against roughly 4.3% overall, a specific, measurable cost landing on a specific cohort. The mechanism is simple and, again, locally rational: companies report 40-55% more code output per sprint after adopting AI coding tools, and one senior engineer with those tools now ships what used to require a senior plus a junior. When one person does the work of one and a half, the extra half doesn't get hired.

What that hiring committee, and a lot of firms like it, aren't fully reckoning with yet is that junior roles were never just cheap labor. They were the mechanism by which the industry manufactured senior judgment, the specific, nameable pattern-recognition, the kind that only develops by spending years making, and fixing, the mistakes junior engineers make on real systems. Cutting that pipeline doesn't just save payroll in the short term. It's quietly cutting the supply of the exact thing that is becoming the valuable, billable thing: engineers who've built up enough hard-won pattern recognition to catch a subtly wrong AI-proposed solution before it reaches production. Some experts are flagging this directly already, warning that fewer new graduates entering the field could create real senior-developer shortages five to ten years out. That's the kind of warning that's easy to discount right now and expensive to have ignored later.

Firms Cutting Junior Hiring vs. Firms Restructuring It

Approach	What It Looks Like Right Now	Where This Is Likely to Lead
Pure hiring freeze	Stopped junior hiring entirely, redirected the savings to senior headcount and AI tooling licenses	A real, measurable senior-pipeline gap five to seven years out, competing hard for a shrinking pool of experienced engineers
"AI does the grunt work, juniors do nothing new"	Kept some junior hiring, but gives juniors only the residual work AI tooling doesn't already handle	Juniors underdeveloped, exposed to too little real judgment-building work to mature into the seniors the firm will need
Restructured junior roles around AI-output review and constrained real ownership	Redesigned entry-level work explicitly around reviewing AI output, owning small but real production components, and structured mentorship time	A genuine, if smaller, pipeline of engineers developing judgment faster than the old apprenticeship model, because they're reviewing more decisions per year, not writing more boilerplate
New entry-level categories: AI-output QA, data labeling and curation, model-behavior validation	Creating explicitly new junior roles rather than shrinking old ones	Some firms positioned to develop judgment through a different but real apprenticeship path, though how well this generalizes to full engineering judgment remains genuinely contested

Recommendation: if your firm's response to AI-driven productivity gains was simply "hire fewer juniors," check what your senior bench is likely to look like five years from now, not just this year's payroll. Firms restructuring junior work around real ownership and AI-output review, instead of eliminating it, are the ones building an actual pipeline.

Rebuilding a Junior Pipeline That Actually Produces Judgment

Stop giving juniors only residual work. If AI tooling handles the boilerplate and juniors get whatever's left over, they're not building judgment. They're doing chores. Deliberately assign juniors ownership of small, real, production-consequential components, with AI as a tool they direct, not a replacement for their decisions.
Build structured review of AI output into junior roles explicitly, not as an afterthought. Reviewing and validating AI-generated code against real production constraints is turning out to accelerate judgment development faster than writing routine implementation ever did. Treat it as core training, not busywork.
Create a real mentorship cadence, and protect senior engineers' time to do it. A senior doing the work of one and a half people has no slack left for mentorship unless that time is explicitly protected and counted as part of their role, not an unpaid extra.
Measure junior progression by judgment indicators, not implementation speed. Catching a subtly wrong AI suggestion, correctly pushing back on a requirement, recognizing a system's known failure mode: track these, not lines shipped, as the signal that the pipeline is actually working.
Accept that this pipeline will be smaller and more deliberate than the old one, and budget for it as a real cost center. The old apprenticeship model scaled with headcount almost automatically. The new one requires an explicit, funded decision to keep training people, because the market no longer forces it on you by default.

Questions to Ask Your Team

If we've cut junior hiring this year, do we know what our senior engineering bench is likely to look like five years from now, or are we assuming the market will supply seniors when we need them?
Are our current juniors getting real ownership and judgment-building work, or just the residual tasks AI tooling doesn't already handle?
Is mentorship time for senior engineers protected and counted as part of their role, or treated as something they're supposed to absorb on top of an already AI-accelerated workload?
Would we recognize a senior-engineer shortage forming in our own pipeline before it became an active hiring crisis, or only after?

Conclusion

The individual hiring-committee math this year isn't wrong. One senior engineer with AI tooling genuinely does the work of one and a half, and paying for the extra half stops making sense at the level of a single decision. What that math doesn't price in is that junior roles were never just labor capacity. They're the industry's mechanism for manufacturing the exact judgment this whole thread has been arguing is becoming the valuable, billable thing. Firms that simply stop hiring juniors get the short-term savings and, on a five-to-ten-year timeline, are likely to face a real gap in the judgment supply they'll need to compete on the deliverable. Firms restructuring junior work, around AI-output review, real ownership, and protected mentorship, are building something smaller than the old pyramid, but real.

An Engineer's Reflection: Watching the SI Business Model Change From the Inside

Bry — Wed, 23 Sep 2026 15:00:00 +0000

Key Points

The identity discomfort is real and widely reported this year. Commentators describe it as bordering on depression for engineers who feel their role slipping from skilled creator to what one report calls "cleanup crew." I don't think that framing is wrong. I think it's incomplete.
What I'm watching happen to the engineers coming through this well isn't that judgment is replacing coding as some abstract virtue. It's that the specific, nameable things they know, a legacy system's quirks, a client's real constraints, when an AI-proposed solution is subtly wrong, are becoming the paid part of the job, while the part that used to define professional identity, writing the implementation, is turning into infrastructure.
Fujitsu's own internal reforms are a useful, concrete marker of how far this is going beyond rhetoric: the company has moved from uniform starting salaries for new graduates to job-competency pay from day one, restructuring roughly 650 roles around it, an explicit break from seniority-based structure that would have been unthinkable at a firm like Fujitsu a few years ago.

Introduction

I want to talk about a specific conversation, not a general trend, because the trend only ever shows up to me one uncomfortable conversation at a time. This one was with a colleague, a genuinely excellent engineer who's spent fifteen years getting better at exactly the kind of implementation work AI tooling is rapidly making cheap. He asked me, not rhetorically, whether he'd wasted those fifteen years. I don't have a fully clean answer for him yet.

The discomfort he feels is widely shared and worth taking seriously on its own terms, not just as a business-model footnote. Commentary this year describes something close to an identity crisis for engineers whose professional self-image has been built on skilled implementation. One widely circulated framing put it as "bordering on depression," with veteran engineers feeling their role slip from skilled creator to what amounts to a cleanup crew reviewing AI output. That's not a business-press exaggeration. I'm watching it happen to people I respect, including, some days, myself.

Here's where I've landed after watching this from inside a role that straddles engineering and business: the resolution isn't that "judgment" swoops in as an abstract replacement virtue for "coding." It's narrower and more specific than that. The things turning out to be worth paying for are nameable: knowing that a particular legacy system behaves unpredictably under a specific load pattern, knowing which of a client's stated requirements are actually politically immovable versus negotiable, catching the specific moment an AI-proposed integration is subtly, dangerously wrong in a way that would only surface months into production. None of that is "judgment" as a vague virtue. It's specific, earned pattern-recognition that's becoming the billable part of the job, because the part that used to be billable, writing the implementation, got cheap. Fujitsu's own workforce reforms are the clearest evidence I've seen that this isn't just rhetoric: the company has broken from uniform, seniority-linked starting salaries toward job-competency pay from day one, restructuring around 650 roles in the process. That's an institution built on decades of seniority-based structure admitting, structurally and publicly, that what it pays for has changed.

An Engineer's Position: What Made You Valuable, Then and Now

Dimension	The Old Basis of Value	Where This Is Heading
What made an engineer valuable	Consistent, correct implementation output	Specific domain and system knowledge, judgment about AI-proposed solutions, willingness to own an outcome
Career ladder shape	Seniority and tenure-linked, fairly uniform within a cohort	Competency-linked from early career, per Fujitsu's own 2026 reform pattern
Most exposed cohort right now	Mid-career implementation specialists with deep tooling expertise, less domain-specific judgment	Likely to settle, but only for engineers who deliberately build domain-specific judgment during the transition
What "keeping up" means	Learning the newest AI coding tool	Deepening specific, nameable expertise a tool can't substitute for
Honest emotional reality right now	Real anxiety, reasonably described as an identity crisis for a meaningful share of the workforce	Likely to settle over time, but the transition is costing real careers and real confidence along the way. Not a costless story

Recommendation: if you're an engineer reading this mid-transition, the actionable version of this table isn't "learn to use AI tools better." It's "get specific about what you know that a tool doesn't": a client's real constraints, a system's real failure modes, a domain's real edge cases. That's the column most likely to survive.

What I'm Telling That Colleague

The fifteen years aren't wasted, but the specific skill that felt most valuable, fast, correct implementation, really is stopping being the paid part of the job. That's a real loss worth naming, not minimizing.
What those fifteen years actually built, underneath the implementation skill, is pattern-recognition about when something is subtly wrong. That's turning out to be the transferable part, and it wasn't obvious that it would be.
The engineers adapting fastest aren't out-learning AI tooling. They're out-specifying their own domain knowledge. They're turning "I've seen this before" into something they can name, price, and sell, instead of leaving it as unpriced background competence.
The institutional signal to watch for isn't a memo about "AI transformation." It's compensation structure actually changing. Fujitsu breaking seniority-linked pay isn't a communications exercise. It's the company admitting what it now pays for, and that kind of structural signal tells you more than any strategy announcement.

Questions to Ask Yourself, Not Just Your Team

What do I know, specifically and by name, that isn't just "I'm experienced": a system's real failure pattern, a client's real constraints, a domain's real edge cases?
Am I still measuring my own value by implementation speed, in a market that's increasingly not paying for that specifically?
If my compensation structure hasn't changed to reflect a competency-based model, is that because my organization hasn't caught up yet, or because it doesn't need to for the kind of work I actually do?
Would I recognize the moment an AI-proposed solution was subtly wrong in my own domain, and could I explain why, specifically, to someone who trusted the AI's output at face value?

Conclusion

This thread started with a pricing unit dying and ends, for now, with what that dying pricing unit is doing to the people underneath it. I don't think the honest reflection is either the doom-laden "identity collapse" framing circulating this year or the tidy "engineers move up the value chain" framing that shows up in a lot of retrospective business writing, including some of my own earlier pieces in this thread. It's both, at once: a real, disorienting loss for people whose professional identity was built on a skill that's stopping being scarce, and a real, achievable adaptation for the people who find the specific, nameable thing they know that a tool doesn't. My colleague hasn't found his yet, as far as I know. I think he will. I don't think it'll be comfortable getting there.

Azure Service Bus: Queues and Topics for When Event Grid Isn't Enough

Bry — Tue, 22 Sep 2026 15:00:00 +0000

Key Points

Basic tier supports queues only — no topics, no subscriptions, no sessions, at any price. It's a feature gate, not a cheaper version of Standard, and Standard is the practical floor the moment any of those three enter the picture.
Sessions are Service Bus's actual differentiator over Event Grid: they group related messages and guarantee in-order, single-consumer processing within a session — Event Grid has no equivalent.
Premium tier drops per-message transaction billing entirely in favor of a flat daily rate per messaging unit — the economics flip once you're at consistent high volume.
A subscription with no filter rule receives every message on the topic via the implicit 1=1 default rule — easy to forget you're relying on that default.

Prerequisites

CLI/SDK version tested against: az-cli 2.6x.x
An Azure resource group (az group create)
Standard tier or higher for the topic/subscription/session examples below — Basic tier will reject topic creation outright

Introduction

Article 352 covered Event Grid and drew a clear line: it's a push-based router for discrete events, not a queue and not a stream. This article covers the service that actually is a queue — with topics, ordering guarantees, and transactional semantics Event Grid was never built to provide.

The single most common surprise I've watched people hit with Service Bus isn't conceptual, it's the tier structure. Someone provisions a Basic-tier namespace to save money, tries to create a topic, and gets a flat rejection — topics simply don't exist at that tier. Not throttled, not limited. Absent. That's worth knowing before you provision anything, not after.

This article builds a session-enabled queue (the feature that actually justifies choosing Service Bus over Event Grid for ordered, related work) and a filtered topic subscription, entirely from the CLI.

The Tier Gate

Diagram: Basic tier is a genuinely different feature set, not a scaled-down Standard — this is the detail that catches people off guard.

Tier	Queues	Topics/Subscriptions	Sessions	Billing model
Basic	Yes	No	No	Per-operation, lowest rate
Standard	Yes	Yes	Yes	Per-operation, moderate rate
Premium	Yes	Yes	Yes	Flat daily rate per messaging unit (1, 2, or 4 units); no per-message charge

If your design needs a topic or a session — and sessions are the reason to pick Service Bus at all in most cases — Basic tier is off the table before you write a single line of application code.

Building a Session-Enabled Queue

Sessions group related messages so they're delivered in order to exactly one consumer at a time, identified by a session ID you assign when sending. This is the feature Event Grid genuinely cannot do — it has no concept of message grouping or delivery ordering across a related set.

az servicebus namespace create \
  --resource-group rg-servicebus-demo \
  --name sb-demo-namespace \
  --location eastus \
  --sku Standard

az servicebus queue create \
  --resource-group rg-servicebus-demo \
  --namespace-name sb-demo-namespace \
  --name orders-sessions-queue \
  --enable-session true

" width="784" height="291">

Diagram: session locking guarantees in-order delivery within a session without blocking unrelated sessions from being processed concurrently by other consumers.

Sending a message into a session queue requires setting the session ID explicitly — omit it and the send fails, because --enable-session true makes session ID mandatory for every message on that queue, not optional.

Building a Filtered Topic Subscription

az servicebus topic create \
  --resource-group rg-servicebus-demo \
  --namespace-name sb-demo-namespace \
  --name orders-topic

az servicebus topic subscription create \
  --resource-group rg-servicebus-demo \
  --namespace-name sb-demo-namespace \
  --topic-name orders-topic \
  --name high-value-orders

# Without this rule, the subscription's default "1=1" filter
# receives every message on the topic — not just high-value orders.
az servicebus topic subscription rule create \
  --resource-group rg-servicebus-demo \
  --namespace-name sb-demo-namespace \
  --topic-name orders-topic \
  --subscription-name high-value-orders \
  --name HighValueFilter \
  --filter-sql-expression "Total > 500"

Any subscription created without an explicit rule keeps the implicit default rule, which matches everything. That's fine when you genuinely want every message — the risk is forgetting the default is there at all and being surprised when a subscription you intended to be narrow is receiving the full topic firehose.

When to Reach for Service Bus Instead of Event Grid

	Service Bus	Event Grid
Delivery model	Pull (consumer receives/completes)	Push (webhook delivery)
Ordering	Yes, via sessions	No
Message grouping	Yes, via sessions	No
Duplicate detection	Yes (Standard+)	No
Transactions	Yes (Standard+)	No
Best for	Ordered work queues, transactional messaging	React-to-discrete-event fan-out

Recommendation: If a stakeholder describes a requirement involving "process these in order" or "these messages belong together and must not interleave," that's sessions, which means Service Bus. If the requirement is "notify several things when X happens" with no ordering constraint, Event Grid remains the lighter, cheaper choice from article 352.

Common Mistakes

Mistake 1: Provisioning Basic tier, then needing a topic
There's no in-place upgrade path that preserves existing resources across a tier change in every case — plan tier selection before provisioning, not as a fix-it-later decision.

Mistake 2: Enabling sessions without assigning session IDs on every send
--enable-session true makes the session ID mandatory. A send call missing it fails outright — this isn't a soft warning.

Mistake 3: Assuming a subscription with no rule is empty until configured
It's not empty — it has the default 1=1 rule and receives everything. If you wanted a narrow subscription, add the filter before anything starts publishing, not after noticing unexpected volume.

Mistake 4: Reaching for Service Bus when Event Grid would do
If there's no ordering or grouping requirement, Service Bus's added complexity and Standard-tier cost floor isn't buying you anything Event Grid didn't already provide cheaper.

Production Considerations

Performance: Standard tier is shared-capacity and can experience throttling under sustained high load. Premium's dedicated messaging units remove that variability — worth the flat rate once volume is consistent and predictable rather than spiky.

Security: Use Azure AD (Entra ID) authentication with managed identities rather than shared access signature connection strings where possible — connection strings embed credentials that are easy to leak into logs or source control.

Cost: Premium's flat daily rate per messaging unit becomes cheaper than Standard's per-operation billing once your message volume is consistently high — model both before committing, since Premium at low volume is a worse deal.

Monitoring: Track ActiveMessageCount and DeadLetterMessageCount per queue and subscription. A rising dead-letter count with no alert is the Service Bus equivalent of the EventBridge and Event Grid dead-letter blind spots covered in this series' earlier articles.

Full Example: Send and Receive with Sessions (TypeScript)

import { ServiceBusClient } from '@azure/service-bus';

const connectionString = process.env.SERVICEBUS_CONNECTION_STRING!;
const client = new ServiceBusClient(connectionString);

/** Sends two related messages into the same session, in order. */
async function sendSessionMessages(sessionId: string): Promise<void> {
  const sender = client.createSender('orders-sessions-queue');
  await sender.sendMessages([
    { body: { orderId: sessionId, step: 'created' }, sessionId },
    { body: { orderId: sessionId, step: 'payment-confirmed' }, sessionId },
  ]);
  await sender.close();
}

/** Receives and processes messages for a specific session in order. */
async function receiveSessionMessages(sessionId: string): Promise<void> {
  const receiver = await client.acceptSession('orders-sessions-queue', sessionId);
  const messages = await receiver.receiveMessages(10, { maxWaitTimeInMs: 5000 });

  for (const message of messages) {
    console.log(`Session ${sessionId}: ${JSON.stringify(message.body)}`);
    await receiver.completeMessage(message);
  }

  await receiver.close();
}

async function main() {
  const sessionId = 'order-abc123';
  await sendSessionMessages(sessionId);
  await receiveSessionMessages(sessionId);
  await client.close();
}

main();

Full source with the topic/subscription example: GitHub → cloud-apis/azure-service-bus-cli/

Conclusion

Reach for Service Bus over Event Grid specifically when ordering or message grouping matters — sessions are the concrete feature that justifies it, not a vague "more enterprise" gesture. Standard tier is the practical floor the moment topics or sessions enter the picture; Basic tier's queue-only feature gate isn't a limitation you can budget around, it's absent entirely. Know which one you actually need before you provision the namespace — moving between tiers after the fact costs more time than picking correctly up front.

Data Residency and APPI Compliance: What Cloud Vendors Entering Japan Need to Know

Bry — Thu, 17 Sep 2026 15:00:00 +0000

Key Points

APPI applies extraterritorially — if you handle personal data of individuals in Japan, you're in scope regardless of where your company is incorporated, and the PPC has authority to order compliance from overseas.
Cross-border transfer requires either the data subject's opt-in consent (with specific disclosures) or a documented, annually-monitored equivalent-protection system at the receiving party — "we have a privacy policy" is not either of those.
Breach notification runs on a tight two-report timeline: an initial report within 3–5 days of recognizing a qualifying breach, a final report within 30 days (60 for cyberattack-related incidents).
Japan has no blanket data-localization statute, but specific sectors (finance, critical infrastructure, government-adjacent) function as if it did — treat "no hard law" as "no automatic pass," not as "no requirement."

Introduction

I've reviewed vendor security questionnaires where the compliance answer for Japan reads "N/A — Japan does not require data localization." That's technically true and functionally the wrong answer, because it treats an absence of a blanket statute as an absence of a real obligation. Article 431 in this series flagged that data residency has become a procurement gate; this article is the regulatory substance behind that gate — what APPI actually requires, and where "no hard localization law" quietly turns into a hard requirement anyway once you're selling into specific sectors.

Nothing here substitutes for a licensed opinion from Japan-qualified counsel — treat this as the shape of the requirements so your legal and engineering teams know what questions to bring to that counsel, not as the final word on your specific situation.

APPI's Extraterritorial Reach

The Act on the Protection of Personal Information, enforced by the Personal Information Protection Commission, applies to any business handling the personal information of individuals in Japan — regardless of where that business is incorporated. Article 171 of the amended APPI explicitly grants the PPC authority over foreign operators, including the power to compel reports and issue orders to a company with no physical presence in Japan at all.

The problem this solves for the regulator, and the problem it creates for you: a foreign SaaS vendor with zero Japan entity, serving Japanese users entirely from a US data center, is still squarely in scope. "We're not a Japanese company" is not a compliance strategy — it's the specific gap Article 171 was written to close.

Cross-Border Transfer: The Two Legitimate Paths

This is the mechanic most foreign vendors get wrong, because the intuitive assumption — "we have a privacy policy, we're covered" — isn't one of the two paths APPI actually recognizes.

Path 1: Opt-in consent. You obtain the data subject's affirmative consent before transferring their personal information outside Japan, and at the point of obtaining that consent you disclose three specific things: the name of the destination country, that country's data protection legal framework, and the specific data protection measures the receiving party has in place. A generic "by using this service you consent to our terms" does not meet this bar — the disclosure requirements are specific and checkable.

Path 2: Equivalent-protection system. You establish a documented personal-information-protection system at the receiving party that meets APPI's substantive standard, and — this is the part vendors most often miss — you're required to monitor that system's continued adequacy at least once a year, not just certify it once at setup and move on.

Most SaaS vendors operating multi-region infrastructure default to Path 2 because Path 1's per-transfer consent flow doesn't fit a normal signup funnel. If you choose Path 2, budget for the annual monitoring obligation as an ongoing compliance line item, not a one-time setup task — this is where "we did this once during onboarding" quietly becomes non-compliant eighteen months later.

Breach Notification: A Tighter Timeline Than It Looks

APPI's breach notification runs on two reports with two different clocks, and both are shorter than most Western equivalents foreign compliance teams are used to planning around.

Report	Deadline	Trigger
Initial report to the PPC	3–5 calendar days from recognizing the breach	Breach involves sensitive personal information, risk of property damage, improper/unauthorized use, or more than 1,000 affected data subjects
Final report to the PPC	30 calendar days (60 days if the breach involved an "unjust purpose," e.g. a cyberattack)	Same qualifying triggers as the initial report

The reporting obligation itself isn't universal — it's triggered by the specific conditions in the table above, not every incident. But the practical guidance is to build your incident response runbook assuming you'll need to move on the 3–5 day clock, because determining whether an incident qualifies is itself something you often can't finish in 3–5 days if you're starting that assessment from zero at the moment of the breach. The teams that hit this deadline comfortably are the ones who decided their qualifying criteria and report template in advance, not during the incident.

Comparison: No Data Residency Commitment vs Multi-Region Flexibility vs Japan-Only Hosting

Criteria	No Residency Commitment	Multi-Region Flexibility	Japan-Only Hosting
APPI cross-border transfer obligation	Triggered on every transfer, by default	Triggered, but manageable via Path 2 + annual monitoring	Largely avoided — data doesn't leave Japan
Enterprise procurement pass rate	Low — increasingly a hard "no" at the security-questionnaire stage	Moderate — depends on how well Path 2 is documented and monitored	High — clears the tier where residency is treated as a gate
Regulated-sector eligibility (finance, gov-adjacent)	Effectively disqualifying	Sector-dependent, often insufficient	Required baseline for most regulated-sector deals
Engineering cost	None (but highest compliance/deal-risk exposure)	Moderate — regional data-partitioning work	Highest — dedicated Japan-region infrastructure

Bottom line: most horizontal SaaS products should target multi-region flexibility with a properly documented and annually-monitored Path 2 system. Reserve full Japan-only hosting for when a specific regulated-sector deal or government-adjacent procurement actually requires it — building it speculatively is expensive infrastructure ahead of a deal that may not materialize.

Pros and Cons

Advantages of Getting This Right Early

Clears the security questionnaire faster: a vendor who can name their transfer mechanism (Path 1 or Path 2) and point to a monitoring record answers in one email what an unprepared vendor spends weeks scrambling to produce.
Avoids the annual-monitoring trap: teams that document the Path 2 monitoring obligation as a recurring calendar item, not a one-time setup task, don't discover the gap during a customer's annual vendor security review.
Extraterritorial exposure is manageable, not existential: once you accept APPI applies regardless of your entity's location, the compliance program itself is a known, bounded scope of work — not an open-ended risk.

Disadvantages and Risks

The 3–5 day initial-report clock is unforgiving if unprepared: an incident response plan that doesn't already define your qualifying criteria will burn most of that window just deciding whether you need to report at all.
Path 2's annual monitoring is easy to let lapse: it's not a dramatic one-time certification, which is exactly why it's the compliance item most likely to quietly go stale.
Sector-specific expectations aren't written into APPI itself: finance and critical-infrastructure buyers apply a stricter bar than the statute's floor, and a vendor reading only the statute (not the sector's actual procurement practice) will underestimate what's actually required to close that specific deal.

Is This Right for You?

Prioritize Japan-only hosting if:

You're targeting finance, healthcare, or critical-infrastructure-adjacent customers specifically.
A specific large deal has already surfaced data residency as a hard requirement, not a nice-to-have.

Multi-region with Path 2 is likely sufficient if:

Your product is horizontal SaaS without sector-specific regulatory exposure.
You can commit real process (not just a policy document) to annual monitoring.

Revisit your posture if:

Your current answer to "how do we handle cross-border transfer" is a generic privacy policy clause — that's not one of APPI's two legitimate paths.

Implementation Approach

Phase 1: Scope and Gap Assessment (Weeks 1–3)

Confirm whether you process personal data of individuals in Japan today, not just whether you have a Japan entity.
Identify your current cross-border transfer mechanism, if any — most foreign vendors discover at this step that they have none.
Engage Japan-qualified counsel for a gap assessment against Path 1 and Path 2 requirements specifically.

Phase 2: Choose and Implement a Transfer Mechanism (Weeks 4–10)

Decide Path 1 (consent-based) or Path 2 (equivalent-protection system) based on your product's signup and data-flow architecture.
If Path 2: document the receiving-party protection measures formally, not informally, and calendar the first annual monitoring review now.
Update privacy disclosures if pursuing Path 1, including the three required disclosure elements.

Phase 3: Breach Response Readiness (Weeks 8–12, overlapping Phase 2)

Define your qualifying-breach criteria in advance (sensitive data, property-damage risk, unauthorized use, 1,000+ subjects) so an incident doesn't start with a definitional debate.
Draft the initial and final report templates now, so the 3–5 day clock is a fill-in-the-template exercise, not a from-scratch drafting exercise.
Assign a named incident owner responsible for the PPC reporting timeline specifically.

Phase 4: Ongoing Compliance (Month 4+)

Execute the annual Path 2 monitoring review on a fixed calendar date, tracked the same way you'd track a compliance certification renewal.
Revisit sector-specific expectations whenever you pursue a new vertical (finance, healthcare, government-adjacent) — APPI's floor and a specific sector's procurement bar are not the same thing.

Cost Considerations

Cost Type	What to Budget For	Typical Range
Legal gap assessment	Japan-qualified counsel review of current transfer mechanism	$10,000–$25,000 one-time
Path 2 documentation and setup	Formal protection-system documentation, initial assessment	$15,000–$40,000 one-time
Annual monitoring (Path 2)	Recurring review of receiving-party protection measures	$5,000–$15,000/year
Incident response readiness	Runbook, report templates, named ownership	2–3 weeks engineering/legal time
Japan-only hosting (if pursued)	Dedicated Japan-region infrastructure	Highly variable — model against your existing multi-region cost baseline

ROI signal: the legal gap assessment and Path 2 setup cost is small relative to a single enterprise contract lost at the security-questionnaire stage — treat it as a cost of being enterprise-sales-ready in Japan, not a discretionary compliance nice-to-have.

Questions to Ask Your Team

"Do we actually know whether we're on Path 1 or Path 2 for cross-border transfer, or is our current answer a generic privacy policy clause?"
"When was our Path 2 protection-system monitoring last actually reviewed, and is it on a calendar for next year?"
"Could we produce an initial PPC breach report within 3–5 days today, or would we spend that window just deciding if we need to report?"
"Are we treating 'Japan has no blanket data-localization law' as 'we have no requirement,' when our target sector's actual procurement bar says otherwise?"
"Does our incident response plan name a specific owner for the PPC reporting timeline, or is that an assumption nobody's confirmed?"

Conclusion

APPI's extraterritorial reach means "we're not a Japanese company" was never a valid compliance position, and its lack of a blanket data-localization statute was never a reason to skip a real transfer mechanism. Pick Path 1 or Path 2 deliberately, calendar the Path 2 annual monitoring obligation before it lapses quietly, and build your incident response runbook around the 3–5 day initial-report clock before you need it under pressure. The vendors who get burned here aren't the ones facing a hostile regulator — they're the ones who read "no hard localization law" as "no real requirement" and never picked a transfer mechanism at all.

The New SI Deliverable: Selling Judgment and Integration, Not Headcount

Bry — Wed, 16 Sep 2026 15:00:00 +0000

Key Points

For most of the industry's history, an SI's deliverable was, functionally, staffed capacity: a defined number of qualified people, for a defined duration, executing a defined scope. AI-assisted delivery is making that unit cheap to produce and hard to sell at the old price.
What's replacing it won't be a single new product. Roughly 85% of major tech firms are already turning to specialist providers for data curation and model fine-tuning, a services category that doesn't route through headcount at all, but through platform access, curated data, and judgment applied to a client's specific constraints.
Japan's largest SIers are placing genuinely different bets on what to sell instead: NTT Data toward "proposal-based SI that redesigns entire business processes," Fujitsu and Hitachi toward vertical, industry-specific judgment paired with AI (what Hitachi calls "Physical AI" for operational technology), NEC toward recurring revenue as its stated biggest management challenge.
The through-line across all of these bets is that none of them are headcount-denominated. A client won't be buying a number of engineer-months for much longer. They'll be buying a specific, demonstrable judgment applied to their specific problem, sold as a defined package rather than a staffing commitment.
SIs still trying to sell headcount with an "AI-powered" label attached to the invoice are already losing ground to competitors who've rebuilt their deliverable around something AI can't produce directly: legacy-system interpretation, cross-system governance, and the willingness to own the outcome, not just the hours.

Introduction

I sat in on a proposal review recently where a prospective client asked, pointedly, what exactly they were buying if not "a team of engineers for six months." That's been the honest answer to that question for as long as anyone in the room has worked in the industry. The account lead's answer, which took longer to land than it should have, was that the client was buying a specific integration outcome across three legacy systems that didn't talk to each other cleanly, delivered by people who'd solved that exact class of problem before, priced against the result rather than the hours it took to get there. That answer, awkward the first few times an SI has to give it out loud, is close to the actual new deliverable the entire industry is having to build toward.

The old deliverable is staffed capacity: a defined headcount, at a defined skill level, for a defined duration, executing a scope the client (or the SI, on the client's behalf) has already worked out. AI-assisted development is making the execution portion of that deliverable cheap enough that selling it at the old price is stopping being credible, which article 500 in this thread covers from the pricing side. What's worth examining directly is what firms are actually building to replace it, because "sell judgment, not headcount" is easy to say and hard to operationalize into an actual, purchasable thing.

The data-and-model-services category is the cleanest example of a genuinely new deliverable, not a relabeled old one. Roughly 85% of major tech firms are already turning to specialist providers for data curation and model fine-tuning, work that doesn't route through a headcount line item at all, but through platform access, curated proprietary data, and applied judgment about what a specific client's models actually need. Japan's largest SIers are making this concrete and public. NTT Data's stated shift is toward "proposal-based SI that redesigns entire business processes," backed by a specific 300-billion-yen AI-revenue target rather than a headcount target. Fujitsu and Hitachi are going the other direction, not toward infrastructure scale, but toward vertical, industry-specific judgment, with Hitachi explicitly betting on "Physical AI" applied to operational technology systems it already understands better than any AI-native competitor could. NEC has framed its own transition bluntly: recurring revenue, not staffing utilization, as its central management challenge going forward.

Old SI Deliverable vs. New SI Deliverable

Criteria	Staffed-Capacity Deliverable (legacy)	Judgment/Integration Deliverable (where this is heading)
What the client is actually buying	A defined headcount for a defined duration	A specific outcome, applied judgment, or a platform capability configured to the client's constraints
How it's priced	Rate card times hours times headcount	Outcome fee, usage-based platform access, or a defined asset license
What makes the vendor hard to replace	Availability of enough qualified staff	Depth of judgment in a specific vertical or system class the client can't easily replicate
Vendor's real product	Labor supply	Curated data, applied domain judgment, integration ownership, or a licensed platform capability
Effect of a client bringing AI tooling in-house	Direct threat: replaces the exact thing being sold	Limited threat: the client still lacks the specific judgment or curated asset being sold

Recommendation: if a client could replicate your entire deliverable just by licensing the same AI tools you use, you're still selling headcount with different branding. The deliverable that survives client-side AI adoption is the one built on something the client genuinely can't get by buying the tool directly.

Building a Judgment-Based Deliverable: What Firms Are Changing Internally

Firms are inventorying what they know that AI tooling doesn't. Legacy-system quirks, a specific regulator's actual enforcement patterns, a client's internal politics around a prior failed migration: these are turning out to be real, sellable assets once firms stop treating them as background context and start treating them as the product.
Firms are building defined, packaged offerings instead of open-ended staffing. NTT Data's "redesigns entire business processes" framing and Hitachi's vertical Physical AI bet are both packaged, named offerings, not "send us your requirements and we'll staff a team," but a defined scope built around a specific judgment claim.
Firms are changing who gets hired and promoted. Selling judgment instead of headcount means fewer people needed for routine execution and a premium on people who can own client relationships, interpret ambiguous legacy systems, and make architectural calls AI tooling can propose but not confidently commit to.
Firms are accepting that the new deliverable serves fewer clients per unit of revenue at first. A judgment-based engagement doesn't scale the way staffed-capacity did. Replacing headcount-scaling with judgment-scaling is a real, temporarily uncomfortable revenue-model change, not a free upgrade.

Questions to Ask Your Team

If a client asked us directly what they're buying if not a team of engineers, do we have a specific answer, a named judgment, a curated asset, an integration commitment, or would we describe our own deliverable in headcount terms without meaning to?
What do we know, about our clients' systems or industry, that AI tooling genuinely doesn't have access to, and are we pricing that specifically, or treating it as unpriced background expertise?
Have we built a packaged, named offering around that judgment, the way NTT Data and Hitachi are, or are we still proposing open-ended staffing with an AI label attached?
Are we prepared for a judgment-based deliverable to serve fewer clients per unit of revenue than staffed capacity did, at least initially?

Conclusion

The systems integrators most likely to make it through this transition won't do it by getting better at the old deliverable faster than competitors. They're building a different one, priced against judgment, curated assets, and integration ownership instead of engineer-hours, and Japan's largest firms are already making genuinely different bets on what that judgment should be: process redesign for NTT Data, vertical Physical AI for Hitachi, recurring revenue as the explicit target for NEC. None of those bets are headcount-denominated, and that's the actual signal. The firms still describing their deliverable as "a team of qualified engineers" are selling something a client can increasingly get more cheaply somewhere else. The ones that aren't are figuring out, specifically and early, what they know that AI doesn't.

Building a RAG Pipeline from Scratch: Embeddings, Retrieval, and Claude

Bry — Tue, 15 Sep 2026 15:00:00 +0000

Key Points

RAG (Retrieval-Augmented Generation) connects an LLM to your private data at query time — no fine-tuning, no retraining, no data leakage into model weights.
The pipeline has five stages: Ingest → Chunk → Embed → Store → Query (retrieve, augment, generate). Getting chunking and retrieval right matters more than which LLM you pick.
ChromaDB runs in-process for local dev with zero infrastructure — collection.add() to insert, collection.query() to retrieve. Production options include Pinecone and pgvector.
Common failure modes — chunks too large, no deduplication, no query expansion, skipping evaluation — are all avoidable. This article shows how.

Introduction

Large language models hallucinate when they don't know the answer. The standard fix — fine-tuning on your private data — costs tens of thousands of dollars, takes weeks, and produces a static model that goes stale as soon as your data changes. RAG solves a different problem: instead of baking knowledge into weights, it retrieves the relevant facts at query time and gives them to the model as context. The model answers from evidence, not memory. I've built RAG pipelines for internal knowledge bases and customer-facing chat tools — the architecture is the same whether you're indexing 500 internal docs or 500,000 support tickets; the parameters are what change.

This article builds a complete RAG pipeline in Python using ChromaDB for vector storage, sentence-transformers for local embeddings, and Claude for generation. You will understand each stage, why the design choices matter, and what breaks in production if you skip them. The full working implementation is in src/pipeline.py and src/main.py.

What RAG Is — and What It Replaces

Before writing code, align on why RAG exists.

Fine-tuning trains the model on your data. It is expensive (typically $5,000–$50,000+ for a production run), takes days to weeks, and produces a checkpoint that bakes in your data as of the training date. If your documentation changes next week, the model doesn't know. Fine-tuning is correct when you need the model to learn a style, a domain vocabulary, or a task format — not when you need it to answer questions from a document set.

Prompt-only (stuffing) puts your documents directly into the context window. It works for small document sets (tens of pages) but breaks on large corpora: context windows have limits, filling them with irrelevant text degrades answer quality, and at scale the cost per query becomes prohibitive.

RAG indexes your documents, retrieves only the chunks relevant to each query, and gives the model a focused context. It handles large corpora, stays current as documents are updated, and costs a fraction of fine-tuning. The tradeoff is a retrieval layer you have to build and maintain.

Approach	Data scale	Latency	Cost	Freshness
Fine-tuning	Any	Low (no retrieval)	High (training + inference)	Static — retrain to update
Prompt-only	Small (< 50 pages)	Low	Medium	Fresh
RAG	Large (any)	Medium (retrieval + LLM)	Low–Medium	Fresh (re-embed on update)

RAG Pipeline Architecture

A RAG pipeline has two sides: the ingest side (run once per document update) and the query side (run on every user request). Together they form five stages.

Diagram: Ingest side (left) runs document processing once. Query side (right) runs on every user request.

The five stages:

Ingest — load raw documents. Source can be text files, PDFs, database records, or API responses.
Chunk — split documents into pieces small enough to embed meaningfully. Chunk size is the most consequential parameter in the pipeline.
Embed — convert each chunk to a vector. The embedding model maps semantic meaning to a point in high-dimensional space.
Store — persist vectors (with their source text and metadata) in a vector database.
Query — embed the user's question, find the nearest chunks, inject them into a prompt, and generate an answer.

Chunking Strategies

Chunking is where most RAG pipelines fail. A bad chunking strategy produces irrelevant retrievals; irrelevant retrievals produce hallucinated answers. The model can't conjure information that wasn't in the retrieved chunks. In practice, I spend more time tuning chunk size and overlap than on any other pipeline parameter — getting it wrong is invisible until a user catches the model confidently citing the wrong passage.

Fixed-Size Chunking

Split every N characters or tokens. Simple to implement, ignores document structure.

# Fixed-size chunking — simple but blunt
def chunk_fixed(text: str, size: int = 400, overlap: int = 50) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start += size - overlap  # overlap preserves context at boundaries
    return chunks

The overlap matters: without it, a sentence split across two chunks retrieves each half separately and both halves are incomplete.

Recursive Chunking

Try splitting on paragraph breaks (\n\n), then line breaks (\n), then spaces. Each level is a fallback when the previous splitter still produces chunks over the target size. This respects document structure — paragraphs before sentences before words.

def _chunk(self, text: str, max_tokens: int = 400) -> list[str]:
    """Recursively split text into chunks, preferring natural boundaries.

    Args:
        text: The input text to split.
        max_tokens: Approximate maximum chunk size in tokens (1 token ≈ 4 chars).

    Returns:
        List of text chunks, each within the max_tokens limit.
    """
    max_chars = max_tokens * 4  # rough token → char estimate

    if len(text) <= max_chars:
        return [text.strip()] if text.strip() else []

    # Try splitting on paragraph breaks first, then newlines, then spaces
    for separator in ["\n\n", "\n", " "]:
        parts = text.split(separator)
        if len(parts) > 1:
            chunks: list[str] = []
            current = ""
            for part in parts:
                candidate = (current + separator + part).strip() if current else part.strip()
                if len(candidate) <= max_chars:
                    current = candidate
                else:
                    if current:
                        chunks.append(current)
                    # Recurse on oversized parts
                    if len(part) > max_chars:
                        chunks.extend(self._chunk(part, max_tokens))
                        current = ""
                    else:
                        current = part.strip()
            if current:
                chunks.append(current)
            return [c for c in chunks if c]

    # No separator found — hard split
    return [text[:max_chars].strip(), text[max_chars:].strip()]

Semantic Chunking

Group sentences by semantic similarity — a sentence joins the current chunk if its embedding is similar enough; otherwise it starts a new chunk. Produces the most coherent chunks but is slower (requires embedding every sentence during ingestion).

Diagram: Chunking strategy selection — recursive approach, falling back from paragraphs to lines to words.

Embeddings

An embedding model converts text into a dense vector — a list of floats (e.g., 384 dimensions for all-MiniLM-L6-v2) where similar texts produce similar vectors. Semantic similarity becomes geometric proximity.

Local Embeddings: `sentence-transformers`

sentence-transformers runs entirely locally — no API key, no per-token cost, no latency from network calls.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

texts = [
    "Python is a high-level programming language.",
    "Guido van Rossum created Python in 1991.",
    "The weather is sunny today.",
]

embeddings = model.encode(texts)  # shape: (3, 384)
print(embeddings.shape)           # (3, 384)

all-MiniLM-L6-v2 is 80MB on disk, encodes ~14,000 sentences per second on CPU, and produces 384-dimensional vectors. It is the right default for local development and prototyping.

API Embeddings

For production, Anthropic's and OpenAI's embedding APIs trade local cost for higher-quality vectors at scale. Use them when your retrieval accuracy on domain-specific content drops below acceptable thresholds.

Option	Model	Dims	Cost	Notes
`sentence-transformers`	`all-MiniLM-L6-v2`	384	Free (local)	Best for dev/prototyping
`sentence-transformers`	`all-mpnet-base-v2`	768	Free (local)	Higher quality, 3× slower
OpenAI API	`text-embedding-3-small`	1536	$0.02 / 1M tokens	Good price/quality tradeoff
Anthropic API	Voyage-3 (via Voyage AI)	1024	$0.06 / 1M tokens	SOTA quality for RAG

Keep your embedding model consistent between ingestion and query time. If you embed documents with all-MiniLM-L6-v2 and query with text-embedding-3-small, your similarity scores will be meaningless.

Vector Storage with ChromaDB

ChromaDB is an in-process vector database for Python. It requires no server, no Docker container, no cloud account — import it and use it.

import chromadb

# In-memory client — resets between runs (good for unit tests)
client = chromadb.Client()

# Persistent client — stores to disk (good for development)
client = chromadb.PersistentClient(path="./chroma_db")

collection = client.get_or_create_collection(name="my_docs")

# Add documents with pre-computed embeddings
collection.add(
    ids=["chunk_0", "chunk_1", "chunk_2"],
    embeddings=[[0.1, 0.2, ...], [0.3, 0.1, ...], [0.9, 0.1, ...]],
    documents=["Python is...", "Guido van Rossum...", "Weather is..."],
    metadatas=[{"source": "python_intro.txt"}, {"source": "python_intro.txt"}, {"source": "weather.txt"}],
)

# Query — returns top-3 nearest chunks
results = collection.query(
    query_embeddings=[[0.15, 0.18, ...]],
    n_results=3,
    include=["documents", "metadatas", "distances"],
)

for doc, meta, dist in zip(
    results["documents"][0],
    results["metadatas"][0],
    results["distances"][0],
):
    print(f"[{dist:.3f}] {meta['source']}: {doc[:80]}")

ChromaDB uses cosine similarity by default. Lower distance = more similar.

Choosing a Vector Database

Feature	ChromaDB	Pinecone	pgvector
Setup	Zero (in-process)	Managed cloud	Add extension to Postgres
Scale	Millions of vectors	Billions	Millions (with tuning)
Cost	Free	$70/month+ (managed)	Postgres hosting cost
Filtering	Metadata filters	Metadata filters	Full SQL WHERE clauses
Persistence	Local disk	Cloud-managed	Postgres storage
Best for	Local dev, prototypes	Production at scale	Teams already on Postgres

Start with ChromaDB locally — it removes all infrastructure friction during development, which is where you need to iterate fastest. I reach for pgvector over Pinecone when the team is already on Postgres: the operational overhead is near-zero and full SQL filtering eliminates an entire class of retrieval bugs that metadata-only filters can't handle.

Retrieval Strategies

Retrieval is not just "find the nearest vectors." Three strategies matter in practice.

Cosine Similarity Search

The default. Compute the cosine similarity between the query vector and every stored vector; return the top-k most similar chunks. Fast, well-understood, and works well when the query language matches the document language.

Maximal Marginal Relevance (MMR)

Standard top-k retrieval can return five chunks that all say the same thing — high similarity, low diversity. MMR trades some similarity for diversity: each additional chunk is selected to be both similar to the query and different from already-selected chunks.

ChromaDB does not implement MMR natively. Implement it post-retrieval:

def _mmr_rerank(
    query_embedding: list[float],
    candidate_embeddings: list[list[float]],
    candidate_docs: list[str],
    k: int = 5,
    lambda_: float = 0.5,
) -> list[str]:
    """Select top-k chunks using Maximal Marginal Relevance.

    Args:
        query_embedding: Embedded user query.
        candidate_embeddings: Embeddings of candidate chunks (over-fetch, e.g., top-20).
        candidate_docs: Text of each candidate chunk.
        k: Number of chunks to return.
        lambda_: Trade-off between relevance (1.0) and diversity (0.0).

    Returns:
        k chunks selected for both relevance and diversity.
    """
    import numpy as np

    q = np.array(query_embedding)
    cands = np.array(candidate_embeddings)

    # Cosine similarity: query vs candidates
    sim_to_query = (cands @ q) / (np.linalg.norm(cands, axis=1) * np.linalg.norm(q) + 1e-9)

    selected_indices: list[int] = []
    remaining = list(range(len(candidate_docs)))

    for _ in range(min(k, len(remaining))):
        if not selected_indices:
            # First pick: highest similarity to query
            best = max(remaining, key=lambda i: sim_to_query[i])
        else:
            # Subsequent picks: balance relevance vs redundancy
            selected_vecs = cands[selected_indices]
            scores = []
            for i in remaining:
                sim_q = sim_to_query[i]
                sim_selected = max(
                    (cands[i] @ selected_vecs[j]) /
                    (np.linalg.norm(cands[i]) * np.linalg.norm(selected_vecs[j]) + 1e-9)
                    for j in range(len(selected_vecs))
                )
                scores.append(lambda_ * sim_q - (1 - lambda_) * sim_selected)
            best = remaining[scores.index(max(scores))]

        selected_indices.append(best)
        remaining.remove(best)

    return [candidate_docs[i] for i in selected_indices]

Hybrid Search (BM25 + Semantic)

Semantic search handles paraphrase and synonymy well. BM25 (keyword search) handles exact terms — product codes, names, technical identifiers — better. Hybrid search runs both and combines scores (typically via Reciprocal Rank Fusion). Use hybrid when your documents contain precise identifiers that semantic search might miss. I default to MMR over top-k similarity the moment a domain has redundant content — policy documentation, API reference pages, and anything generated from a template will poison straight similarity retrieval with near-duplicate chunks every time.

Context Augmentation

Retrieved chunks are useful only if the prompt tells Claude how to use them. A weak prompt ("here are some documents, answer the question") produces weak answers. A well-structured prompt produces grounded, citable answers.

def _build_prompt(self, question: str, chunks: list[str]) -> str:
    """Format retrieved chunks and question into a grounded generation prompt.

    Args:
        question: The user's original question.
        chunks: Retrieved document chunks, ordered by relevance (most relevant first).

    Returns:
        A formatted prompt that instructs Claude to cite sources and acknowledge gaps.
    """
    context_block = "\n\n---\n\n".join(
        f"[Source {i + 1}]\n{chunk}" for i, chunk in enumerate(chunks)
    )

    return f"""You are a helpful assistant. Answer the question below using only the provided sources.

Rules:
- Cite which source(s) support each claim (e.g., "According to Source 2...").
- If the sources do not contain enough information to answer the question, say so explicitly.
- Do not use knowledge outside the provided sources.

Sources:
{context_block}

Question: {question}

Answer:"""

The three rules matter. Citing sources forces the model to ground claims in retrieved text rather than training data. Acknowledging gaps prevents confident-sounding hallucinations. Forbidding outside knowledge keeps the model from blending retrieved context with parametric knowledge unpredictably.

Generation with Claude

The query method ties everything together: embed the question, retrieve chunks, build the prompt, call Claude, and return the answer.

Diagram: Full query path from user question to Claude-generated answer.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from environment

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": prompt}],
)

# Check why generation stopped
match response.stop_reason:
    case "end_turn":
        answer = response.content[0].text
    case "max_tokens":
        # Response was cut off — increase max_tokens or reduce prompt length
        answer = response.content[0].text + "\n\n[Response truncated — max_tokens reached]"
    case _:
        answer = f"[Generation stopped: {response.stop_reason}]"

Always check stop_reason. end_turn means the model finished naturally. max_tokens means the answer was cut off — a common silent failure when retrieved chunks are large and the combined prompt + answer exceeds your token budget.

For cheap classification tasks within a RAG system (e.g., query routing, intent detection, relevance filtering), use claude-haiku-4-5-20251001 instead of claude-sonnet-4-6. Haiku is significantly faster and cheaper for short-context classification where generation quality is less critical.

The `RAGPipeline` Class

The full implementation encapsulates all stages into a single class with three public methods: ingest, query, and the private helpers.

Diagram: RAGPipeline class structure — three public methods, three private helpers.

The constructor initializes all three dependencies. Chunk and embed are private because callers should not need to call them directly — they are implementation details of ingest and query.

Evaluation

A RAG pipeline with no evaluation is a guess. Add measurement before shipping.

Three metrics from the RAGAS framework cover the core failure modes:

Metric	What it measures	How it catches failures
Faithfulness	Is every claim in the answer supported by a retrieved chunk?	Catches hallucinations — the model added facts not in context
Answer Relevancy	Does the answer actually address the question?	Catches topic drift — the model answered a different question
Context Recall	Did retrieval surface the chunks needed to answer?	Catches retrieval failures — the right chunks weren't found

Run evaluation on a golden dataset — 20–50 question/answer pairs you've manually verified. Automate it in CI so a configuration change that breaks retrieval doesn't silently ship.

# RAGAS evaluation — requires: pip install ragas
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset

eval_data = {
    "question": ["What is Python?", "Who created Python?"],
    "answer": ["Python is a programming language.", "Guido van Rossum."],
    "contexts": [
        ["Python is a high-level language..."],
        ["Guido van Rossum created Python in 1991..."],
    ],
    "ground_truth": ["Python is a high-level programming language.", "Guido van Rossum."],
}

result = evaluate(
    Dataset.from_dict(eval_data),
    metrics=[faithfulness, answer_relevancy, context_recall],
)
print(result)

Common Mistakes

Mistake 1: Chunks larger than 600 tokens drown the retrieval signal.
A 600-token chunk covers multiple topics. When you embed it, the vector averages over those topics and becomes less specific to any one of them. Similarity search returns chunks that are vaguely related to the query, not precisely relevant. Keep chunks under 400 tokens. If a passage requires more context, overlap consecutive chunks by 50–100 tokens rather than enlarging the chunk size.

Mistake 2: Not deduplicating similar chunks before storing.
If your corpus contains repeated passages (headers, boilerplate, legal disclaimers), those chunks will dominate retrieval — every query returns five variations of the same paragraph. Deduplicate before ingestion: compute a hash of each chunk's text and discard exact duplicates, then use a similarity threshold (cosine similarity > 0.95) to collapse near-duplicates. I've seen this sink a support-bot demo: the corpus was a product manual where the safety warning appeared verbatim on 40 of 200 pages — top-5 retrieval returned the warning for nearly every query, and the model dutifully answered "please refer to a qualified technician" regardless of what was asked.

Mistake 3: Embedding the raw query without expansion.
A user asking "how does Python handle memory?" may not use the same words as your documentation ("garbage collection", "reference counting", "memory management"). Query expansion generates alternative phrasings before embedding: original_query + hypothetical_answer_keywords. Hypothetical Document Embeddings (HyDE) generates a fake answer to the question and embeds that instead — the fake answer often matches document language better than the raw question.

Mistake 4: Skipping evaluation entirely.
Most RAG pipelines are shipped based on developer intuition ("the answers look right in testing"). Without a golden dataset and automated metrics, you won't know when a configuration change — a new chunk size, a different top-k, an updated embedding model — makes retrieval worse. Define your evaluation dataset before you tune parameters, not after. I've watched a team spend two weeks tuning chunk size upward because answers "felt" more complete — then discover via RAGAS that faithfulness had dropped from 0.89 to 0.71 because larger chunks were diluting the retrieved context with off-topic sentences.

Mistake 5: Using the same model for embedding and generation.
Anthropic's Claude models are generation models, not embedding models. Using a generation model's hidden states as embeddings produces inferior vectors compared to models trained specifically for semantic similarity (sentence-transformers, OpenAI's text-embedding-3-*, Voyage AI). Keep embedding and generation as separate concerns: sentence-transformers (or a dedicated embedding API) for vectors, Claude for generation.

Full Example

The complete implementation lives in two files:

src/pipeline.py — RAGPipeline class with all methods documented and implemented.
src/main.py — demo that ingests five documents about Python and runs three queries.

Setup:

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

cp .env.example .env
# Add your ANTHROPIC_API_KEY to .env

python src/main.py

Expected output:

Ingested 5 documents → 12 chunks stored.

Q: What is Python's GIL?
A: According to Source 1, Python's Global Interpreter Lock (GIL) is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecodes simultaneously...

Q: Who created Python and when?
A: According to Source 3, Python was created by Guido van Rossum and first released in 1991...

Q: What is list comprehension?
A: According to Source 2, list comprehension is a concise syntax for creating lists based on existing iterables...

Full source: GitHub link → ai-integration/rag-pipeline/ — see README for setup steps.

Conclusion

RAG is the right architecture when your LLM needs access to private, large, or frequently updated data. The implementation is not what makes or breaks it — the parameters are: chunk size, overlap, retrieval strategy, and whether you bother to measure any of it. Most pipelines that fail in production fail because they were tuned by intuition and never measured. Add the RAGAS evaluation step before you ship anything to users, keep your chunks under 400 tokens, and you will avoid the failure modes that make RAG look unreliable. The architecture isn't fragile — the shortcuts are.

Who Absorbs the Margin — Client, SIer, or the AI Vendor: Renegotiating SI Economics

Bry — Wed, 09 Sep 2026 15:00:00 +0000

Key Points

Every dollar an AI tool saves an SI on delivery cost is a dollar three different parties have a plausible claim to: the client (who's aware operating costs went down), the AI platform vendor (whose licensing terms increasingly shift toward capturing usage-based value), and the SI (whose judgment turned a raw capability into a delivered outcome).
Procurement teams are learning to sort renewals into three buckets: renew and absorb, renegotiate to consumption, or replace and build. Expect more SIs to get pushed into the second and third buckets as clients understand AI has genuinely lowered the SI's own cost base.
93% of sellers reported struggling to quantify and defend the value they'd actually added, which puts most SIs into these renegotiations without a real number to counter a client's discount demand.
SIs that come through this pressure well are the ones that stop treating margin as something to defend by obscuring cost structure, and start treating it as something to defend by pricing a specific, demonstrable judgment contribution the client couldn't get from the AI platform directly.

Introduction

A client's CFO said something to me directly not long ago that a lot of SI account teams are starting to hear some version of: "I know what your AI licensing costs. I know what it used to cost you to deliver this. Explain to me why I'm still paying the old number." It's not an unreasonable question, and at this point in the AI-services transition, most SIs don't have a rehearsed answer for it. For a while, plenty of firms have been quietly pocketing the AI-driven cost reduction as pure margin, hoping nobody with enough visibility into the underlying economics would ask.

The honest framing is that AI-driven delivery savings don't belong to any one party by default. They're contested, three ways. Clients have a real claim: operational costs genuinely came down, and clients increasingly know it, because AI tool pricing itself became public and metered rather than opaque. AI platform vendors have a claim too, since a growing share of enterprise software licensing is shifting toward capturing usage-based value directly, rather than just charging a flat seat fee and letting downstream margin sit wherever it lands. And the SI has a claim, the one most vulnerable to getting argued away in a renegotiation: the actual judgment, integration work, and delivery risk absorbed in turning a raw AI capability into something a client can actually rely on in production.

Procurement organizations are getting sophisticated about this fast. A fairly standard renewal framework is already sorting vendor relationships into three buckets: renew and absorb (accept the current pricing, usually for low-volatility, well-understood work), renegotiate to consumption (move rigid seat- or hour-based pricing to a usage-based model that tracks real utilization), or replace and build (walk away from the vendor relationship entirely and bring the capability in-house, now that AI has made that a realistic option for more capabilities than it used to be). SIs that show up to these conversations without a defensible, quantified account of their own value contribution are getting sorted into renegotiate-to-consumption or replace-and-build far more often than SIs that can show precisely what they still add. A Holden Advisors study found 93% of sellers admitted they struggled to accurately quantify and defend their value, which means most SIs are walking into these renegotiations structurally unprepared for a client who's done their homework.

Where the Margin Actually Goes: Three Claims on the Same Savings

Claimant	Basis for the Claim	What Wins the Argument in Practice
Client	Aware that AI genuinely lowered the vendor's delivery cost; expects some pass-through	A vendor with a metered, transparent cost basis wins credibility; one hiding behind an unchanged invoice loses it fast
AI platform vendor	Owns the underlying capability; increasingly prices on usage rather than flat seat fees	Whatever margin the platform vendor's own licensing terms capture directly. This claim is largely settled by contract, not negotiation
Systems integrator	Provided judgment, integration, delivery risk absorption, and a working result the client couldn't have gotten by licensing the AI tool directly	A quantified account of that judgment, specific defect rates avoided, integration complexity resolved, risk absorbed, wins. A vague "we add value" claim loses

Recommendation: don't walk into a renewal conversation assuming your margin is safe because it always has been. Assume the client already has a rough estimate of your AI-driven cost reduction, and come with a specific, demonstrable account of what you add beyond the tool itself, or expect to get renegotiated into the consumption bucket.

Defending Margin With a Real Number Instead of a Vague Claim

Quantify your own AI-driven cost reduction honestly, before the client does it for you. If you know the number, you control how it's framed. If the client calculates it independently and you're caught unprepared, you're negotiating from a defensive position for the rest of the relationship.
Separate "what the AI platform did" from "what we did with it," in terms specific enough to price. Defect rates avoided, integration issues resolved that the AI output alone would have missed, delivery risk actually absorbed: name the specific contribution, not a general appeal to expertise.
Offer the renegotiation before the client demands it. SIs that proactively bring a fair, transparent pricing update to a renewal conversation are keeping far more of their margin than SIs that wait to be confronted, because proactive framing reads as partnership and reactive framing reads as having been caught.
Know which of your engagements are genuinely replace-and-build candidates, and don't fight that battle. Some capabilities really have become cheap enough for a client to bring in-house. Contesting that reality burns credibility you'll need for the engagements where your judgment genuinely isn't replaceable.

Questions to Ask Your Team

Do we know our own AI-driven delivery cost reduction, on our top client relationships, well enough to defend a number if the client brings one first?
Can we name, specifically, what we add beyond the AI platform's raw output on our current engagements, or would we be making a vague appeal to "expertise" in a real renegotiation?
Are we proactively bringing pricing conversations to clients as AI changes our cost base, or waiting for a CFO to ask the question first?
Which of our current engagements are honest replace-and-build candidates for the client, and are we prepared for that conversation instead of resisting it?

Conclusion

There's no default answer to who keeps the margin AI creates. It depends on who shows up to the renegotiation with a real number. Clients who understand the underlying cost shift will ask for a share of it, and they're not wrong to. AI platform vendors are already capturing their share through licensing terms most SIs don't control. What's actually contestable is the SI's own piece, and 93% of sellers walking into that conversation without a quantified value story is exactly why so much margin is likely to get renegotiated away from firms that could have kept more of it. The ones that hold their ground will be the ones treating their judgment as a line item with a number attached, not an assumption everyone keeps taking on faith.

Amazon SQS vs SNS: Queues, Fan-Out, and Picking the Right One from the CLI

Bry — Tue, 08 Sep 2026 15:00:00 +0000

Key Points

SQS is a queue — one message, delivered to one consumer, held until processed. SNS is pub/sub — one message, fanned out to every subscriber. The fan-out pattern combining both is the standard, not an either/or choice.
SNS-to-SQS delivery requires the queue's access policy to explicitly allow the topic to call SendMessage — skip this and messages vanish with no error on the publish side.
FIFO queues cap at 3,000 messages/second batched (300/s unbatched) unless you enable high-throughput mode, which raises that to 70,000/s. Both SQS and SNS also bill in 64 KB payload chunks, same mechanic as EventBridge.

Prerequisites

CLI/SDK version tested against: aws-cli/2.35.x
An IAM role with sqs:* and sns:* permissions
jq installed for parsing CLI JSON output in the examples below

Introduction

I still get asked "should I use SQS or SNS?" as though it's a fork in the road. Most of the time the honest answer is both, wired together — SNS decides who gets notified, SQS makes sure each of them actually gets to process the message at their own pace without losing it if they're briefly unavailable. Treating them as competing choices is how teams end up building a fan-out mechanism inside application code that SNS already does for free.

The part that trips people up isn't the concept. It's the access policy. Subscribe an SQS queue to an SNS topic without granting the topic permission to send to that queue, and the subscription succeeds, the topic publish succeeds, and the message simply never arrives — no error surfaces anywhere in that chain. I've debugged this exact silent failure in a client's fan-out pipeline, and it's the single most common gap in CLI tutorials for this pattern.

This article builds a standalone queue, a standalone topic, and the full fan-out pattern with the access policy step included, then covers where FIFO changes the throughput math.

Queue vs Topic, Conceptually

Diagram: SQS delivers each message to exactly one consumer; SNS delivers a copy to every subscriber.

A message sitting in an SQS queue waits until something polls for it. A message published to an SNS topic is pushed immediately to every current subscriber — there's no "waiting" concept on the topic itself, which is exactly why SNS alone is a poor fit for a subscriber that might be temporarily down. That's what the fan-out pattern fixes.

Standalone SQS

# Standard queue — no ordering guarantee, near-unlimited throughput.
aws sqs create-queue --queue-name orders-standard-queue

QUEUE_URL=$(aws sqs get-queue-url --queue-name orders-standard-queue --query QueueUrl --output text)

aws sqs send-message \
  --queue-url "$QUEUE_URL" \
  --message-body '{"orderId":"order-abc123","status":"pending"}'

aws sqs receive-message --queue-url "$QUEUE_URL" --max-number-of-messages 1

# FIFO queue — strict ordering within a message group, capped throughput.
aws sqs create-queue \
  --queue-name orders-fifo-queue.fifo \
  --attributes '{"FifoQueue":"true","ContentBasedDeduplication":"true"}'

Queue type is permanent. There's no update-queue-type command — you create a new queue and migrate, you don't convert an existing one. Decide standard vs FIFO before you have production traffic depending on the answer.

Standalone SNS

aws sns create-topic --name orders-topic

TOPIC_ARN=$(aws sns list-topics --query "Topics[?contains(TopicArn,'orders-topic')].TopicArn" --output text)

aws sns publish \
  --topic-arn "$TOPIC_ARN" \
  --message '{"orderId":"order-abc123","status":"pending"}' \
  --subject "Order Created"

That publish immediately pushes to every current subscriber — email, SMS, HTTPS endpoint, Lambda, or SQS. Nothing is retained after delivery attempts complete. If you need durability — a subscriber that's slow or briefly offline shouldn't lose the message — that's the case for combining SNS with SQS, not using SNS alone.

The Fan-Out Pattern — Including the Step Everyone Skips

Diagram: fan-out delivery depends entirely on each queue's access policy explicitly trusting the topic — there's no other permission gate in this chain.

aws sqs create-queue --queue-name orders-notifications-queue
QUEUE_URL=$(aws sqs get-queue-url --queue-name orders-notifications-queue --query QueueUrl --output text)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

# This is the step that gets skipped. Without it, the subscription
# below will succeed and messages will vanish with zero errors.
aws sqs set-queue-attributes \
  --queue-url "$QUEUE_URL" \
  --attributes "{\"Policy\":\"{\\\"Version\\\":\\\"2012-10-17\\\",\\\"Statement\\\":[{\\\"Effect\\\":\\\"Allow\\\",\\\"Principal\\\":{\\\"Service\\\":\\\"sns.amazonaws.com\\\"},\\\"Action\\\":\\\"sqs:SendMessage\\\",\\\"Resource\\\":\\\"${QUEUE_ARN}\\\",\\\"Condition\\\":{\\\"ArnEquals\\\":{\\\"aws:SourceArn\\\":\\\"${TOPIC_ARN}\\\"}}}]}\"}"

aws sns subscribe \
  --topic-arn "$TOPIC_ARN" \
  --protocol sqs \
  --notification-endpoint "$QUEUE_ARN" \
  --attributes RawMessageDelivery=true

RawMessageDelivery=true matters as much as the policy step. Without it, the queue receives SNS's full JSON envelope — Type, MessageId, TopicArn, and the actual message nested inside a Message string field — instead of your original payload directly. Consumers that expect the raw body will parse the wrong structure and either error out or silently read undefined fields.

FIFO Throughput Limits

Configuration	Throughput ceiling
FIFO queue, batched	3,000 messages/second
FIFO queue, unbatched	300 messages/second
FIFO queue, high-throughput mode	70,000 messages/second
SNS FIFO topic	3,000 messages/second or 20 MB/second, whichever is hit first
Standard queue/topic	No published ceiling

High-throughput mode isn't automatic — it's an explicit setting (DeduplicationScope and FifoThroughputLimit attributes set to messageGroup and perMessageGroupId respectively) that trades a small amount of strict cross-group ordering guarantee for the throughput increase. If you need strict global ordering across all message groups, don't enable it. If your ordering requirement is per-customer or per-order (a common case), high-throughput mode is almost always the right call.

Common Mistakes

Mistake 1: Subscribing SQS to SNS without a queue access policy
The subscription API call succeeds regardless. The failure is invisible until someone notices messages aren't arriving — which could be minutes or weeks later depending on traffic patterns.

Mistake 2: Forgetting RawMessageDelivery
Consumers built against the raw payload shape break silently or throw confusing parsing errors when they receive SNS's wrapped envelope instead.

Mistake 3: Choosing FIFO by default "to be safe"
FIFO's throughput ceiling and stricter deduplication requirements are a real cost. Most systems don't actually need strict ordering — verify the requirement before paying for it in complexity and throughput headroom.

Mistake 4: Trying to convert a standard queue to FIFO
There's no such command. You create a new FIFO queue and migrate producers and consumers to it — plan for that as a deploy, not a config change.

Production Considerations

Performance: Long polling (--wait-time-seconds 20 on receive-message) eliminates the empty-receive charges that pile up from aggressive short polling — this alone is one of the two biggest SQS cost levers, the other being batching sends and receives.

Security: Scope queue access policies to the specific topic ARN, not a wildcard principal. The AWS Tip source in this article's research log describes a real incident where a queue policy pointed at a stale topic ARN after a topic recreation — silent failure, same root cause as skipping the policy entirely.

Cost: Both services bill in 64 KB payload chunks. A consistently large message body (nested JSON, embedded metadata) multiplies your bill the same way it does on EventBridge — trim payloads, pass references where you can.

Monitoring: Alarm on ApproximateAgeOfOldestMessage for SQS queues — a rising value means consumers aren't keeping up, well before the queue depth itself looks alarming.

Full Example: Fan-Out Setup Script

#!/usr/bin/env bash
set -euo pipefail

TOPIC_NAME="${TOPIC_NAME:-orders-topic}"
QUEUE_NAME="${QUEUE_NAME:-orders-notifications-queue}"

aws sns create-topic --name "$TOPIC_NAME" >/dev/null
TOPIC_ARN=$(aws sns list-topics --query "Topics[?contains(TopicArn,'${TOPIC_NAME}')].TopicArn" --output text)

aws sqs create-queue --queue-name "$QUEUE_NAME" >/dev/null
QUEUE_URL=$(aws sqs get-queue-url --queue-name "$QUEUE_NAME" --query QueueUrl --output text)
QUEUE_ARN=$(aws sqs get-queue-attributes --queue-url "$QUEUE_URL" --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

POLICY=$(cat << EOF
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sns.amazonaws.com"},"Action":"sqs:SendMessage","Resource":"${QUEUE_ARN}","Condition":{"ArnEquals":{"aws:SourceArn":"${TOPIC_ARN}"}}}]}
EOF
)

aws sqs set-queue-attributes --queue-url "$QUEUE_URL" --attributes "{\"Policy\":$(echo "$POLICY" | jq -Rs .)}"
aws sns subscribe --topic-arn "$TOPIC_ARN" --protocol sqs --notification-endpoint "$QUEUE_ARN" --attributes RawMessageDelivery=true

echo "Fan-out ready: $TOPIC_NAME -> $QUEUE_NAME"

Full source including a batched consumer with long polling: GitHub → cloud-apis/amazon-sqs-vs-sns-cli/

Conclusion

Stop framing SQS and SNS as competing choices — SNS decides who hears about something, SQS makes sure each listener actually gets to act on it without losing the message if they're briefly unavailable. The fan-out pattern combining both is the default for a reason. The one step that will actually cost you debugging time if skipped is the queue access policy: get it wrong and everything upstream reports success while the message goes nowhere.

The Multi-Tier Subcontracting Pyramid Under Pressure: What Happens to Body-Shop SIers When AI Writes the Code

Bry — Fri, 04 Sep 2026 13:30:00 +0000

Key Points

Japan's tajuu-shitauke (multi-layer subcontracting) structure routes enterprise IT work from a prime contractor (motoke) down through one or more subcontractor tiers to the engineers who actually write and test the code, a pattern that's persisted, largely unchanged, since the mainframe era.
Each tier historically takes its margin off the top before passing reduced-price work downward. A client-paid ¥1,000,000 monthly rate can leave the engineer actually doing the work with roughly ¥350,000 to ¥400,000 after cascading through two or three intermediary layers.
AI-assisted coding tools compress exactly the layer this structure depends on most: routine implementation, testing, and documentation work performed by second- and third-tier subcontractors and individual SES engineers.
The pyramid's top (client relationships, architecture, governance) looks likely to hold up. Its middle and bottom, firms and engineers whose entire value proposition is executing defined tasks at a lower price than the tier above, are facing the sharpest margin compression in the industry.
Japan's 2026 Subcontract Act reform, which expanded transparency requirements to creative and consulting services, gives lower-tier firms a real legal lever to push back on unfair pass-through pricing right as AI is already reshaping what "fair" pricing even means.

Introduction

A subcontractor relationship I've watched come under real strain recently is a fairly typical version of what's starting to happen across the industry. This is a second-tier firm that built a stable, unglamorous business supplying detailed-design and coding-and-testing engineers to a prime contractor's enterprise projects, at a rate that undercuts the tier above it. That's the entire business model: execute defined tasks, reliably, for less than the next tier up would charge. As the prime contractor's own AI-assisted tooling starts producing that same detailed-design and coding output directly, at a fraction of the cost and turnaround time, there's no lower price the second-tier firm can offer to stay competitive. It isn't underpriced. It's becoming structurally obsolete.

The pyramid this firm sits inside is one of the most distinctive features of Japan's IT industry. The prime contractor, or motoke, wins the client relationship and the overall project, then delegates substantial portions of detailed design, implementation, integration testing, and commissioning to one or more subcontractor tiers below it. Prime contractors capture the largest margins, commonly cited around 30-40%. First-tier subcontractors run 20-30%. Second-tier and below drop to 10-15%. Individual engineers, often working under System Engineering Service (SES) staffing arrangements, sit at the bottom, with compensation shrinking at each layer above them. A documented rate cascade shows the mechanism concretely: a client paying roughly ¥1,000,000 a month for an engineer's time can leave that engineer with something in the range of ¥350,000 to ¥400,000 after the intermediary tiers each take their cut.

This structure has survived for decades because it solved a real coordination problem: large enterprise projects needed a way to scale headcount up and down without every firm in the chain carrying the client relationship or the delivery risk directly. AI-assisted coding isn't attacking that coordination function. It's attacking the thing the lower tiers are actually selling, person-hours of implementation and testing labor, priced progressively lower the further down the pyramid you look. As a prime contractor, or increasingly the client itself, can generate a larger share of that same output directly, the lower tiers' entire value proposition, execute cheaply, stops being a viable business on its own.

Pyramid Layer Economics Under AI Compression

graph TD
    Client[Client] -->|pays full rate| Prime[Prime Contractor Motoke]
    Prime -->|passes down reduced rate, keeps 30 to 40 percent margin| Tier1[First-Tier Subcontractor]
    Tier1 -->|passes down reduced rate, keeps 20 to 30 percent margin| Tier2[Second-Tier Subcontractor]
    Tier2 -->|passes down reduced rate, keeps 10 to 15 percent margin| SES[Individual Engineer SES]
    AI[AI-Assisted Coding Tooling] -.->|replaces routine output of| Tier2
    AI -.->|replaces routine output of| SES

Layer	Historical Role	Historical Margin	Effect of AI-Assisted Coding
Client	Pays full contracted rate	N/A	Increasingly aware of the gap between what AI can produce and what's still being billed for
Prime contractor (motoke)	Client relationship, architecture, governance, overall delivery risk	~30-40%	Largely intact so far. Judgment and relationship work AI doesn't replace
First-tier subcontractor	Mid-scale delivery management, some architecture	~20-30%	Under pressure, but retains value where it manages integration complexity AI can't own alone
Second-tier subcontractor and below	Detailed design, coding, testing execution	~10-15%	Hardest hit. This is exactly the work AI-assisted tooling compresses
Individual engineer (SES)	Task-level implementation	Remainder after cascading cuts	Displaced where the task is routine; in higher demand where verification and AI-output review are the actual job

Recommendation: if your firm's position in this chain has always been "we execute the same task the tier above us does, for less," that position won't survive AI-assisted coding regardless of how thin you cut your margin further. The layers most likely to survive are the ones pricing judgment, integration complexity, or client trust, not marginal labor cost.

What's Likely to Survive, and What a Lower-Tier Firm Should Change Now

Firms moving up the value chain, not just down on price. The subcontractors most likely to survive won't compete on being cheaper than the tier above. They're taking on integration and governance responsibility that used to sit with the prime contractor, becoming harder to replace with AI output alone.
Firms specializing in verifying AI-generated output, not just producing more of it. Reviewing and validating AI-generated code against a client's actual production constraints is turning out to require exactly the kind of experienced-engineer judgment that routine implementation work didn't.
Firms using Japan's 2026 Subcontract Act reform actively. The reform's expanded transparency requirements give lower-tier firms real standing to contest unfair pass-through pricing. Firms that understand and use this leverage should renegotiate from a stronger position than firms that don't know the reform applies to them.
Firms consolidating instead of competing on the same shrinking margin. Some second- and third-tier firms are merging capabilities, combining a compressed coding-and-testing practice with a smaller firm's domain expertise, to offer something closer to the first-tier's integration value than to commodity execution.
Firms that change nothing. These are the most exposed to winding down, getting acquired for their client relationships and remaining engineers, or shrinking into a much smaller commodity-execution niche with correspondingly thinner margins.

Questions to Ask Your Team

Where does our firm actually sit in our clients' delivery chains, and is our value proposition still "we execute this cheaper than the tier above," or has it genuinely moved to something AI-assisted tooling can't produce directly?
If a prime contractor above us started generating our layer's output directly with AI tooling, what would we still have left to sell them?
Are we aware of what Japan's 2026 Subcontract Act reform actually changed for transparency and fair pricing in multi-tier arrangements, and have we used it in a renegotiation?
Have we tested whether our engineers are more valuable reviewing and validating AI-generated output than producing new output themselves, and are we pricing that shift yet?

Conclusion

The multi-tier subcontracting pyramid isn't collapsing, and it's not going to. What it built its lower tiers on, the assumption that there's always a cheaper price for the same routine task one layer further down, is what's breaking. Firms most likely to weather the pressure are moving toward judgment, integration, and verification work. Firms trying to survive by cutting their price further into an already-thin margin are the most exposed. The prime-contractor layer looks likely to hold up fine, because it was never really selling person-hours in the first place. Everyone underneath it needs to figure out, quickly, whether they've been selling person-hours the whole time, and if so, what else they actually have.

AI Cost Optimization: A Business Guide to LLM API Budgeting

Bry — Tue, 01 Sep 2026 15:00:00 +0000

Key Points

LLM API costs double every six months for most growing organizations — model selection alone can reduce your bill by 50–80% without changing what you build.
Prompt caching and response caching together eliminate the majority of repeated compute costs, saving 60–90% on input tokens for the right workloads.
Batch processing cuts per-token prices by 50% for any task that doesn't require a real-time answer.
Visibility comes first: you cannot optimize spend you cannot see — budget alerts, cost attribution by feature, and a 4-phase adoption roadmap make savings stick.

The AI Bill Nobody Expected

The CFO opens the cloud invoice. Line 37: LLM API usage — $48,000. Last month it was $22,000. Three months ago it was $4,000.

This is not a hypothetical. Enterprise LLM API spending doubled in under six months as teams moved from experiments to production workloads, and most organizations had no framework in place to manage the acceleration. Unlike compute or storage — where costs scale predictably with users — AI API costs are shaped by choices made at the code level: which model you pick, how you structure requests, whether you cache responses, and whether you batch non-urgent jobs. Those choices are invisible to finance and often undiscussed with engineering leadership.

I've watched this play out across organizations at different scales — from startups where one engineer's prototype quietly became a $30K/month production workload, to larger teams where cost attribution was a year-long engineering project after the fact. This guide gives business leaders — CTOs, product managers, and anyone signing cloud invoices — a clear, non-technical framework for getting AI spend under control without sacrificing the capabilities your teams depend on.

Why AI Costs Spiral

The pattern is consistent across organizations of every size: costs spiral because of three compounding defaults.

Teams default to the most powerful model. When engineers prototype an AI feature, they reach for the frontier model — it produces the best output, reduces debugging time, and avoids internal debates about quality. That default rarely gets revisited after launch. The same flagship model handling nuanced legal analysis ends up powering an FAQ bot that answers "What are your business hours?"

Nobody tracks token usage by feature. Unlike server costs that map neatly to infrastructure, LLM costs are buried in a single API line item. Without attribution — which feature consumed how many tokens — there is no feedback loop. Expensive patterns persist indefinitely because nobody can see them.

Users trigger expensive chains without limits. In production, one user action can silently trigger a cascade of API calls: an initial query, a summarization step, a classification call, a response synthesis step. Each hop multiplies the token count. Without circuit breakers or cost caps, a single power user can consume more than the entire intended monthly budget in a week. I've seen a five-step document analysis pipeline — perfectly reasonable in isolation — go uncapped in production and rack up more in a weekend than the team had budgeted for the entire month, because one user stress-tested it with 400-page PDFs.

How LLM Pricing Works

AI APIs charge by the token — a chunk of text roughly equivalent to 0.75 words in English. A 100-word paragraph is approximately 130 tokens. Two separate charges apply to every request: input tokens (what you send to the model) and output tokens (what the model generates in response).

Output tokens are significantly more expensive. Across all major providers, output pricing runs 4–5 times higher per token than input pricing. This matters because it means that asking a model to write a long report costs far more than asking it to classify a sentence — even if both requests contain the same amount of input text.

Here is the current pricing landscape for the major models your engineering team is most likely using:

Model	Input $/1M tokens	Output $/1M tokens	Context	Best For
Claude Opus 4.8	$5.00	$25.00	200K	Complex reasoning, multi-step analysis, legal/financial review
Claude Sonnet 4.6	$3.00	$15.00	200K	Balanced quality and cost; most production workloads
Claude Haiku 4.5	$1.00	$5.00	200K	High-volume, simple tasks: classification, extraction, FAQ
GPT-4o	$2.50	$10.00	128K	General-purpose; strong tool-use and structured output
Gemini 2.5 Pro	$1.25	$10.00	1M	Very long documents; high-context summarization

Pricing as of June 2026. Verify current rates at official provider pricing pages before committing to budget projections.

Context window refers to the maximum amount of text a model can consider in a single request. Larger context windows are critical for processing lengthy contracts, codebases, or conversation histories — but they also increase token costs proportionally.

The Biggest Lever: Model Selection

If you take one action from this guide, make it this: match task complexity to model tier.

The cost gap between tiers is not incremental — it is multiplicative. Using Claude Opus for a simple classification task costs roughly 20 times more per request than using Claude Haiku for the same task. At scale, that ratio becomes a six-figure annual line item for a mid-sized product.

The business principle is straightforward: the model's capability should match the task's demand. Sending a routine FAQ to your most powerful model is the equivalent of deploying a senior partner to answer reception phone calls.

Use this decision framework when evaluating which model tier a task belongs in:

My recommendation: start the audit with your highest-volume features, not your most complex ones. FAQ responses, form classification, data extraction, and templated summaries almost always belong in the Haiku tier — and those tend to be your volume leaders. Complex document analysis, strategic synthesis, and nuanced content generation belong in the Sonnet or Opus tier, but they're rarely the source of the runaway bill.

Prompt Caching: Pay Once, Reuse Many Times

Every AI application sends instructions to the model with every request. A customer service bot might include a 2,000-word system prompt describing the company's products, policies, and tone guidelines — and that prompt gets sent, and charged, with every single user message.

Prompt caching solves this. When you send the same instructions repeatedly, the provider stores that content in a temporary cache. Subsequent requests that use the same cached prefix are charged at 10% of the normal input price — a 90% discount on those tokens.

For applications with long, stable system prompts — support bots, document processors, product assistants — prompt caching typically reduces input token costs by 60–90%. The cached content stays valid for a configurable duration (5 minutes or 1 hour on Anthropic's API) and is automatically refreshed when accessed.

From a business perspective: if your engineering team is not using prompt caching on any application that includes a system prompt longer than a few hundred words, you are paying full price for tokens you have already paid for. In practice, this is the first thing I check when a team tells me their AI costs feel out of control — it is almost always uncached, and enabling it is usually a one-day engineering task that pays for itself within the first billing cycle.

Response Caching: Skip the API Call Entirely

Prompt caching operates at the provider level and reduces the cost of repeated inputs. Response caching operates at your application level and eliminates the API call entirely for repeated questions.

The concept is simple: when a user asks a question, your application checks a local database before calling the AI API. If the same question has been asked before and the answer is still valid, return the stored answer. No tokens consumed, no API cost, no latency.

This approach works well for:

FAQ bots: The same 200 questions account for 80% of support volume in most businesses
Product descriptions: Thousands of users view the same product content
Templated reports: Weekly summaries generated from the same data schema

It does not work well for:

Real-time data requests: Questions about live inventory, current prices, or today's metrics require fresh API calls
User-specific personalization: Responses that incorporate individual user history or preferences cannot be safely reused across users

The infrastructure investment is modest — a simple key-value store or database table — and the return on high-repetition workloads is significant. I use response caching as the first conversation to have with any team running a support bot or product FAQ: the ROI calculation is fast, the engineering effort is low, and approval is easy when you can show finance that 60% of calls will cost nothing after day three. On FAQ-heavy applications, organizations typically see 40–70% of requests served from cache after the first few days of production traffic.

Batch Processing: Trade Speed for 50% Off

Real-time API calls — where your application waits for an immediate response — carry a premium price. For tasks where a response is not needed within seconds, every major provider offers a batch processing API at 50% off standard pricing.

Anthropic's Batch API and OpenAI's Batch API both accept large volumes of requests submitted at once and return results within hours, typically overnight. The same models, the same quality — at half the cost.

Batch processing is well-suited for:

Overnight report generation
Bulk document classification or extraction
Weekly content summarization jobs
Training data generation or quality review
Large-scale sentiment analysis

The key trade-off is latency. Batch jobs are asynchronous — you submit the work and retrieve results later. For any workflow where a user is waiting for a response, batch processing is not appropriate. For workflows that run on a schedule or process accumulated data, it is one of the simplest cost reductions available.

A team spending $10,000 per month on overnight AI processing jobs that currently use the real-time API can reduce that line item to $5,000 with a single architectural change. If your organization runs any scheduled AI jobs — weekly digests, monthly classification sweeps, overnight data enrichment — batch processing should be a standing agenda item in your next budget review: the savings are predictable, the risk is low, and the approval case is straightforward.

Budget Controls and Monitoring

Cost optimization techniques only work if you can see whether they are working. Visibility is the prerequisite for everything else.

Set spend alerts in provider dashboards. Both the Anthropic Console and the OpenAI usage dashboard allow you to configure email alerts when monthly spend crosses a threshold. Set alerts at 50%, 75%, and 100% of your planned monthly budget — not just at the limit. Early warning gives engineering time to investigate before costs become a crisis.

Implement soft limits in application code. Provider-level alerts fire after costs have already accumulated. Application-level limits stop the accumulation. Work with your engineering team to implement per-user, per-feature, and per-workflow token budgets. When a workflow hits its limit, it either degrades gracefully (using a cheaper model) or queues the request for batch processing rather than failing loudly.

Track cost per feature, per user, per workflow. A single API line item tells you nothing actionable. Attribution — which feature consumed which tokens — is what makes optimization possible. Organizations that implement cost attribution consistently report identifying two or three "cost sink" features that account for the majority of spend, often features that had never been considered high-cost during development.

Adoption Roadmap: 4 Phases to Controlled AI Spend

Cost optimization is not a one-time project. It is an ongoing practice. Organizations that achieve and sustain 50–80% cost reductions follow a consistent phased approach.

Phase 1 — Measure your baseline. Before changing anything, establish what you are spending, by feature and model. This takes two to four weeks and requires engineering effort to add cost attribution to your existing AI calls. Without this data, you are optimizing blind.

Phase 2 — Identify expensive patterns. With attribution in place, surface the top cost drivers. Typical findings: a high-volume feature using the wrong model tier, a long system prompt without caching, a batch-eligible workflow running in real-time.

Phase 3 — Apply targeted optimizations. Address findings in order of impact. Model right-sizing is usually first — it requires minimal engineering effort and delivers immediate, compounding returns. Prompt caching is second. Response caching and batch processing follow.

Phase 4 — Monitor and iterate. New features introduce new cost patterns. Set a recurring monthly review cadence where engineering and finance align on spend vs. budget and flag new anomalies. The loop between Phase 4 and Phase 2 is what prevents costs from spiraling again after the initial optimization effort.

Questions to Ask Your Engineering Team

Before your next budget review or AI project kickoff, bring these questions to your engineering leadership:

Which AI features are using which models? Can you show me a list of every production AI feature and the model tier it currently runs on?
Do we have cost attribution? Can we see our monthly AI spend broken down by feature, workflow, or user segment — not just as a single total?
Are we using prompt caching on any feature with a long system prompt? If a feature sends the same instructions with every request and is not using prompt caching, what is the estimated monthly savings from enabling it?
Which AI workflows run in real-time that could run overnight? Is there a list of report generation, bulk processing, or classification jobs that currently use the real-time API but don't need to?
Do we have spend alerts configured? At what thresholds do we receive notifications, and who receives them?
Are there application-level rate limits or cost caps per user? What prevents a single user or workflow from consuming an outsized share of our monthly token budget?
When did we last review whether each feature is on the right model tier? Has any feature been moved from a flagship model to a lower-cost tier after initial development — or do we still run the same models we used during prototyping?

Conclusion

AI API costs are not a fixed overhead — they are a function of architectural decisions your engineering team makes every day. The organizations I've seen get this right share one thing: they treated visibility as non-negotiable from the start, not as a cleanup project after the bill became alarming. Once you can see where the money goes, the optimizations follow naturally — and they tend to be faster and cheaper to implement than anyone expected. The goal is not to spend less on AI. It is to stop funding waste so the budget can go toward the AI capabilities that actually move your business forward.

From Billing Hours to Billing Outcomes: New Contract Structures for AI-Assisted SI Projects

Bry — Wed, 26 Aug 2026 15:00:00 +0000

Key Points

Time-and-materials billing gives the vendor no financial incentive to finish faster. Every extra hour is another invoice line. AI-accelerated delivery is making that misalignment obvious to clients who can see the work speeding up while the invoice logic stays the same.
73% of consulting clients already tell researchers they prefer value-based or outcome-driven pricing over hourly rates. That preference is likely to harden into the default contract structure across most of the systems-integration market within the next several years.
Outcome-based contracts don't eliminate risk. They move it. Vendors absorb schedule risk on defined deliverables, which is forcing real changes to scoping discipline, delivery governance, and which projects an SI will even accept.
Time-and-materials isn't disappearing. It survives, correctly, for genuinely exploratory work where the scope can't be defined upfront. The mistake to watch for is applying it to well-defined work out of habit, not because T&M is inherently wrong.
IDC's forecast, 30% of IT services contracts outcome-based by 2029, is starting to look conservative. If AI-driven delivery speed keeps compounding, outcome-based and hybrid subscription-plus-usage structures could cover a majority of new systems-integration engagements within the decade, in markets where clients have any real pricing leverage.

Introduction

Why the Man-Month Is Dying: How AI Broke IT Services' Oldest Pricing Unit in this thread covered why the man-month is dying as a pricing unit. This one covers what's actually starting to get written into contracts once firms stop defaulting to it, because "bill outcomes, not hours" is a slogan, and slogans don't survive contact with a real statement of work.

The economic case against pure time-and-materials was already documented before AI made it urgent. PMI's Pulse of the Profession data shows T&M engagements running 23% over budget on average, which on a $50,000 project is $11,500 of unplanned client spend, with no penalty to the vendor for the overrun. AI-assisted delivery hasn't fixed that structural misalignment. If anything, it's making it worse in the near term, because a vendor billing by the hour has an active disincentive to let AI tools cut delivery time, and clients increasingly can tell. Futurum Research already found 73% of consulting clients favor value-based or outcome-driven pricing over hourly rates, largely because AI's delivery-speed gains have made time-based billing look indefensible rather than merely inefficient.

What's replacing it won't be a single template. Outcome-based contracts, payment tied to specific, measurable deliverables with acceptance criteria, are taking over the well-defined end of the market: data migrations, defined integrations, modernization projects with a clear "done" state. Hybrid structures, pairing a base subscription for fixed costs with a usage or outcome-based component above a baseline, are showing up in ongoing managed-service relationships. Time-and-materials should survive where it always made sense: genuinely exploratory work, early-stage product discovery, engagements where the client themselves doesn't yet know the final scope. The mistake to avoid over the next few years isn't choosing outcome-based pricing. It's applying whichever model a firm already knows how to bill, regardless of which one actually fits the work.

Contract Model Fit by Project Type

Project Type	Best-Fit Model	Why
Defined data migration or system integration	Outcome-based	Scope and success criteria are specifiable in advance; AI-accelerated delivery becomes vendor margin, not client discount
Early-stage product discovery, undefined scope	Time-and-materials	Scope genuinely can't be fixed upfront; forcing outcome pricing here just relabels the guesswork
Ongoing managed services / maintenance	Hybrid subscription + usage	Base cost stays predictable for the client; usage component tracks real variability instead of headcount
Fixed-scope modernization with a hard deadline	Fixed-price with milestone acceptance	Client needs cost certainty; vendor absorbs schedule risk in exchange for a defined, unchanging scope
Regulatory-driven compliance rebuild	Outcome-based, tied to audit-passable state	Client cares about a certifiable end state, not hours logged getting there

Recommendation: match the contract model to how well-defined the scope actually is, not to which model your firm is most comfortable billing. An SI that only offers T&M today is telling well-defined-scope clients, correctly, that it hasn't done the scoping work outcome-based pricing requires.

Making the Switch: A Phased Approach

Build the measurement infrastructure before changing the invoice template. Outcome-based billing requires a checkable definition of "done": acceptance criteria, a metering system for usage, or an audit standard. Firms that flip their contract language before building this lose money on undefined "outcomes."
Start with your most well-understood, most frequently repeated project type. A data migration you've delivered fifty times is far easier to price on outcome than a novel integration you've never scoped before. Don't lead the transition with your hardest, most novel work.
Renegotiate existing T&M relationships in the open, not by stealth. Clients who discover a vendor quietly pocketing AI-driven speed gains under an unchanged T&M invoice react far worse than clients told directly that pricing is moving to outcomes and shown why.
Keep a genuine T&M option for genuinely exploratory engagements. Retiring T&M entirely just pushes clients with undefined scope toward vendors willing to be honest that some work can't be priced by outcome yet.
Revisit pricing on every engagement renewal, not just new business. The real lesson behind IDC's forecast isn't the 30% number. It's that firms which wait for contract renewal cycles to force the pricing conversation, rather than initiating it, cede the framing to whichever competitor gets there first.

Questions to Ask Your Team

For our current book of business, how many engagements are still billed by the hour purely out of habit, on work that's actually well-defined enough to price on outcome?
If a client asked us directly why our pricing model doesn't reward them for AI-driven delivery speed, do we have an honest answer, or an evasive one?
Do we have real acceptance criteria and measurement infrastructure in place before we've committed to an outcome-based number, or are we guessing at "outcomes" the same way we used to guess at hours?
Are we keeping time-and-materials available for the engagements that genuinely need it, or treating it as a legacy model to be phased out everywhere regardless of fit?

Conclusion

The contract structures most likely to survive the man-month's decline won't be chosen because they sound better in a sales deck. Outcome-based pricing is winning the well-defined end of the market because it aligns vendor incentive with AI-driven speed instead of punishing it. Time-and-materials should survive at the genuinely exploratory end because forcing a false "outcome" definition onto undefined scope just relabels the same uncertainty. The firms most likely to struggle through this transition aren't the ones that pick the wrong model. They're the ones that pick one model and apply it everywhere, regardless of whether the work in front of them actually fits it.

DEV Community: Bry

Serverless vs Containers: A Manager's Guide

Key Points

Introduction

The Core Question

The Analogy That Makes This Clear

What This Means for Your Team

The Cost Model Without the Math

What Serverless Cannot Do

How Most Organizations Actually Do This

Decision Flowchart

Adoption Roadmap

Questions to Ask Your Engineering Team

Conclusion

Further Reading

Junior Engineer Pipeline Collapse: Who Trains the Next Generation When AI Does the Grunt Work

Key Points

Introduction

Firms Cutting Junior Hiring vs. Firms Restructuring It

Rebuilding a Junior Pipeline That Actually Produces Judgment

Questions to Ask Your Team

Conclusion

Further Reading

An Engineer's Reflection: Watching the SI Business Model Change From the Inside

Key Points

Introduction

An Engineer's Position: What Made You Valuable, Then and Now

What I'm Telling That Colleague

Questions to Ask Yourself, Not Just Your Team

Conclusion

Further Reading

Azure Service Bus: Queues and Topics for When Event Grid Isn't Enough

Key Points

Prerequisites

Introduction

The Tier Gate

Building a Session-Enabled Queue

Building a Filtered Topic Subscription

When to Reach for Service Bus Instead of Event Grid

Common Mistakes

Production Considerations

Full Example: Send and Receive with Sessions (TypeScript)

Conclusion

Further Reading

Data Residency and APPI Compliance: What Cloud Vendors Entering Japan Need to Know

Key Points

Introduction

APPI's Extraterritorial Reach

Cross-Border Transfer: The Two Legitimate Paths

Breach Notification: A Tighter Timeline Than It Looks

Comparison: No Data Residency Commitment vs Multi-Region Flexibility vs Japan-Only Hosting

Pros and Cons

Advantages of Getting This Right Early

Disadvantages and Risks

Is This Right for You?

Implementation Approach

Phase 1: Scope and Gap Assessment (Weeks 1–3)

Phase 2: Choose and Implement a Transfer Mechanism (Weeks 4–10)

Phase 3: Breach Response Readiness (Weeks 8–12, overlapping Phase 2)

Phase 4: Ongoing Compliance (Month 4+)

Cost Considerations

Questions to Ask Your Team

Conclusion

Further Reading

The New SI Deliverable: Selling Judgment and Integration, Not Headcount

Key Points

Introduction

Old SI Deliverable vs. New SI Deliverable

Building a Judgment-Based Deliverable: What Firms Are Changing Internally

Questions to Ask Your Team

Conclusion

Further Reading

Building a RAG Pipeline from Scratch: Embeddings, Retrieval, and Claude

Key Points

Introduction

What RAG Is — and What It Replaces

RAG Pipeline Architecture

Chunking Strategies

Fixed-Size Chunking

Recursive Chunking

Local Embeddings: `sentence-transformers`

The `RAGPipeline` Class