DEV Community: Abhijit Das

AI Model Routing in Production: The Architecture Pattern Your Dev Team Probably Skipped

Abhijit Das — Fri, 24 Jul 2026 06:25:36 +0000

Originally published on madgeek.ai. We build production AI systems and custom software for enterprises.

Model routing is the practice of directing different AI tasks to different model tiers based on what each task actually requires. A classification task goes to the cheapest model. A summarisation task goes to a mid-tier model. Complex reasoning -- if it exists in the workflow -- goes to the most capable model. Each task hits the cheapest endpoint that meets its quality threshold.

This pattern reduces API costs by 80-95% in most production AI systems. It is the single most impactful architectural decision for AI cost efficiency. And most development teams skip it.

Why is model routing not standard practice?

Because it requires more engineering work upfront for a benefit the client doesn't see until production volume arrives.

A development team building without routing picks one model, writes one set of prompts, and ships. A team building with routing has to decompose every workflow into discrete tasks, evaluate each task across multiple model tiers, engineer task-specific prompts for each tier, build the routing logic itself, and instrument monitoring to track quality and cost per route. That's 3-5x more engineering work for a result that looks identical to the client during the demo.

The cost difference only becomes visible at scale. At 100 API calls per day, nobody notices. At 10,000, it's $300/day versus $16/day. At 50,000, the difference is existential -- $1,500/day versus $80/day.

Vendors building AI systems on fixed-price contracts have no incentive to do this extra work. They're paid to deliver a working product, not to optimise its operating cost. The client pays the API bill, not the vendor.

How does model routing actually work?

A routing layer sits between your application logic and the AI provider's API. When a task needs to be processed, the router evaluates what kind of task it is and directs it to the appropriate model tier. There are three common routing strategies:

Static routing assigns a fixed model tier to each task type at build time. Extraction tasks always go to Haiku. Summarisation always goes to Sonnet. This is the simplest approach and handles 90% of use cases. The mapping is determined during the architecture phase by testing each task across model tiers and selecting the cheapest one that meets the quality threshold.

Complexity-based routing analyses the input before selecting a model. Simple inputs (short, structured, standard format) go to the cheap tier. Complex inputs (long, unstructured, edge-case-heavy) go to a more capable tier. This works well for tasks where input complexity varies significantly -- most inputs are easy, but occasional ones are genuinely hard.

Fallback routing starts with the cheapest model and escalates. Send the task to Haiku first. If the output fails a quality check (confidence score, format validation, consistency check), automatically retry on Sonnet. If that fails, escalate to Opus. Most calls resolve on the first attempt. The system only pays for expensive models when cheaper ones demonstrably can't handle the specific input.

In practice, we use a combination. Static routing for the bulk of well-understood tasks. Fallback routing for tasks where edge case frequency is unpredictable.

What does the routing architecture look like in code?

The routing layer isn't a separate service. It's an architectural pattern built into the AI layer of the application. At its simplest, it's a configuration map:

Each task in the system has a defined skill -- a discrete, testable function with its own prompt, model assignment, and quality threshold. The orchestrator calls skills in sequence, each routing to the appropriate model. The task definition includes the model tier, the prompt template, the expected output schema, and the quality validation function.

This is what we mean by agent skill design. Each skill is a self-contained unit that can be tested independently, swapped to a different model tier without touching the rest of the system, and monitored for cost and quality in isolation.

What's the catch?

Two things.

First, each model tier has different prompt requirements. A prompt that works on Opus won't necessarily work on Haiku. Opus is forgiving -- it infers intent from vague instructions. Haiku is literal -- it needs explicit structure, output format definitions, and edge case handling built into the prompt. Writing effective prompts for cheaper models is a different skill from writing prompts for capable ones.

Second, you need quality validation at the task level. When you route a task to a cheaper model, you need a way to verify the output meets your threshold. For structured tasks (extraction, classification), this is straightforward -- schema validation, field completeness checks, confidence scoring. For generative tasks (summarisation, content creation), quality validation is harder and sometimes requires a second model call to evaluate the first one's output.

Both of these are solvable problems. They're just not problems that a team doing basic API integration thinks about, because they don't surface until production volume makes the cost problem visible.

Can you retrofit routing into an existing system?

It depends on how the system was built.

If tasks are already separated into distinct function calls with clean interfaces, adding routing is a configuration exercise. Define the model tier per task, write task-specific prompts for each tier, add validation, deploy. Timeline: 2-4 weeks.

If the system uses monolithic prompts -- one API call performing multiple tasks in a single pass -- routing requires decomposing the pipeline first. Break the monolithic call into discrete tasks, test each task independently across model tiers, build the routing layer, and rebuild the orchestration. Timeline: 6-10 weeks.

In both cases, the product doesn't change. The user sees the same interface, provides the same inputs, gets the same outputs. What changes is the engineering underneath -- and the monthly API cost.

What does this look like in a real production system?

We built an operations AI platform that scaled from 50 to 80+ agents in three months. The system analyses call recordings, scores agent performance, identifies coaching opportunities, and generates reports.

Each of those is a different task with different complexity requirements. Transcription preprocessing runs through a lightweight model. Scoring against a predefined rubric -- pattern matching with structured output -- runs on the cheapest tier available. Identifying nuanced coaching opportunities that require understanding conversational dynamics routes to a mid-tier model. Generating management-level performance summaries that need to be well-written routes to a capable model.

If every call had gone through one model, the cost per agent would have been high enough that scaling from 50 to 80+ agents would have been economically impossible. Model routing was the difference between a system that could scale and one that couldn't survive its own growth.

That's what a properly architected AI system looks like. Not one model doing everything. Multiple models, each handling what they're best at, orchestrated by a routing layer that was designed before the first line of code was written.

How Much Does AI Development Cost in 2026?

Abhijit Das — Fri, 24 Jul 2026 05:43:42 +0000

A custom AI system costs between $15,000 and $500,000+ to build in 2026, depending on three variables: the complexity of the decision the AI makes, how deeply it integrates with your existing systems, and whether it needs to operate autonomously or with human oversight. Most companies exploring AI for the first time should budget $40,000-$80,000 for a production-ready system that handles one well-defined business process.

That range surprises people. They expect either $5,000 (a wrapper around ChatGPT) or $2 million (an enterprise AI transformation). The reality sits in the middle, and the gap between a $40K build and a $400K build is rarely about the AI itself — it is about what surrounds it.

What drives the cost of AI development?

Four factors determine 90% of your AI development budget. The AI model itself — the part most people fixate on — is usually the smallest line item.

Data preparation and pipeline engineering consume 40-60% of most AI project budgets. If your business data lives in spreadsheets, three different databases, and someone's email inbox, the work to clean, structure, and pipe that data into a model dwarfs the cost of the model itself. A manufacturer we worked with had 11 years of cost estimation data spread across ERP exports, Excel files, and handwritten notes. Structuring that data for an AI cost estimator took longer than building the estimator.

Integration complexity is the second driver. An AI system that reads data from one source and outputs a recommendation in a dashboard is straightforward. An AI system that connects to your ERP, pulls live inventory, cross-references supplier pricing, runs a cost model, and writes the result back into your procurement workflow — that is a fundamentally different engineering challenge. Each integration point adds 15-25% to the build cost.

Autonomy level matters more than model sophistication. A system that suggests and waits for human approval costs less to build and validate than one that acts independently. Autonomous AI systems need guardrails, fallback logic, monitoring dashboards, and audit trails — each adding engineering hours.

Compliance and security requirements can double the cost for regulated industries. Healthcare, financial services, and insurance AI builds require HIPAA/SOC 2 compliance, data residency controls, and audit logging that a marketing AI tool does not.

What does each price range actually buy?

The table below maps budget ranges to what you can realistically expect to get built. These are based on production projects, not estimates.

Budget Range	What You Get	Example	Timeline
$15K-$30K	Single-task AI automation. One data source, one output, human-in-the-loop.	AI document classifier, chatbot with knowledge base, lead scoring model	4-8 weeks
$40K-$80K	Production AI system. Multiple data sources, 1-2 integrations, monitoring, handles one complete business process.	AI quality monitoring for operations teams, automated procurement routing, cost estimation engine	8-16 weeks
$80K-$200K	Multi-process AI platform. 3+ integrations, autonomous operation with guardrails, admin dashboard, role-based access.	Enterprise approval workflow with AI routing, multi-department operations platform with AI agents	3-6 months
$200K-$500K+	Enterprise AI transformation. Custom models, multi-system integration, compliance-grade security, ongoing model training.	AI-powered underwriting platform, supply chain optimization, custom LLM trained on proprietary data	6-12+ months

Why do AI projects cost more than expected?

The most common budget overrun in AI projects is not the AI. It is everything the AI needs to work correctly.

Companies budget for the model and forget the infrastructure. A production AI system needs data pipelines, error handling, monitoring, fallback logic, user interfaces, access controls, and logging. The model is the engine. Everything else is the car. You cannot drive an engine.

Scope creep from "AI excitement" is the second killer. A project starts as "automate invoice classification" and by week three someone asks, "Can it also predict cash flow?" Those are two different AI systems with different data requirements, model architectures, and validation needs. Each new capability is not a feature request — it is a separate engineering workstream.

Data quality gaps surface late. You discover in month two that 30% of your training data is mislabelled, or that three departments encode the same field differently. Cleaning that data is unplanned work that pushes timelines and budgets.

How do ongoing AI costs compare to the initial build?

Plan for ongoing costs of 15-25% of the initial build cost per year. A $60K AI system typically costs $9,000-$15,000 annually to maintain and improve.

That breaks down into three buckets. API and infrastructure costs run $500-$5,000 per month depending on volume and model tier. Companies using model routing — sending simple queries to cheaper models and only routing complex decisions to expensive ones — reduce these costs by 60-80%. For more on the architecture behind this, see our guide to AI model routing.

Model monitoring and retraining is the second bucket. AI models degrade as your business changes — new products, new customer segments, new regulations. A model trained on 2024 data makes increasingly wrong predictions in 2026. Quarterly model evaluation and annual retraining cycles keep performance stable.

Feature iteration is the third. Once an AI system is in production and delivering results, the business finds adjacent problems worth solving. The operations team that got automated invoice classification now wants automated approval routing. These are incremental builds — typically 20-40% of the original cost per new capability.

What is the difference between AI development cost in India vs the US?

AI development in India costs 40-60% less than equivalent work in the US, UK, or Western Europe. A $120K US-built AI system costs $50K-$70K when built by a senior engineering team in India.

The cost difference is entirely in engineering labour rates. API costs (OpenAI, Anthropic, AWS) are identical regardless of where the team sits. Infrastructure costs are the same. The models do not care where the developer lives. What differs is the rate for the engineers who build the data pipelines, integrations, interfaces, and monitoring around the model.

The risk with low-cost offshore AI development is not the rate — it is the team. India has top-tier AI engineers and commodity shops charging $15/hour. The difference between them is process: code reviews, CI/CD pipelines, structured delivery, and engineering leadership that stays involved past the sales call. If the team cannot explain their model evaluation methodology in the first conversation, the low rate will cost you more in rework than the savings.

Should you build custom AI or use an off-the-shelf platform?

Use off-the-shelf AI tools when your problem is generic and your data is not a competitive advantage. Build custom when the AI needs to understand your specific business logic, connect to your internal systems, or make decisions that require your proprietary data.

A marketing team that needs AI-generated social media posts should use an existing tool. A manufacturer that needs AI to estimate job costs based on 11 years of historical bids, material pricing, and machine utilization — that is a custom build. No off-the-shelf tool has your cost data, your margin targets, or your supplier relationships.

The crossover point is usually around $2,000-$3,000 per month in SaaS AI tool costs. If you are paying that much for a tool that covers 60% of what you need and your team spends hours working around the other 40%, the math favours a custom build that does exactly what you need. A $60K custom system paid back in 18-24 months against a $3K/month SaaS bill is a straightforward ROI calculation.

Where do companies waste money on AI development?

The biggest waste is building a proof of concept that cannot become a production system. A PoC that uses hard-coded data, skips error handling, and runs on a developer's laptop proves the AI model works. It proves nothing about whether the system works in your business. Then you pay to rebuild everything from scratch because the PoC architecture cannot scale, integrate, or handle real-world edge cases.

The second waste is over-engineering the model before validating the business case. Companies spend $200K on a custom-trained model when a prompted GPT-4o with good data retrieval would have delivered 90% of the accuracy at 20% of the cost. Start with the simplest approach that solves the problem. Move to custom models only when you hit a measurable accuracy ceiling with prompted general-purpose models.

The third is ignoring AI API cost optimization from day one. Teams that route every request through the most expensive model — regardless of complexity — pay 5-10x what they need to. A tiered approach where simple classifications hit a small model and only complex reasoning hits a large model reduces ongoing costs dramatically without measurable accuracy loss.

How should you budget for your first AI project?

Start with one process, not a platform. Pick the business process where AI will have the most measurable impact — usually the one with the highest volume of repetitive decisions and the clearest data trail.

Budget $40,000-$80,000 for the build and $1,000-$2,500 per month for ongoing operation. Expect 8-16 weeks from kickoff to production. Plan for a 2-4 week discovery phase where the engineering team maps your data, defines the AI's decision boundaries, and identifies integration points — this phase prevents the budget overruns described above.