DEV Community: Waleed Siddiqui

How Much Does a SaaS MVP Cost? What Moves the Number

Waleed Siddiqui — Mon, 02 Nov 2026 13:00:00 +0000

Founders ask "how much will this cost?" before almost anything else, and most answers online are either a vague "it depends" or a range so wide it is useless. Here is ours, and — more usefully — what moves a project from one end of it to the other.

Our SaaS MVP package starts at $38,500 and typically takes 8 to 12 weeks from signed scope to launch. A narrower validation build can cost less; a product with iOS and Android apps, payments and several integrations costs more. Those numbers are for a production-ready first version on a codebase you keep building on, not a clickable prototype.

What an MVP budget actually pays for

An MVP is a test: you want to learn, quickly and cheaply, whether people will use and pay for the thing. That means the budget should buy three things and nothing else:

One core workflow, done well — the single job your first users hire the product to do.
Real signals — real sign-up, real payment if revenue is part of the test, and analytics on the actions that matter.
A foundation you won't throw away — authentication, data model, payments and deployment built properly, so version two extends the MVP instead of replacing it.

Everything else — admin dashboards with charts, five integrations, a native app "because competitors have one" — is a candidate to wait.

The seven drivers that move the number

Screens are rarely what makes an MVP expensive. Rules are. These are the drivers we look at when we estimate:

Driver	Cheaper end	Expensive end	Why it costs
User roles	One type of user	Buyers and sellers, admins, team members with different rights	Every role multiplies screens, permission rules and test cases
Billing model	One plan via Stripe Checkout	Tiers, trials, seat or usage pricing, upgrades and downgrades	Subscription edge cases (failed payments, proration, scheduled downgrades) need careful webhook handling
Integrations	None, or one well-documented API	CRM, calendar, accounting, SSO, legacy systems	Each integration brings auth, rate limits, retries and two-way sync decisions
Platforms	Responsive web app	Web plus iOS and Android	Mobile adds store review, in-app purchase rules and release management
AI features	None, or one well-scoped feature	AI is the core value	You need an evaluation set, cost controls and fallbacks, not just an API call
Compliance and data sensitivity	Standard SaaS data	Health, financial or children’s data	Audit logging, access controls, data residency and vendor agreements
Data migration	Starting fresh	Moving users and payments off an existing tool	Mapping, reconciliation and a cut-over plan for live customers

Two of these come from projects in our portfolio. A membership platform we built needed tiered Stripe subscriptions, scheduled downgrades, payment-failure grace periods and a production data migration off a no-code tool — the billing and migration rows accounted for a large share of the engineering. Our own mobile product needed iOS and Android, in-app subscriptions and strict privacy rules enforced in the database. Neither was expensive because of its screens.

What we never cut, even in an MVP

Cutting scope is how you control an MVP budget. Cutting these is how you end up paying twice:

Authentication and access rules. Retrofitting security after users have data is expensive and risky. Use a proven auth provider and enforce access on the server — ideally in the database itself.
Real payments, if revenue is part of the test. "We'll invoice manually" works for five customers; build Stripe (web) or RevenueCat (in-app purchases) properly when payment is the signal you need. Treat the payment provider's webhooks as the source of truth, and make them idempotent.
The data model. It is the hardest decision to change later. A day spent here saves weeks after launch.
Deployment you own. Production infrastructure in your accounts, with CI/CD and error tracking, so the MVP is a product rather than a demo on someone's laptop.

How to genuinely reduce the cost

Remove features, not quality. Ask of every feature: would a first user refuse to pay without it? If not, it waits.
Do things manually behind the scenes. Onboarding, reporting and even some matching can be done by a person at first. Automate once you know the process is worth automating.
Buy the commodity parts. Authentication, billing, email delivery and file storage are solved problems. Managed services cost less than building them, especially at MVP scale.
Start with web. A responsive web app tests most B2B ideas. Add native apps when usage shows people need them — React Native lets you reuse much of the logic when that day comes.
Defer the admin panel. Your database console and a few saved queries are an acceptable admin tool for the first months.

The costs that arrive after launch

Build cost is a one-off. These are recurring, and they are the ones founders most often leave out of their plan. Prices below are as published on each provider's pricing page in October 2026 — check them yourself before budgeting, because they change.

Cost	Typical starting point	Notes
Hosting (frontend / serverless)	Vercel Pro: $20 per month per developer seat	Vercel’s free Hobby plan is for personal, non-commercial use only
Database and auth	Supabase Pro: from $25 per month	Free tier exists, but free projects pause after a week of inactivity
Payment processing	Stripe: 2.9% + 30¢ per successful US domestic card transaction	Stripe Billing (subscriptions) adds 0.7% of billing volume on pay-as-you-go
Email, monitoring, analytics	Often free at MVP volume	Becomes a real line item as you grow
AI model usage	Per token, per provider	Measure cost per user action during the build so pricing covers it
Maintenance	Ongoing engineering time	Dependency updates, security patches, bug fixes and small improvements

Plan for maintenance from day one

Software that nobody touches decays: dependencies go out of date, APIs you rely on change, app stores raise their requirements. Budget for a small amount of engineering every month after launch, even if no new features are planned.

Fixed price or time and materials?

	Fixed price	Time and materials
Best when	Scope is clear and written down	Scope is genuinely uncertain, or you expect to change direction often
Your risk	Lower on budget; changes go through a change request	Higher on budget; lower on flexibility
What makes it work	A scoping workshop and a written scope both sides sign	Short cycles, weekly demos and a clear priority list

For MVPs we prefer a fixed price for an agreed scope: it forces the scoping conversation that keeps an MVP small, and it gives a founder a number they can plan around. Work after launch usually moves to a monthly retainer.

How to read the quotes you receive

If you collect several quotes, they will often differ by a factor of three or more. That spread usually reflects different assumptions rather than different hourly rates. Before comparing totals, compare what each quote actually includes:

Check	Why it matters
Is the scope written down feature by feature?	A one-line description ("SaaS platform with subscriptions") lets each vendor imagine a different product
Are billing edge cases mentioned?	Failed payments, upgrades, downgrades and cancellations are where subscription work hides
Who owns the code, repositories and cloud accounts?	You should — from the first commit
Is deployment, monitoring and a staging environment included?	Without them you get a demo, not a product
Is testing included, and on what?	At minimum: sign-up, payment and the core workflow
What happens after launch?	Bug-fix period, maintenance options and how changes are priced

A quote far below the others is often missing several of these. A quote far above may be pricing in a larger team, more process, or a scope the vendor inflated to protect themselves. Ask each vendor what they would cut to halve the price — the answer tells you how well they understood your product.

Red flag

Any estimate given before anyone has asked about your user roles, how you will charge, or which systems the product must connect to. Those three questions decide most of the cost.

Getting an estimate you can trust

A credible estimate needs more than a feature list. Bring the problem you are solving, who the first users are, how you plan to charge, which systems it must connect to, and your deadline. Be wary of any quote produced without questions about roles, billing or data — those are where the real cost hides.

If you would like us to do this with you, our MVP development service starts with a scoping workshop and ends with a fixed estimate. For products that are past the MVP stage, see SaaS development.

FAQ

How much does a SaaS MVP cost?

Our SaaS MVP package starts at $38,500. Narrower validation builds can cost less; products with mobile apps, payments and several integrations cost more. We give a fixed estimate after a scoping workshop.

How long does it take to build a SaaS MVP?

Typically 6 to 12 weeks from signed scope to launch for a focused MVP. Adding native mobile apps, complex billing or heavy AI features extends that.

Can I build an MVP with no-code tools instead?

For testing demand, sometimes yes. The trade-off appears later: when the product needs complex billing, permissions or integrations, teams often have to migrate — and a production data migration is real work. We have done one off a no-code platform, and we price it as its own line item.

What ongoing costs should I budget after launch?

Hosting, database, payment processing fees, email and monitoring services, AI usage if relevant, and a small monthly amount of engineering time for maintenance and security updates.

Sources

Originally published on the DablewDevs blog. DablewDevs builds production AI systems and software — AI agents, RAG, MCP servers and SaaS products. See what we build and what it costs.

How to Build a Production MCP Server for Your SaaS

Waleed Siddiqui — Fri, 30 Oct 2026 13:00:00 +0000

Most MCP tutorials end with a local script that exposes two tools and reads an API key from an environment variable. That is a fine way to learn the protocol. It is not something you can give to customers, or to forty staff who each have different permissions.

We recently built a remote MCP server for a client's operations platform — a leadership-development consultancy that runs invoicing, scheduling, contracts, HR and time tracking on a custom admin app we had already built for them. Staff now connect Claude to it in one click and run routine work by asking. The server exposes roughly 115 tools. This article covers the decisions that mattered, in the order you will hit them.

Remote or local: decide before you write a tool

MCP defines two standard transports. stdio runs the server as a local process the client launches; Streamable HTTP runs it as a web endpoint the client connects to by URL. The protocol messages are the same; almost everything around them is different.

	Local (stdio)	Remote (Streamable HTTP)
Setup for users	Install a package, edit a config file	Paste a URL, sign in
Credentials	Read from the environment (often a shared API key)	OAuth — each connection acts as one user
Who it suits	Developers, single-user tools	Customers, non-technical staff, teams
Updates	Every user upgrades their install	You deploy once
Auth spec applies?	No — the spec says stdio servers should take credentials from the environment	Yes — the MCP authorization spec is written for HTTP transports

Our first version for the client was a local stdio server. It worked, and nobody outside engineering used it. Moving to a hosted endpoint — no install, no config file, sign in with the account you already have — is what turned it from a demo into something staff use. If your users are not developers, start remote.

Authentication: the server acts as a person, not as "the AI"

For HTTP transports, the MCP authorization spec builds on OAuth 2.1. Your MCP server plays the role of an OAuth resource server: it accepts access tokens and serves requests. A separate (or co-hosted) authorization server signs the user in and issues those tokens. The pieces you need to understand:

Protected Resource Metadata (RFC 9728). The MCP server must publish metadata telling clients which authorization server to use. An unauthenticated request gets a 401 whose WWW-Authenticate header points at that metadata, and the client takes it from there.
Client registration. Clients need a client ID before starting the flow. The current spec prefers Client ID Metadata Documents, allows pre-registration, and keeps Dynamic Client Registration (RFC 7591) for backwards compatibility. Check which mechanisms the clients your users run actually support.
PKCE and the resource parameter. Clients use PKCE and must send a resource parameter naming your server, so the token is issued for your server specifically.
Audience validation. Your server must reject tokens that were not issued for it. Do not accept, and never pass through, tokens meant for some other API.
Scopes. A 403 with error="insufficient_scope" and the required scopes lets clients step up permissions only when needed, instead of asking for everything up front.

If you already have an identity provider, you usually do not need to build an authorization server from scratch — but you do need to check it supports what MCP clients expect. In our build, sign-in reuses the platform's existing accounts, so revoking someone's access to the app also revokes their assistant.

The rule that matters most

Every tool call runs with the identity and permissions of the person who connected the client. There is no service account with god-mode access sitting behind the assistant. When something goes wrong, the audit log names a human.

Make tools thin proxies over the API you already trust

The fastest way to create a security hole is to give the MCP server its own database access and re-implement business rules inside tool handlers. Now you have two copies of "who can send an invoice" and they will drift.

Every one of our ~115 tools calls an existing, authenticated endpoint of the platform — the same validation the web app relies on. The MCP layer translates between "what the model asked for" and "what the API accepts", and nothing else. Two consequences:

The model cannot do anything the API would reject for that user. Prompt injection can at worst trigger an action the user was already allowed to take — which is why confirmations still matter (below).
The server stays small. Adding a tool is mostly writing a description and an input schema, then mapping to an endpoint.

If your product has no clean API yet, build that first. It is the part of the work that pays off regardless of MCP — see our API integration services.

What registering a tool looks like

Here is a minimal, generic example using the official TypeScript SDK (the v2 @modelcontextprotocol/server package). It builds a server per request for the signed-in user, registers one read-only tool, and serves it over Streamable HTTP from a web-standard route handler such as a Next.js App Router route. Check the SDK documentation for your installed version — the SDK moved from @modelcontextprotocol/sdk (v1) to separate server and client packages in v2.

import { createMcpHandler, McpServer } from '@modelcontextprotocol/server';
import * as z from 'zod/v4';
import { verifyAccessToken } from '@/lib/auth';   // your token + audience check
import { api } from '@/lib/api';                  // your existing, validated API client

function buildServer(user: { id: string; roles: string[] }) {
  const server = new McpServer({ name: 'acme-operations', version: '1.0.0' });

  server.registerTool(
    'list_overdue_invoices',
    {
      description:
        'List invoices in the user\'s workspace that are past their due date. Read-only. ' +
        'Use before drafting reminder emails.',
      inputSchema: z.object({
        olderThanDays: z.number().int().min(1).max(365).default(30),
      }),
    },
    async ({ olderThanDays }) => {
      // Same endpoint the web app uses, called as this user.
      const invoices = await api.invoices.listOverdue({ olderThanDays }, { actingAs: user });
      return { content: [{ type: 'text', text: JSON.stringify(invoices) }] };
    },
  );

  return server;
}

export async function POST(req: Request) {
  const user = await verifyAccessToken(req);   // 401 with WWW-Authenticate if missing/invalid
  if (user instanceof Response) return user;
  return createMcpHandler(() => buildServer(user)).fetch(req);
}

Building the server per request keeps it stateless and makes per-user tool sets trivial: a read-only user simply never gets the write tools registered.

Tool design is prompt design

With a handful of tools, models pick the right one almost every time. With a hundred, tool descriptions and schemas become the main thing standing between "it works" and "it picks the wrong tool and apologises". What helped us most:

Design around jobs, not endpoints. Users ask "chase the overdue invoices", not "GET /invoices?status=overdue". Name and describe tools in the language of the task. Sometimes that means one tool combining two endpoints, or splitting one endpoint into two clearer tools.
Consistent verb_noun names. list_, get_, create_, update_, send_ across every domain. The model learns the pattern; so do your engineers.
Say what the tool is not for. "Read-only", "does not send the email — use send_email after the user confirms" and similar lines prevent a surprising number of mistakes.
Tight input schemas. Enums instead of free strings, bounded numbers, explicit date formats. Every constraint the schema enforces is one the model cannot get wrong.
Compact outputs. Return the fields the model needs to answer and act, not the full database row. Large outputs cost tokens and bury the useful part.

Confirmations, audit logs and limits

Permissions decide what an assistant may do. They do not decide what it should do on a particular request. For anything irreversible or external — sending an email, issuing an invoice, deleting a record — we draft first and ask the user to confirm in the conversation before the write tool runs.

Audit every call with user, tool, arguments (minus secrets) and result status. You will need it the first time someone asks "did the assistant send that?"
Rate-limit per user, separately from your normal API limits — agents can loop.
Treat tool results as untrusted input to the model. Content from emails, documents or customer notes can contain instructions. Keep destructive actions behind confirmation precisely because of this.

Test with the clients your users actually run

Unit tests on tool handlers are necessary and nowhere near sufficient. The failure modes that matter show up only when a real model chooses among your real tools. We kept a list of realistic tasks ("reschedule Thursday's session and tell the participant", "which invoices from last month are unpaid?") and ran them through Claude after every batch of tool changes, watching which tools it chose and where it hesitated. Most fixes were edits to descriptions, not code.

Also test the sign-in flow end to end in each client you support. Client support for the authorization features differs, and it moves quickly.

Deployment: put it next to the API

We hosted the MCP server inside the existing Next.js application as one more route, so it ships with the product, uses the same configuration and secrets management, and cannot fall out of step with the API it proxies. A separate service is reasonable when your backend is not HTTP-friendly or you need independent scaling — but co-location removes a whole class of version-skew bugs.

Plan for maintenance. The MCP specification and SDKs are still evolving, and clients adopt new versions at different speeds. Budget a little time each quarter to track spec changes and re-test with your supported clients.

A checklist before you launch

Area	Ready when…
Transport	Remote endpoint over Streamable HTTP; users add it by URL
Auth	Protected Resource Metadata published; tokens audience-checked; revocation works
Permissions	Each call runs as the connected user; tool set follows their role
Tools	Thin proxies over validated endpoints; job-oriented names; tight schemas
Safety	Destructive actions need confirmation; per-user rate limits; full audit log
Testing	Realistic task list run in every supported client after each change
Operations	Monitoring and error alerts; owner for spec and SDK upgrades

Read the full case study, or see how we approach MCP server development for SaaS products and internal platforms.

FAQ

How long does it take to build an MCP server?

A read-only server over a clean existing API, with OAuth and a handful of tools, is a matter of weeks. Time grows with the number of tools, write actions that need confirmation flows, and how much API groundwork is missing. Testing tool selection with real clients is a real part of the schedule, not an afterthought.

Do we need our own OAuth authorization server?

Not necessarily. MCP servers act as OAuth resource servers; the authorization server can be your existing identity provider if it supports what MCP clients need (discovery metadata, PKCE, a client registration mechanism the clients support). If it does not, you can host a small authorization server alongside the MCP server.

Should every API endpoint become a tool?

No. Map the jobs users want to delegate and design tools around those. Large, endpoint-shaped tool sets make models worse at choosing the right tool and are harder to secure.

Is an MCP server safe to expose to customers?

It can be, if every call runs as the authenticated user, tools go through your validated API, destructive actions require confirmation, and calls are rate-limited and logged. The risk is the same as any API that acts on a user's behalf — plus prompt injection from untrusted content, which is why confirmations matter.

Sources

Originally published on the DablewDevs blog. DablewDevs builds production AI systems and software — AI agents, RAG, MCP servers and SaaS products. See what we build and what it costs.

How to Add AI Features to an Existing SaaS (No Rewrite)

Waleed Siddiqui — Tue, 27 Oct 2026 13:00:00 +0000

Your customers are asking for AI features, and your codebase was not designed with them in mind. The good news: you almost never need a rewrite or a separate "AI platform". An LLM feature is a feature like any other — it lives next to the code it serves, goes through your review process and ships behind a flag.

What makes AI features different is that their output is probabilistic, their cost scales with usage, and they send data to a third party. This is the playbook we use to handle those three facts when we add AI to existing products as part of our AI integration work.

Step 1: pick the first feature with two questions

Score candidate features on two axes: how much value a correct output creates, and whether the data the model needs is already in your system and accessible. The first feature should score high on both.

Feature type	Typical value	Data usually available?	Good first feature?
Extracting fields from documents or emails into records	High — removes manual data entry	Yes, it is the input	Often the best first choice
Summaries and generated reports from existing data	High for busy users	Yes	Yes, if users can see and correct the result
Classification and routing (tickets, leads, transactions)	Medium–high	Yes, plus historical labels	Yes — and easy to measure
Semantic search over the product’s content	Medium–high	Yes, needs indexing	Yes, but plan for retrieval work
An assistant that takes actions in the app	High	Yes, via your API	Better as a second feature, once the scaffolding exists
A general chatbot on the marketing site	Low–medium	Partly	Rarely the best place to start

"Users can see and correct the result" matters more than it looks. A feature where a person reviews the AI output before it is used is far more forgiving than one that acts on its own — and it generates the labelled examples you will use to evaluate and improve it.

Step 2: four architecture defaults

1. A provider-agnostic model layer

Put model calls behind one small interface in your codebase, with the model chosen by configuration. Providers, prices and model quality change every few months; switching should be a config change plus an evaluation run, not a refactor. SDKs such as the Vercel AI SDK or LangChain already give you this abstraction across Anthropic, OpenAI and Google models.

2. Structured output, validated against a schema

If software consumes the result, never parse free text. Ask for output that matches a schema and validate it. In the Vercel AI SDK this is generateText with an Output.object schema; the parsed, validated value comes back as output, and a response that cannot be parsed into the schema throws a NoObjectGeneratedError you can handle explicitly:

import { generateText, NoObjectGeneratedError, Output } from 'ai';
import { z } from 'zod';

const Invoice = z.object({
  vendor: z.string(),
  invoiceNumber: z.string(),
  dueDate: z.string().describe('ISO 8601 date'),
  total: z.number(),
  currency: z.string().length(3),
});

export async function extractInvoice(text: string, model: string) {
  try {
    const { output } = await generateText({
      model, // e.g. from config, so the provider is swappable
      output: Output.object({ schema: Invoice }),
      prompt: 'Extract the invoice fields from this document:\n\n' + text,
    });
    return { ok: true as const, invoice: output };
  } catch (error) {
    if (NoObjectGeneratedError.isInstance(error)) {
      // Route to a human review queue instead of saving bad data.
      return { ok: false as const, reason: 'unparseable', raw: error.text };
    }
    throw error;
  }
}

We used the same principle in an AI assessment app we built: assessment templates are generated as schema-constrained JSON, so the app never has to guess whether the model "probably" returned valid structure.

3. Streaming where users wait, background jobs everywhere else

Interactive features (an assistant, a draft the user watches being written) should stream. Everything else — extraction, classification, report generation — belongs in a background job with retries, so model latency and provider outages never block a web request. Your existing queue is usually fine.

4. Ship behind a feature flag

Roll out to internal users, then a few friendly customers, then everyone. Capture a thumbs-up/down or an edit signal on every output from day one; it is the cheapest evaluation data you will ever get.

Step 3: get multi-tenancy right before launch

Most AI tutorials assume one user and one dataset. A SaaS has many tenants, and the AI layer must respect the same boundaries as the rest of the product:

Isolation in retrieval. If the feature searches customer content, filter by tenant inside the search query itself — never retrieve broadly and filter afterwards. Storing embeddings in PostgreSQL with pgvector keeps vectors next to your existing tenant and permission columns.
No cross-tenant prompts. Prompt templates, few-shot examples and caches must never include another customer's data.
Per-tenant usage metering. Record tokens, model and feature for every call, tagged with tenant and user. You need it for pricing, for abuse limits and to answer "why did our bill double?".
Per-tenant limits and kill switches. One customer's bulk import should not exhaust your rate limits or budget for everyone else.

Price the feature with real numbers

Measure cost per user action during the pilot. Only then decide whether the feature is included in existing plans, metered, or sold as an add-on — guessing here is how AI features quietly erode margins.

Step 4: know exactly what data goes where

Your customers' security questionnaires will ask which data reaches which AI provider. Answer it on purpose:

Send the minimum. Redact or drop fields the model does not need. It improves accuracy and cost as well as privacy.
Use paid API tiers and read their data terms. As of October 2026: OpenAI states that data sent to its API is not used to train its models unless you opt in, and that abuse-monitoring logs are kept for up to 30 days, with zero data retention available to approved customers. Anthropic states it does not use inputs or outputs from its commercial products for training by default. Google's Gemini API terms say prompts and responses from paid services are not used to improve its products, while unpaid-tier data may be used and reviewed by people.
Document it. Keep a short data-flow note per AI feature: fields sent, provider, retention, region. Update your subprocessor list and privacy policy.

Step 5: put evaluation in CI

Every prompt edit or model upgrade can make the feature better on the example you were looking at and worse on ten you weren't. Treat AI output like any other behaviour under test:

Collect 30–100 real inputs with the output a domain expert considers correct.
Score automatically: exact checks for structured fields, and an LLM-as-judge rubric for free text — but only after checking the judge's verdicts against human labels, so you know when it can be trusted.
Run the suite in CI on every change to prompts, models or retrieval, and block merges that regress.
Add every real-world failure your users report to the suite.

Mistakes we see when teams add AI themselves

Starting with the most impressive feature instead of the most valuable one. A general assistant demos well; extraction or classification usually pays back first.
Putting raw tables in the prompt. Models do better with less, well-labelled context — the relevant records, not the whole table. It is cheaper too.
Trusting a single happy-path test. A prompt that works on the example you wrote it against tells you very little. An evaluation set of real inputs tells you a lot.
Hard-coding one provider everywhere. When prices drop or a better model ships, you want to switch in an afternoon.
Forgetting the failure path. Timeouts, rate limits and refusals happen in production. Decide what the user sees when they do, and make sure the rest of the product keeps working.
No owner after launch. Model behaviour drifts as providers update models and your data changes. Someone needs to watch the feedback signal and the evaluation results.

A rollout checklist

Before launch	Done when
Model behind an interface	Changing provider is a config change
Schema-validated outputs	Invalid output is handled, never stored
Evaluation suite in CI	Merges that regress are blocked
Tenant isolation	Retrieval and prompts are tenant-scoped and tested
Usage metering and limits	Cost per tenant and per feature is visible
Fallback behaviour	The product still works when the provider is down
Data-flow documentation	Security questionnaires can be answered from it
Feature flag and feedback capture	You can roll back instantly and see user corrections

None of this requires a new platform or a rewrite. It does require engineers who treat AI as production software. That is what we do in our AI integration service — inside your codebase, following your conventions — and if you would rather have an engineer join your team directly, you can hire AI developers from us. For the SaaS foundations around it, see SaaS development.

FAQ

Do we need to rebuild our product to add AI?

Almost never. AI features can be added inside your existing codebase and infrastructure, behind a feature flag, using the same review and deployment process as any other feature.

Which AI feature should we build first?

One where a correct output creates clear value and the data the model needs is already in your system — structured extraction, summaries, classification or semantic search are common first choices. Features where users review the output before it is used are the safest start.

Will our customers’ data be used to train AI models?

Not on the paid API tiers of the major providers by default, according to their published terms as of October 2026 — but terms differ (for example, Google treats unpaid Gemini API usage differently). We document exactly which data reaches which provider for each feature.

How do we control AI costs?

Meter tokens per tenant and per feature, choose the smallest model that passes your evaluation set, cache where possible, set per-tenant limits, and price the feature from measured cost per action.

Sources

Originally published on the DablewDevs blog. DablewDevs builds production AI systems and software — AI agents, RAG, MCP servers and SaaS products. See what we build and what it costs.

How Much Does It Cost to Build an AI Agent?

Waleed Siddiqui — Sat, 24 Oct 2026 13:00:00 +0000

Ask ten agencies what an AI agent costs and you will get ten numbers, most of them meaningless without the scope behind them. "An agent" can mean a tool that drafts replies for a person to send, or a system that issues invoices and moves records between three products without anyone watching.

So instead of a price list, this article explains what actually drives the two bills every agent has — the cost to build it and the cost to run it — with worked running-cost arithmetic at current API list prices. It is based on the agent and AI-integration work we do for clients and the production agent systems our founder builds, and it deliberately avoids quoting numbers we cannot back up.

Two bills: building the agent and running it

Build cost is engineering time: scoping, connecting the agent to your systems, making it safe, testing it, and rolling it out. It is mostly a one-off, front-loaded cost.

Running cost is model usage (tokens), plus the infrastructure, monitoring and upkeep that keep the agent accurate as your data, workflows and models change. It scales with usage and never quite goes to zero.

Teams routinely over-focus on token prices, which for most business agents are the smaller of the two. Model calls for an internal workflow agent often cost less per month than a single day of engineering. The expensive part is making the agent trustworthy enough to leave running.

What drives the build cost

Driver	Cheaper end	Expensive end	Why it matters
Systems the agent touches	One system with a clean API	Several systems, some without APIs	Every integration needs tools, auth, error handling and tests; missing APIs must be built first
Reversibility of actions	Read-only or drafts for a person	Sends, pays, deletes, files externally	Irreversible actions need approval steps, idempotency and audit trails
Data quality	Structured records, consistent formats	Scanned PDFs, free text, conflicting sources	Messy inputs need extraction, validation and fallbacks before the agent can reason over them
Accuracy bar	"Good enough to draft"	"Right every time it acts alone"	A higher bar needs a bigger evaluation set and more iterations to meet it
Volume and latency	Batch jobs, minutes are fine	User-facing, seconds matter	Low latency pushes you to smaller models, caching, streaming and parallel tool calls
Security and compliance	Internal data, no regulation	Personal, financial or health data	Data handling, retention, provider agreements and access controls take real design time

When we scope an agent, these six questions decide most of the estimate. A read-only internal assistant over one well-structured system sits at one end; an autonomous agent that takes irreversible actions across several systems with regulated data sits at the other. Both are "AI agents".

Where the build effort goes, phase by phase

These phases hold for almost every agent we have built or reviewed. Durations depend heavily on the drivers above, so treat the ranges as orders of magnitude, not quotes.

Scoping and the evaluation set (typically about a week or two). Pick one workflow, write down what "done correctly" means, and collect real examples — often 30 to 100. This is the cheapest phase and the one that most often saves money, because it can show the idea will not work before anything is built.
Prototype against real data (often a few weeks). A thin agent on your actual systems in a sandbox, scored against the evaluation set. You learn whether the model, the data or the tools are the bottleneck.
Production hardening (frequently the largest phase). Permissions inherited from real users, approval steps, retries, idempotent actions, durable state for long-running work, tracing, cost tracking and an interface people actually use.
Gradual rollout (weeks, overlapping with operation). Shadow mode first (the agent proposes, a person acts), then supervised autonomy, then autonomy only on the cases the evaluation shows it handles reliably.

The common budgeting mistake

Budgeting for the prototype and calling it the project. A prototype that works on good examples is often well under half of the work; hardening and rollout are where agents earn the trust to run unattended. Our AI proof of concept to production article covers that gap in detail.

Running cost: worked token arithmetic at current prices

Model usage is billed per token, separately for input (what you send: instructions, tool definitions, retrieved context, conversation so far) and output (what the model writes). Agents are input-heavy, because every step re-sends the growing context.

Take a modest workflow agent: 8 steps per run, adding up to 40,000 input tokens and 3,000 output tokens per run in total. Here is the cost per run and per 1,000 runs at list prices published on the providers' pricing pages on 20 October 2026 (standard tier, short prompts, before any caching or batch discounts):

Model (list price per 1M tokens, in / out)	Input cost per run	Output cost per run	Per run	Per 1,000 runs
Claude Sonnet 5.5 ($2 / $10)	40,000 × $2 ÷ 1M = $0.080	3,000 × $10 ÷ 1M = $0.030	$0.110	$110
OpenAI gpt-6.1-sol ($2 / $10)	$0.080	$0.030	$0.110	$110
Gemini 3.6 Flash ($0.75 / $3.75, through 31 Dec 2026)	40,000 × $0.75 ÷ 1M = $0.030	3,000 × $3.75 ÷ 1M = $0.011	$0.041	$41.25
Claude Haiku 5.5 ($0.10 / $0.50, prompts ≤ 100K)	40,000 × $0.10 ÷ 1M = $0.004	3,000 × $0.50 ÷ 1M = $0.0015	$0.0055	$5.50

At 1,000 runs a month, the token bill ranges from a few dollars to a little over a hundred depending on the model. Two things move it far more than the per-token price:

Prompt caching. Much of an agent's input — system instructions and tool definitions — repeats on every step. Providers bill cached input at a fraction of the normal rate (for example $0.10 instead of $2 per million on Claude Sonnet 5.5 and gpt-6.1-sol). If 30,000 of the 40,000 input tokens were cache reads, the Sonnet run above would fall from about $0.11 to about $0.053 — roughly half. Cache writes can be billed differently; check the current pricing page for your provider.
Choosing the smallest model that passes your evaluation set. Routing easy steps (classification, extraction) to a small model and reserving a larger one for genuine reasoning often cuts cost by an order of magnitude without hurting accuracy. The evaluation set is what makes that decision safe rather than a guess.

Prices change often and differ by region, tier and context length. Re-run this arithmetic with current prices and your own token counts from a prototype — we measure tokens per run during the prototype phase so the operating cost is known before rollout.

The running costs nobody budgets for

Evaluation upkeep. New failure cases get added to the test set and every prompt, tool or model change is re-scored. This is cheap per run and essential; skipping it is how agents quietly get worse.
Monitoring and tracing. Storing traces of every step and tool call so you can answer "why did it do that?" Tools such as Langfuse or an OpenTelemetry pipeline cost money or hosting time, and someone has to look at them.
Model upgrades and deprecations. Providers retire models. Each migration means re-running the evaluation set, adjusting prompts and sometimes tools. Keeping the model behind an interface makes this a routine task instead of a rewrite.
Integration drift. The APIs your agent calls change too. An agent is a piece of software with dependencies, not a configuration you set once.
Human review time. In supervised mode, people review the agent's proposals. That time is a real cost — and the main thing a good agent reduces over time.

When you should not build a custom agent

Custom agents are worth it when the workflow is specific to your business, touches systems you control, and runs often enough to matter. They are poor value when:

An off-the-shelf product already does the job inside a tool you use (many help desks, CRMs and document tools now ship decent built-in assistants).
The process runs a few times a month — a checklist and a person will be cheaper for years.
Nobody can define what a correct result looks like. If you cannot build an evaluation set, you cannot know whether the agent works.
The real problem is missing data or a missing API. Fix that first; it pays off with or without AI.

We say this in scoping calls too: sometimes the right recommendation is a small integration or a better form, not an agent.

How to get an estimate you can trust

Any agency that quotes a firm price for "an AI agent" before seeing your workflow, systems and data is guessing. A useful estimate comes from a short scoping exercise that produces:

A written description of one workflow and what "correct" means for it.
An evaluation set of real cases, even a small one.
A list of systems the agent must read from and write to, with the state of their APIs.
A decision on which actions need human approval.
Measured tokens per run from a thin prototype, turned into a monthly running-cost figure.

That is exactly how we start AI agent development projects. If you already have an agent prototype and want it production-ready, see AI integration services — or hire AI developers to work inside your own team.

FAQ

How much does it cost to run an AI agent per month?

For the example in this article (40,000 input and 3,000 output tokens per run), 1,000 runs cost from about $5.50 to $110 in model fees at October 2026 list prices, depending on the model — less with prompt caching. Add hosting, tracing and evaluation upkeep, which usually matter more than tokens at modest volumes.

Why do AI agent quotes vary so much?

Because "AI agent" covers everything from a read-only drafting assistant to an autonomous system that takes irreversible actions across several products. The number of systems touched, whether actions are reversible, data quality, the accuracy bar, latency and compliance requirements drive most of the difference.

Is it cheaper to use a bigger model and skip engineering?

Rarely. A bigger model can paper over a weak prototype, but it does not add permissions, approvals, idempotency, monitoring or an evaluation set — the things that make an agent safe to leave running. It usually also costs more per run.

Can we start small?

Yes, and we recommend it: one workflow, a real evaluation set, shadow mode first. You learn the true build and running costs on a contained scope before committing to more.

Sources

Originally published on the DablewDevs blog. DablewDevs builds production AI systems and software — AI agents, RAG, MCP servers and SaaS products. See what we build and what it costs.

RAG vs Fine-Tuning: A Decision Framework for Business AI

Waleed Siddiqui — Wed, 21 Oct 2026 09:59:28 +0000

"Should we fine-tune a model on our documents?" is one of the most common questions we hear from teams starting an AI project. Usually the honest answer is no — not because fine-tuning is bad, but because it solves a different problem from the one they have.

This article is the decision framework we use with clients. It is written from production experience: our founder built and runs a multi-tenant retrieval-augmented generation (RAG) pipeline on an AI EdTech platform used across 20+ US schools, with hybrid semantic and keyword search, per-school data isolation on pgvector, and vision-model ingestion of PDFs. The lessons below come from that system and from the client work we do at DablewDevs.

The short answer

RAG changes what the model knows at the moment it answers. Fine-tuning changes how the model behaves. If your problem is "the model doesn't know our policies, products, contracts or customers", you need retrieval. If your problem is "the model knows enough, but answers in the wrong format, tone or structure, or a cheaper model can't follow the task", fine-tuning is a candidate.

Most business knowledge projects are the first kind. Your documents change, different users may see different documents, and people need to check where an answer came from. Retrieval handles all three; fine-tuning handles none of them.

Before either: check whether you need retrieval at all

Anthropic notes that if a knowledge base is under roughly 200,000 tokens (about 500 pages), you can often put the whole thing in the prompt and use prompt caching to keep cost and latency down. For a small, stable document set, that is simpler than building a retrieval pipeline.

Decision table: RAG vs fine-tuning

We score a use case against these questions. If most answers land in the RAG column, start there.

Question	Points to RAG	Points to fine-tuning
How often does the source information change?	Weekly, daily or continuously — re-index and the answer changes immediately	Rarely; the behaviour you want is stable
Do users need to see where an answer came from?	Yes — citations to the exact passage	No, or the output is not a factual answer
Do different users see different data?	Yes — permissions are enforced at retrieval time	No — everything the model learns, every user can extract
What is wrong with today’s output?	Missing or wrong facts	Right facts, wrong format, tone, structure or classification
How much labelled data do you have?	Documents are enough to start	Hundreds to thousands of high-quality input/output examples
Can you delete information on request?	Yes — remove it from the index	Hard — it may be in the weights until you retrain
What are you optimising for at scale?	Answer accuracy and freshness	Shorter prompts, lower per-call cost, or a smaller model doing a narrow task

The last row is where fine-tuning genuinely shines. OpenAI's own optimisation guide frames fine-tuning as a way to supply more examples than fit in a prompt, use shorter prompts at scale, and train a smaller, cheaper, faster model to excel at a particular task. Those are behaviour and cost goals — not "teach the model our handbook".

The misconception: fine-tuning to teach facts

Fine-tuning on your documents feels like it should produce a model that "knows" them. In practice it is a poor way to load facts:

No citations. The model cannot point to the passage it learned something from, so reviewers have nothing to check against.
Stale by default. Every document change means another training run, evaluation and deployment.
No access control. Anything in the training data can potentially surface for any user who asks the right question. In a multi-tenant product, that alone rules it out for customer data.
Confident blending. A model that has partially absorbed two versions of a policy will answer fluently from both. Retrieval at least lets you show which version was used.

There is also a practical signal in the market: OpenAI's model optimisation guide states that it is winding down its fine-tuning platform and that it is no longer accessible to new users. Fine-tuning remains available elsewhere, but it is not where the major providers are steering knowledge use cases.

When to combine them

The two approaches are not exclusive. The pattern we recommend when both are justified is retrieve the facts, tune the behaviour:

A support assistant retrieves the relevant help-centre articles (RAG) and uses a tuned or heavily prompted model to answer in your house style and escalation format.
A document-processing pipeline retrieves the relevant contract clauses, then a small tuned classifier labels them against your risk taxonomy at a fraction of a large model's cost.
A high-volume workflow starts on a large model with retrieval; once you have thousands of reviewed outputs, those become training data for a smaller model doing the same narrow task.

Notice the order. You almost always build retrieval first, because it produces the reviewed examples you would later need for fine-tuning anyway.

Where RAG actually fails (it is rarely the model)

Choosing RAG is the easy part. When a RAG system gives a wrong answer, the cause is usually that the right passage was never retrieved. These are the failure modes we see most:

1. Ingestion destroys structure

PDFs parsed into a flat stream of text lose their headings, tables lose their column headers, and scanned pages come out empty. On the EdTech platform our founder works on, plain text extraction was not good enough for long course documents, so ingestion uses a vision model to read pages, anchored to the document's table of contents, with deterministic settings and content-addressed deduplication so re-ingesting the same file does not create duplicates.

2. Chunks lose their context

A chunk that says "the fee is waived after 30 days" is useless if it no longer says which product or plan it belongs to. Hierarchical chunking (keeping the section path with each chunk) helps. Anthropic's contextual retrieval technique goes further, prepending a short, model-generated description of where each chunk sits in its document before embedding it.

3. Pure vector search misses exact terms

Embeddings are good at meaning and bad at identifiers: part numbers, error codes, course codes, names. Keyword search (BM25 or PostgreSQL full-text) is the opposite. We run both and merge the results with reciprocal rank fusion (RRF), which scores each document by summing 1 / (k + rank) across the result lists. It needs no training and, in Cormack, Clarke and Büttcher's original SIGIR 2009 paper, outperformed both individual rankers and the Condorcet method. In Anthropic's published experiment, adding contextual BM25 to contextual embeddings cut the top-20 retrieval failure rate by 49%, and adding reranking took the reduction to 67%.

4. Filters applied too late

If you retrieve the top 20 chunks and then remove the ones the user may not see, you can end up with three results, all irrelevant. Tenant and permission filters belong inside the retrieval query. pgvector makes this natural: vectors live in the same PostgreSQL tables as your permission data, so a WHERE clause enforces isolation in the same query that ranks similarity — which is how per-school isolation works in the system described above.

-- Hybrid retrieval, tenant-filtered inside the query (simplified).
WITH semantic AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks
  WHERE tenant_id = $2
  ORDER BY embedding <=> $1
  LIMIT 50
),
keyword AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank
  FROM chunks, plainto_tsquery('english', $3) AS query
  WHERE tenant_id = $2 AND tsv @@ query
  ORDER BY ts_rank_cd(tsv, query) DESC
  LIMIT 50
)
SELECT id, SUM(1.0 / (60 + rank)) AS rrf_score   -- k = 60, as in the RRF paper
FROM (SELECT * FROM semantic UNION ALL SELECT * FROM keyword) AS ranked
GROUP BY id
ORDER BY rrf_score DESC
LIMIT 20;

How to measure which approach is working

Whatever you choose, decide how you will know it is right before you build. We measure two things separately, because they fail for different reasons:

Metric	Question it answers	How we measure it
Retrieval recall	Did we fetch the passages that contain the answer?	For each test question, list the source passages a human expects; check whether they appear in the top-k results
Answer faithfulness	Did the answer stay true to what was retrieved?	A rubric scored by a person, then by an LLM judge whose verdicts are checked against human labels before we trust it
Refusal correctness	Does it say "I don’t know" when nothing relevant was found?	Include out-of-scope questions in the test set

Start with 50 to 200 real questions from real users, with the documents that should answer them. If retrieval recall is low, no amount of prompt tuning or fine-tuning will fix the answers — improve parsing, chunking or search first. If recall is high but faithfulness is low, then prompt and model changes are worth exploring.

A five-step way to decide

Write down the failure. Missing facts, wrong permissions, or wrong behaviour? Missing facts point to retrieval; wrong behaviour may point to tuning.
Measure the size of the knowledge. Under a few hundred pages and stable? Try long-context prompting with caching first.
Build retrieval with an evaluation set. Hybrid search, metadata filters, citations, and the metrics above.
Fix the weakest stage. Ingestion, chunking, search or prompt — whichever loses the most answers.
Consider fine-tuning only for behaviour or cost, once you have enough reviewed examples to train on and a measured reason to expect improvement.

If you already have a RAG prototype whose answers are not good enough to ship, the same framework applies — we start by building the evaluation set and finding where answers are lost. See our RAG development service, or, if the knowledge needs to live inside an existing product, our AI integration work.

FAQ

Is RAG cheaper than fine-tuning?

Usually to start, yes: there is no training run, and updating knowledge means re-indexing documents rather than retraining. At very high volume, a fine-tuned smaller model can be cheaper per call for a narrow task — which is why we sometimes combine the two once a system is proven.

Can fine-tuning reduce hallucinations?

It can make a model follow your format and refuse out-of-scope requests more consistently, but it does not give the model reliable access to facts. For factual answers, grounding the model in retrieved sources and asking it to cite them is the more dependable approach.

Do we need a dedicated vector database for RAG?

Not necessarily. If you already run PostgreSQL, pgvector supports HNSW and IVFFlat indexes and works alongside PostgreSQL full-text search for hybrid retrieval, with permissions enforced in the same query. A separate vector database is worth its operational cost only when scale or specific features demand it.

How long does it take to build a production RAG system?

A focused system over one well-structured document set can be built in a few weeks. Messy documents, many sources, complex permissions and high accuracy requirements extend that. We size it after reviewing a sample of the content.

Sources

Originally published on the DablewDevs blog. DablewDevs builds production AI systems and software — AI agents, RAG, MCP servers and SaaS products. See what we build and what it costs.

AI Proof of Concept to Production: What Demos Skip

Waleed Siddiqui — Wed, 21 Oct 2026 09:57:32 +0000

A good AI demo takes an afternoon. A model reads a document, produces a summary, and everyone in the room is impressed. Then the project stalls for months, because the demo worked on five hand-picked examples and nobody knows what it does on the five-thousandth real one.

The gap between proof of concept and production is not mainly about choosing a better model. It is ordinary engineering that demos skip: knowing whether the output is right, making it fit your software, controlling what it can do, knowing what it costs, and rolling it out so a bad day does not reach every customer. This is the checklist we work through when taking AI features to production.

Demo vs production at a glance

Concern	In the demo	In production
Is it right?	Looks good on a few examples	Scored against an evaluation set of real cases on every change
Output format	Free text, read by a person	Schema-constrained output, validated before it touches your data
What can it do?	Whatever the API key allows	Acts as a specific user, with that user's permissions
Repeated actions	Not considered	Idempotent writes — retries cannot double-send or double-charge
Failures	Refresh and try again	Timeouts, retries, rate-limit handling and a non-AI fallback
Cost	Unknown	Tokens and cost tracked per feature and per customer
Visibility	Console logs	Traces of inputs, outputs, tool calls and latency
Rollout	Everyone at once	Shadow mode, feature flag, gradual exposure

1. Build an evaluation set before you tune anything

The single biggest difference between teams that ship AI features and teams that stall is an evaluation set: a collection of real inputs, with what a correct output looks like, that you can score automatically. Without one, every prompt change is a guess and every model upgrade is a gamble.

Use real data, not invented examples. Pull cases from production logs, support tickets or documents — including the messy ones.
Keep a held-out set. Tune prompts on one split and report results on another, so you are not grading your own homework.
Score what matters. For classification, accuracy per class. For extraction, field-level correctness. For generated text, a rubric of specific yes/no questions ("does the reply invent facts not in the source?") rather than a vague 1–10 score.
Add every production failure to the set. The evaluation set should grow with the feature.

2. If an LLM grades the output, grade the grader

For open-ended output, a second model scoring the first against a rubric ("LLM-as-judge") is often the only scalable option. It is also easy to get wrong: an uncalibrated judge happily passes bad answers or fails good ones, and you end up optimising toward its quirks.

The method we use, including in our own testing: have people label a sample of outputs against the same rubric, run the judge on that sample, and compare. Track false passes and false fails separately — a judge that passes everything looks great on average. Iterate on the rubric and judge prompt until agreement with human labels is high, and only then trust the judge on the full set. We have also found that a model's self-reported confidence is a poor stand-in for this; measure against labels instead.

Rubrics beat scores

Five specific yes/no questions written from how a human actually grades the output are far easier to calibrate than a single "quality out of 10".

3. Make the output fit your software

If code consumes the model's output, "usually valid JSON" is a production incident waiting to happen. Major model APIs support schema-constrained (structured) output: you supply a schema and the response is generated to match it. Validate it anyway before it touches your database.

In an AI assessment and reporting app we built for a leadership consultancy, templates are generated as typed JSON from a free-text topic, and executive summaries follow a fixed section contract. We also gave the summariser labelled findings instead of raw scores, which removed a whole class of confident, numerically wrong statements. Giving the model less — only what it needs — often improves accuracy and lowers cost at the same time.

4. Control what it can do, and make actions safe to repeat

Prototypes typically run with an admin API key. Production AI that takes actions must act as a specific user with that user's permissions, so it can never do more than the person on whose behalf it runs.

Route actions through your existing validated API, not direct database writes, so business rules are enforced in one place.
Make writes idempotent. Network retries, model retries and duplicate tool calls happen. An invoice must not be sent twice because a request timed out.
Require confirmation for irreversible actions, especially when the AI reads untrusted content (emails, documents) that could contain injected instructions.
Log every action with the user, inputs and result.

These are the defaults behind the remote MCP server we built so a consultancy's staff could run operations from Claude: each tool proxies an existing authenticated endpoint and inherits the signed-in user's permissions.

5. Plan for the model being slow, wrong or unavailable

Timeouts and retries with backoff, distinguishing per-minute rate limits from daily quotas and from real errors — each needs a different response.
Surface retry timing to users instead of failing silently when a provider returns a quota error.
A non-AI fallback. If summarisation fails, show the source; if classification fails, route to a person. The feature degrades instead of breaking.
Background jobs for slow work. Do not hold a web request open for a 30-second model call; queue it and stream or notify.

6. Know what each request costs

Track tokens and cost per feature, per customer and per model from day one. It turns pricing decisions — included, metered, add-on — into arithmetic, and it catches runaway loops early. It also lets you safely move easy steps to a smaller model: if the evaluation score holds, the saving is real. Our AI agent cost breakdown walks through the token arithmetic at current prices.

7. Trace everything

When a customer reports a strange answer, support needs to see exactly what the model received, which tools it called, what came back and how long each step took. LLM-observability tools such as Langfuse, or traces exported through OpenTelemetry, give you that per-request history. Store prompts and outputs only where your data agreements allow, and redact what you do not need.

8. Roll out gradually

Shadow mode. The AI runs on real traffic but only proposes; people still act. Compare its proposals with what they did.
Feature flag to a small group. Real users, real feedback, easy rollback.
Supervised autonomy. The AI acts on the cases the evaluation set shows it handles reliably; everything else goes to a person.
Wider rollout, with the evaluation suite running on every prompt or model change and production failures feeding back into it.

When the right answer is to stop

Sometimes the evaluation set shows the idea does not work well enough on real data, and no amount of prompt tuning closes the gap. That is a successful outcome of a proof of concept, not a failure — it cost weeks instead of a launch. The honest options are then to narrow the scope, fix the underlying data, keep a person in the loop permanently, or drop the feature.

If you have a prototype that needs to become a dependable feature, that is the core of our AI integration services. For agents that take actions across systems, see AI agent development.

FAQ

Why do so many AI proofs of concept never reach production?

Usually because nobody can show the feature is reliable on real data. Without an evaluation set, teams cannot tell whether changes help, cannot set an accuracy bar, and cannot justify the risk of launching. Missing permissions, cost tracking and failure handling add to the delay.

How big should an evaluation set be?

Large enough to cover the real variety of inputs, including the awkward cases. Many features start with 50 to 200 examples and grow as production failures are added. A small honest set beats a large set of easy, invented examples.

Can we trust an LLM to grade AI output?

Only after checking it against human labels. Compare the judge with people on a sample, track false passes and false fails separately, and refine the rubric until they agree. Then re-check periodically.

Do we need to rebuild our prototype for production?

Not usually. Most of the work is adding what the prototype skipped — evaluation, structured output, permissions, failure handling, tracing and a rollout plan — around the core idea, inside your existing codebase.

Sources

Originally published on the DablewDevs blog. DablewDevs builds production AI systems and software — AI agents, RAG, MCP servers and SaaS products. See what we build and what it costs.

Made a free Salah (prayer) app: accurate times, Adhan, Qibla — fully on-device, no data collection

Waleed Siddiqui — Mon, 22 Jun 2026 18:17:24 +0000

I'm a solo dev, and as a Muslim I use a prayer-times app every day — but the popular ones got bloated, ad-heavy, and quietly data-hungry (accounts, location uploads, trackers). I wanted the opposite, so I built Salah Tracker.
Everything runs on your device — no account, no login, no servers of mine, no data collection. Your location is only used locally to calculate prayer times and Qibla.

What it does:
- Accurate prayer times for your exact location (calculation method + madhab adjustable to match your local mosque)

Adhan (call to prayer) notifications, on time, even when the app is closed
Qibla compass
Prayer tracking with streaks, plus Azkar & a Tasbih counter
Light/dark mode It's free (one small banner ad) and Android-only for now. It's a v1 from a solo dev, so I'd genuinely love honest feedback on accuracy and the notifications — and what you'd want next. Disclosure: I'm the developer. Play Store link in a comment 👇 Play Store: https://play.google.com/store/apps/details?id=com.istiqamah.app Happy to answer anything about how the prayer-time calculations work (methods, madhab, Qibla) — and feedback is very welcome 🙏