DEV Community: Teamvoy

What Does LLMOps Cover That MLOps Does Not?

Teamvoy — Mon, 21 Sep 2026 16:25:11 +0000

What the operations layer around a production language model actually contains, and what it costs to skip it.

Picture the failure. Support deflection has sat at 61% for six weeks. On a Tuesday it reads 48%. Your repo has no new commits. Latency is flat, the error rate is zero, every HTTP response is a 200. The provider rolled a point release behind the model identifier you pinned loosely, and the summarization step now writes three sentences where it wrote one, so the routing logic downstream stopped matching.

You find out from a customer.

That gap between “the service is up” and “the answers are right” is the entire subject of LLMOps. It is the engineering between a model endpoint and the people depending on it, and in 2026 it is the part of the stack most teams shipped without.

The thing you are operating is a configuration, not a model

Here is the distinction that makes LLMOps its own discipline rather than MLOps with new vocabulary.

In MLOps you version training data, features and weights. You own the artifact. When it degrades, the cause is data drift or a broken feature pipeline, and the fix is a retrain on your schedule.

In LLMOps the artifact belongs to a vendor. What you own is everything wrapped around it: prompt versions, a retrieval corpus, tool contracts, a routing config, a provider and model version. A regression comes from a version bump you did not schedule, a corpus update someone ran at 4pm, or a one-word prompt edit that passed review because nobody had a way to test it.

Rollback means restoring a previous combination of prompt, corpus and provider. There is no artifact to roll back to. That difference cascades into five components, and most engagements start with whichever one is missing.

Evaluation suites. A versioned case set covering the paths users take, run in CI against every prompt edit, corpus change and provider bump, with a pass threshold gating release.
Tracing. Every model call, retrieval and tool invocation, captured at the request level.
Versioning. Prompts, corpora and tool contracts under change control, the same way code is.
Cost telemetry. Spend attributed per request across a multi-step path.
Release control. A documented rollback path someone has actually used.

Evaluation is the control plane, and logging is not

The industry spent two years buying tracing tools and calling it observability. Tracing tells you what happened after it happened. An evaluation suite stops the bad change from shipping.

A working suite is a versioned case set with graded expectations, run in the same pipeline as your unit tests. Four kinds of case earn their place:

Golden cases with known-correct answers.
Adversarial cases covering ambiguous and out-of-scope input.
Multi-step cases that exercise the tool path, including the loop that runs twice.
Regressions pulled from real traces, added the week each one broke.

The pass threshold blocks the merge. That single mechanic turns a prompt edit from a judgment call into a diff someone reviews on evidence, and it turns a provider release from an incident into a report: the suite runs against the new version, the delta comes back case by case, and the routing config either moves or stays.
Now the honest part. Real evaluation adds weeks to a first release and returns nothing on that release. It pays back on the first model change and every one after. A team shipping one prompt into a low-volume internal tool should skip it and spend the time elsewhere.

What a trace has to carry before it is useful

Most LLM tracing captures the model call and stops. That covers maybe a third of your failure modes. A trace that answers real questions has four layers.

Retrieval. Which documents came back, what they scored, and whether the answer actually cited them. Most quality complaints resolve here, and they resolve fast when the retrieved chunks sit next to the output.
The model call. Provider, model identifier, prompt version, tokens in and out, time to first token, total latency.
Tool calls. Which tool the model picked, the arguments it passed, what came back, and how many times the loop ran before it stopped.
Outcome. Whether the request resolved, escalated to a person, or got retried.

Join that last layer to cost and the number changes meaning. Spend per million tokens is an accounting figure. Spend per resolved request is an operating one, and it is the only version of the metric that tells you whether a cheaper model is actually cheaper once it escalates twice as often.

In a regulated deployment, the ops layer is the evidence layer

If you run in fintech, insurance or health, the auditor will ask for records rather than architecture. Four artifacts carry that weight, and each is a design decision taken before launch rather than a report generated after one.

Version lineage per decision. For any output that affected a customer: the prompt version, model identifier, retrieval snapshot and tool responses behind it. Under SR 11–7 model risk guidance, this is what makes a language model component reviewable in the same terms as a credit scorecard.
Human review records. Which outputs a person approved, changed or rejected, timestamped and attributable. Where a fully automated decision would sit in scope under the EU AI Act, this is the control teams most often rely on to stay outside it.
Retention and residency. Traces hold customer text. Retention windows, redaction and inference region are GDPR questions rather than infrastructure preferences, and DORA adds concentration risk where one provider serves every route.
Change control. SOC 2 change management applied to prompts and corpora, not only to code, because both move model behavior.

Teams that treat these as a compliance workstream separate from engineering build the system twice.

Where the classical model still wins

One more thing worth saying out loud, because vendors rarely say it.

On a stable, high-volume classification step, a small trained classifier or a deterministic rule beats a model call on cost, latency and predictability. Every time. If the step is “route this ticket to one of nine queues” and you have labeled history, the language model is the expensive option with the worse audit story.

Teams already running classical models are in the best position here anyway. The registry, the deployment pipeline, the monitoring backbone and the incident rota serve both disciplines. Splitting them doubles the operational surface for no return. What diverges is the clock: a classical model gets retrained on your data, a model route gets reconfigured on a vendor’s release schedule.

Four questions for whoever you hire

Apply these to us as readily as to anyone else.

Who writes the evaluation suite, and when? If the answer is “after the first release,” evaluation is an add-on rather than part of the engineering.
Where does inference run and who holds the keys? Your cloud account, a vendor platform, or a mix, and the data path in each case.
What is the handover artifact? A running system with traces, a documented case set and an on-call rota is a different deliverable from an architecture diagram.
What did the team do the week a provider last deprecated a model?

This one is hard to fake.
We wrote up how we run this work, including a 3–5 day readiness audit that leaves you with a written gap list you can act on with any partner: LLMOps consulting services.

Or take the shorter path. A 15-minute call with our CTO about where your model runs, what your traces show today, and what breaks the next time a provider ships a version bump.

If Reviewers Accept 95% of Drafts Unchanged, They Stopped Reading

Teamvoy — Tue, 15 Sep 2026 15:44:02 +0000

Track one number when you put a generative model into a review workflow: how often reviewers accept the draft unchanged.

Above roughly 95% and they've stopped reading. That's automation bias, and it looks exactly like success on every dashboard you have. The person's job changed from doing the work to approving it, which is a different job with different failure modes, and the main one is approving what they should have caught.

This is the shape of the whole problem. A prototype proves the model can produce the artifact. Production asks whether you can show a regulator, six months later, why a particular output was produced — and whether a customer can challenge it. Those are design problems long before they're engineering problems.

The integration work is roughly 20% model and 80% everything else: data pipelines, evals, audit logging, human review paths, and the interface showing a user what the model did.

Three things change when the output is probabilistic
The same input can produce a different artifact tomorrow. Interfaces built on deterministic assumptions break quietly. Pin temperature and seed or you cannot reproduce a decision.
The audit surface grows. Every inference becomes a record you may need to reconstruct.
The human role moves from producing to approving, with the acceptance-rate problem above.
Generative and predictive are different products wearing one name

A credit model returning a default probability is predictive. An assistant drafting the adverse-action notice that follows is generative. Vendors blur this and buyers inherit the blur — one platform sells "AI for lending" and you've bought two unrelated risk profiles under one contract.

Predictive  Generative

Output Score, class, ranking Text, document, summary
Evaluation Precision, recall, AUC vs labels Faithfulness, groundedness, human preference
Failure mode Systematic bias against a group Confident fabrication, one output at a time
Found by Aggregate analysis Per-instance review
Reproducibility Deterministic given version Varies unless parameters pinned
Cost profile Training-heavy, inference-cheap Training-light, inference-expensive

A generative feature needs a review path. A predictive feature needs a challenger model. Building one and governing it as the other is the most expensive mistake in the category, because it's usually found during an examination rather than during a sprint.

RAG sits between them and confuses things further. Govern the retrieval as a data-access control and the generation as a content control. Treating it as one component makes both halves harder to defend.

Four prerequisites, each with a pass/fail test

Run these in week one, on paper, before anyone writes a prompt:

Data — can you pull 200 real cases with labeled correct outcomes without a manual export? If that takes more than two days, the data problem is your project.
Team — is a compliance reviewer named in the sprint, not on the distribution list?
Baseline — is there a current metric with today's number written next to it?
Sponsor — does one executive own the outcome and the budget, in writing?

The data test fails most often, and it fails in week five, after the model work is already scheduled. That moves the launch by a quarter.

Write the eval suite before picking the model

The sequence teams actually run: pick the model in week one because it's the interesting decision, design the interface around whatever it produces, hand the whole thing to compliance in week ten. Reversing that is most of what separates a six-week build from a six-month one.

The dependency order that can't move:

You cannot validate a model against evals you haven't written.
You cannot write evals without knowing what a correct output looks like in the workflow.
So the domain expert who does this work today has to be in the room before model selection, scoring 100 to 300 real cases.

Cost, latency, data residency, and vendor retention policy narrow the field faster than benchmark scores do. And test for disparate outcomes on protected characteristics even when the output is a draft rather than a decision — a drafting assistant that writes warmer letters to one group is a fair-lending problem wearing a different hat.

Design the reviewer's screen before the customer's

The review interface decides whether reviewers catch errors or rubber-stamp them, and that single behaviour sets the feature's real error rate more than model quality does. It's the part of the work that never shows up in a portfolio.

Beyond acceptance rate, two things help: put the retrieved records next to the generated draft so the grounding is visible, and seed known-bad cases into the queue to check people are still catching them.

On the customer side, three patterns carry the trust load:

Confidence disclosure that states the basis, not a number. "Drafted from your March statement and two prior disputes" tells a user more than "87% confidence" and is much harder to misread.
Reversible action. Anything the model initiates gets an undo with a stated window. Where reversal is impossible — a sent payment — require confirmation before rather than explanation after.
A visible human path. One click, with expected response time on the button.

When a fraud hold lands on a legitimate transaction, the customer doesn't want an explanation of the model. They want the hold lifted and they want to know how long that takes. Trust in financial products is mostly a function of recoverability. Users trust what they can undo more than what they can read.

Two anti-patterns: acting without confirmation on anything material, and decorative explainability — a "why did I see this" link returning generic text. A weak explanation is worse than none, because it signals the team knew the answer mattered and chose not to give one.

Audit log before prompt

The artifact people underbuild. It has to store the prompt, the retrieved context, the model version, the parameters, the output, and the reviewer's decision — keyed so one case can be reconstructed on request.

The examiner's first question is rarely about the model. It's who signed off, what they saw when they did, and whether you can produce that record now. Teams that treat compliance as a document written after the build find the document has to describe decisions nobody wrote down.

Cheap to build in week one. Expensive to retrofit, and the feature sits idle while it happens.

What it costs, and where

A first production feature in a regulated fintech workflow: roughly $110,000 to $220,000 for a customer-support-style assistant, $320,000 to $1.1M for transaction monitoring, over three to six months. Run costs $2,000 to $25,000 a month depending on volume.

The model is the cheap part — inference on a well-scoped workflow is often under $2,000 a month at pilot volume. The spread comes from data readiness and regulatory tier, not model choice. Rough split: data prep 25–35%, model integration and evals 15–20%, interface and review design 20–25%, audit logging 10–15%, validation and documentation 10–20%.

What pushes a build past the top of the range is nearly always the same thing: the workflow wasn't scoped to one queue, and it grew a second and third use case before the first one shipped.

One trade-off this approach doesn't solve. Running compliance in parallel from step one makes your first feature slower than a team that skips it — measurably, often four to six weeks. The payback lands on features two and three, when the control framework is reusable. If your organization judges the first project on speed alone, negotiate the measure before the project starts rather than cutting the governance work.

Scope to one workflow with a number attached. Write the kill criteria before the code.

Full guide — the seven-step process, the regulatory table, and build vs buy: see the full guide

Written by Bohdan Varshchuk, CTO at Teamvoy. More engineering writing at teamvoy.com/blog.

Ten Things to Wire Up Before an Agent Touches Production

Teamvoy — Tue, 08 Sep 2026 18:53:46 +0000

A developer deployed a customer-support agent that got stuck in a retry loop with a CRM tool. No hard circuit breaker. It spent six hours overnight repeating the same broken call while he slept, and he woke up to roughly a $4,200 OpenAI bill for doing nothing useful.

That's the failure mode. Not a dramatic hack — a boring loop with no external stop.

Roughly 95% of enterprise generative-AI pilots in 2025 delivered no measurable return, and almost none of those were model failures. Teams optimized the brain and skipped the nervous system. A read-only chatbot is a wiki with better search. A deployed agent does things — writes to a database, calls an API, refunds a customer. That shift from reading to doing is where pilots die.

The useful mental model: treat the LLM as a fallible kernel, not a magic box. You don't trust a kernel blindly. You wrap it in checks, limits, and a way to roll back.

Agents break three assumptions your ops playbook depends on

Non-determinism. The same input can produce different actions on different days. An agent succeeds Monday and fails on the identical request Tuesday. Call it ghost debugging — the bug won't sit still long enough to catch. Design for a 3% to 15% tool-call failure rate as a normal state, not an exception.

Cost grows with loop length, not request count. This is the mechanism most guides skip. The agent loops: think, call a tool, read the result, think again. Frameworks append every step and every tool error to the running history, then resend the whole cumulative log on the next call. Token use grows quadratically. A 20-step loop is not twice a 10-step loop — each step re-pays for everything before it. Related trap: past roughly the 40% context-fill mark, answers get worse. Load the window with tool definitions and raw JSON and you're doing your real work in the dumb zone.

Every tool you connect widens the blast radius. Which brings us to the controls.

Four controls the agent cannot touch

Wire these before you touch agent logic, not after. The principle is determinism of enforcement: limits live outside the model, in code it cannot rewrite. An agent told to stay under budget will, often enough, talk itself into one more call.

Circuit breaker. Hard cap on retries and loop iterations. Hit it and the agent stops, full stop.
Deny-by-default allowlist. The agent calls only tools you explicitly permit. Everything else is blocked, not warned.
Confirmation gate. Anything destructive or irreversible — delete, refund, send — waits for a human yes.
Cost ceiling. Spend limit per run and per day, enforced by infrastructure rather than by asking the model nicely.

Research on action-level privilege control found enforcement cut attack success from 70.3% to 7.3%, and to zero with manual policies. The control plane protects you, not the prompt.

Worth being honest about the limit: external controls stop catastrophes. They don't make a weak agent good. They buy you the safety to improve it in production instead of gambling with it.

Three identities, not one

Most teams give an agent one set of credentials. That's how a single prompt injection becomes a full breach. Split it:

The user — who asked for the action.
The agent — its own machine identity, separate from any human.
The tool token — a scoped, on-behalf-of credential per tool call.

In a DORA- or PCI-DSS-bound system, "the AI did it" is not an answer an auditor accepts.

There's a quieter risk in the tool descriptors themselves. A study of 1,899 live MCP servers found 7.2% carried general vulnerabilities and 5.5% were open to tool poisoning, where a malicious description hijacks the agent. Pin and version your descriptors. Treat them as supply-chain code, not config.

And assume prompt injection — direct, where a user types it, and indirect, where the agent reads a poisoned page or document and follows hidden orders. You cannot prompt your way out of this. Checks run outside the model, in deterministic code. One scan of 5,000 AI-built apps found 60% vulnerable.

Ship it in five stages

Never all at once. Each stage answers one question before the next earns traffic.

Offline eval suite — proves broadly correct on known cases, before any real traffic.
Shadow mode — the agent decides, nothing executes. Proves it behaves sanely on live data at zero risk.
Canary — a small slice of low-risk real traffic. Proves it survives messy production input.
Graduated rollout — widen in steps, confirmation gates on anything destructive.
Post-deploy validation — behaviour drifts as inputs and model versions change. Proves it stays correct.

The rollback detail teams get wrong: an agent's behaviour is set by four things together — code, prompts, tool catalog, model version. Reverting code while keeping yesterday's prompt is not a rollback. It's a new configuration you have never run. Revert all four as one atomic bundle.

Monitor the trajectory, not the endpoint

Uptime dashboards tell you the agent returned a 200. They don't tell you the answer was confidently wrong. Three surfaces:

Operational — is it running? p95 latency, error rates, cost per run.
Cognitive — what did it decide, and why? Full reasoning trace and every tool call, in order.
Contextual — what did it see? Exact inputs, retrieved documents, tool outputs.

Track task-completion rate, tool-call success rate, tokens per run, and latency. Drift in any of them is the early warning.

Almost-right output passes every uptime check. It's the expensive failure mode precisely because completely wrong gets caught — tests fail, the build breaks — while almost right ships and compounds for months. AI-generated pull requests average 10.8 issues against 6.4 in human code, and the tell is often suppression rather than error: one review turned up a file carrying eleven eslint-disable comments where the agent had silenced the type errors instead of fixing them.

Three questions before any agent-written change merges: does it reuse what exists or reinvent it badly, does it follow your conventions or invent its own, and can a developer explain it without reading the AI's comments?

What it actually costs

Inference is rarely the big number. Integration and compliance run 40% to 60% of total deployment spend. Get the data layer wrong and everything else inflates.

Pick hosting by workload shape rather than by which is "cheaper": serverless for spiky or unpredictable volume, containers for steady high volume, hybrid if you genuinely have both and can afford to run two systems. Cloud is rented elasticity — if your load is steady, you're paying a premium for flexibility you aren't using.

Then cap the variable part: token budgets per run and per day enforced in infrastructure, semantic caching so repeated questions don't re-pay the model, and model tiering so the expensive model only runs on the steps that need it.

The ten-line go/no-go

Empty box means you're shipping a demo:

Eval suite passes.
Staged rollout ready — shadow, canary, graduated, with gates.
Circuit breaker live, enforced outside the model.
Cost ceiling set in infrastructure.
Tool allowlist is deny-by-default.
Three-identity auth.
Trajectory logging on.
Atomic rollback bundle — code, prompts, tools, model version.
Spec-first verification with the three-question test.
Named incident owner — a specific human, not a team alias.

If you went quiet on three or four of those, you're in the same place as most teams shipping their first production agent. Writing the agent is the cheap part. The gap between a working demo and something you trust at 2 a.m. is everything above.

Full guide — architecture patterns, identity delegation, cost modelling, and deploying onto a legacy core: teamvoy.com/blog/ai-agent-deployment-best-practices

Written by Taras Voytovych, Founder & CEO at Teamvoy. More engineering writing at teamvoy.com/blog.

The Systems That Need Modernizing Most Aren't the Broken Ones

Teamvoy — Thu, 03 Sep 2026 11:54:36 +0000

The applications that most need modernizing are usually not the ones that are obviously broken.

They're the stable ones. Running for years, rarely paging anyone, and increasingly impossible to change. Nothing on a dashboard flags that, because "hard to change" isn't an outage — it's a slow tax paid in every estimate that comes back three times longer than the feature deserves.

Which is why the first question shouldn't be what technology replaces the current stack. It should be: what does the current architecture prevent the product and engineering teams from doing? That question has a concrete answer. "Should we move to microservices" does not.

Modernization is not a synonym for rewrite

This is the assumption that turns a scoped project into a two-year program. The real decision is what stays, what changes, and where the investment actually removes a constraint on the business.

The 7 R's are useful precisely because two of the seven are "don't touch it":

Retire — it no longer provides value. Delete it.
Retain — still useful, still cost-effective, leave it alone.
Rehost — move it, minimal changes. Fastest path to a new environment, solves the least.
Replatform — new platform, minor changes to optimize.
Refactor — improve the code's structure without changing external behaviour.
Rearchitect — redesign for new capability or scale.
Replace — retire it and adopt something else, often SaaS.

Run every application in the portfolio through that list before picking a strategy for any of them. Teams that skip this step tend to apply one R uniformly, usually rearchitect, because it's the most interesting one.

Worth being blunt about the trade at each level: rehosting is the fastest way to the cloud and will carry every legacy problem with it. Rearchitecting and rebuilding are where the real capability change happens, and where the schedule risk lives.

What actually goes wrong

Three challenges, and each has one thing that contains it:

Technical debt. Legacy code makes every update slower than it should be. Contained by structured assessment first — you cannot sequence work you haven't inventoried.
Integration complexity. Connecting modern components to legacy systems is where the unknown work hides. Contained by phased rollout, so an integration surprise costs one phase instead of the program.
Security risk. Changing infrastructure exposes new surface. Contained by continuous monitoring through the migration, not a review at the end.

None of those three are technology problems. They're sequencing problems, which is why the assessment phase is the one worth over-investing in.

Tie it to a number or it drifts

Modernization without a defined outcome becomes an infrastructure project that nobody can defend at budget time. Three things keep it honest:

Named KPIs. Cost per deploy, time to ship a feature, incident rate — whatever the constraint actually was. Define it before the work starts.
Cross-functional ownership. IT, business, and operations working from the same target. Modernization that only engineering can explain the value of gets cut.
Prioritized workloads. Start with the applications where the constraint costs the business the most, not the ones easiest to migrate.

For context on scale: 83% of C-suite executives see app and data modernization as central to business strategy, and worldwide public cloud end-user spending was forecast at $723.4 billion for 2025, growing 21.5% year over year. The money is moving. Whether your slice of it removes a constraint or just relocates one depends on the sequencing above.

Where AI helps, and where the data layer stops it

Two things worth separating. AI in the modernized system — automation, forecasting, analytics — needs clean, governed data to be worth anything. If the data layer is the actual blocker, that's the modernization project, and no amount of application refactoring routes around it.

AI in the migration is a different claim. It compresses the mechanical parts: reading unfamiliar code, generating tests for untested paths, drafting the boring half of a refactor. It doesn't compress the assessment or the integration work, because those are constrained by other people's systems and by decisions someone has to own.

The sprint framing is the one part of the process worth copying regardless of scale: short iterative cycles against one measurable constraint, so you find out early whether the approach works. A modernization program that shows nothing for six months isn't lower risk than a rewrite. It's a rewrite with better branding.

Start with what the architecture prevents, not with what would replace it.

Full guide — all seven R's, the sprint model, and partner selection criteria: see the full guide

Written by Bohdan Varshchuk, CTO at Teamvoy. More engineering writing at teamvoy.com/blog.

Every AI Readiness Score Skips the Part That Actually Blocks You

Teamvoy — Tue, 01 Sep 2026 17:42:44 +0000

Over 85% of AI projects fail to reach their full potential, and the reason is almost never that the model didn't work. It's infrastructure, data hygiene, governance, or missing skills.

Which is what readiness assessments are supposed to catch. Except most of them stop exactly where the blocking problems start. They score strategy, data, infrastructure, governance, and culture — all at the level a survey can reach, filled in by one executive, without an engineer in the room.

So you get a "governance: strong" rating that means a policy document exists. It does not mean a model-risk committee, a SOC 2 auditor, or an enterprise security team will accept that policy as evidence. Those are different questions, and you find out which one you were answering months later.

Score yourself before paying anyone

Eight questions. More than two "no" answers means a formal assessment will find real gaps — useful information, not a reason to skip it.

Strategy — Is there a named AI initiative with a defined business outcome, or is "we're exploring AI" the whole plan?
Data — Can the team point to the specific source an AI feature would read from, and confirm it's clean enough to trust?
Infrastructure — Does the current setup support the latency and scaling the feature needs, or would it need rework first?
Governance — Is there a named owner for AI decisions with a documented approval path, or does responsibility sit nowhere specific?
Culture — Have the engineers who'd build this used the intended tools before, or would it be a first attempt under deadline pressure?
Codebase health — Would a new engineer joining today understand the code well enough to safely add an AI component, or is onboarding already a multi-week process?
Evaluation coverage — Is there any automated way to check whether a model's output is correct, or does a person read it?
Regulatory evidence (regulated industries) — Can you produce model risk documentation, an audit trail, and a data residency answer today, or would all three be built from scratch?

Six or more "yes" and you're closer to production-ready than most tools assume by default. Fewer than four points at a real assessment before committing engineering time.

The two questions nobody's scorecard asks

Codebase health. This is the one that actually decides whether a pilot ships. 88% of organizations use AI in some form, but fewer than 20% track results for generative AI specifically — a measurement failure that starts in engineering, not strategy. Pilots stall for three repeating reasons: the codebase is too tangled to hold a new component safely, test coverage is thin enough that every change risks something else, and the team is overloaded or missing the specific skills the project assumed. None of those show up on a strategy survey.

Eval coverage. If correctness is checked by a person reading output, you don't have evaluation, you have review. That works until volume arrives.

If you're regulated, "governance" is six separate things

A generic assessment treats governance as one pillar. For a bank, lender, payments company, or insurer it has to produce evidence someone external will accept:

Model risk management — SR 11-7-style validation. Who approved the model, what it was tested against, how drift is monitored after deployment.
Audit trail — every model decision that touched a real transaction, retained and retrievable, not logged and forgotten.
Data residency — where training and inference data physically sits, and whether a provider outside the EU ever touches EU customer data.
EU AI Act classification — whether the feature is high-risk, which changes the documentation burden before deployment.
SOC 2 and PCI DSS — proof the AI layer didn't open a gap in controls that were already passing.
FFIEC and NYDFS Part 500 — examiner expectations for third-party risk now extend to AI vendors and AI-generated decisions.

Pass a generic assessment on governance, then stall in compliance review months later. Same failure the assessment was meant to prevent, just moved downstream.

If you're AI-native, the question is inverted

You're not asking "can we deploy AI safely." You're asking "will our product survive an enterprise buyer's security review." Five things get asked, in roughly this order:

Multi-tenancy isolation — can one customer's data, prompts, or fine-tuning artifacts reach another's session? This is the first question, every time.
Eval harness maturity — automated evaluation covering multi-step paths, not a spot check before release.
RAG infrastructure — document versioning, access control at the chunk level, stale-index detection.
Production monitoring — observability on every model and tool call, so a bad output traces back to the input that caused it instead of being reported by a customer.
SOC 2 readiness — whether the posture needed to close the deal exists or is aspirational in a deck.

Score well on a generic readiness tool and still lose the deal, because the tool never asked any of these.

The mistakes that turn a diagnostic into shelfware

Treating the score as pass/fail. A 60% isn't a verdict, it's an ordered list of what to fix. Waiting for a perfect score delays a project that could have shipped after three specific fixes.
Running it without the engineers who'd build the feature.
Never re-running it. Readiness changes as gaps close, and you have no way to confirm the fixes worked.
Choosing by brand recognition rather than scope. A strategy tool answers whether leadership is aligned. An engineering tool answers whether the codebase can carry what's planned. They aren't substitutes.

The useful thing about a readiness assessment isn't the number. It's finding out, before the budget moves, which of these questions you can't currently answer.

Full comparison of ten tools — what each scores, who it fits, what it costs: teamvoy.com/blog/best-ai-readiness-assessment-tools-in-2026

Written by Zhanna Yuskevych, CPO at Teamvoy. More engineering writing at teamvoy.com/blog.

Most of You Should Not Be Custom-Building an Ecommerce Store

Teamvoy — Mon, 31 Aug 2026 12:08:25 +0000

Ecommerce projects rarely get hard because of the storefront. They get hard when the storefront has to meet the needs of the rest of the business.

Inventory already lives somewhere. Orders land somewhere else. Finance has its own systems. Payments drag security and compliance into scope. And the platform picked in month one defines what's easy, expensive, or impossible two years later.

Which is why the first question shouldn't be Shopify vs Magento vs WooCommerce vs custom. Starting there means arguing about features before anyone understands the constraints. The better opening question: what are we actually building, and which parts genuinely need to be custom?

For most people reading this, the honest answer is buy a platform and move on.

The phase ordering that costs the most

Seven phases: discovery, architecture and platform selection, design, build and integration, data migration, compliance and security review, launch and first optimization. Four to seven months for a mid-market build.

The sequence matters more than the labels, and one inversion is expensive above all others: choosing a platform before you've scoped compliance. Do it in that order and a compliance finding can force a re-architecture eight weeks into the build.

Two phases account for most overruns:

Build slips because discovery missed a system. The integration nobody scoped in phase 1 surfaces two months late, when the schedule has no slack left. Discovery in commerce isn't about pages — it's about finding every system holding a number the store needs, and finding who can grant access to it. That list is always longer than the brief.
Compliance review slips because it was treated as a formality. Remediation lands on a team that has already released its engineers.

Both are schedule problems created long before they appear.

Staffing note worth agreeing in writing before kickoff: phases 1, 2, and 6 need someone on the client side — a person who can get you into the ERP, whoever signs the contract, and named owners for the payment page and accessibility. Phases 3, 4, and 5 can sit almost entirely with the build team. The three phases you have to staff are the three that stall when nobody's free.

Three dates that already passed

These are no longer planning questions. They're evidence questions.

PCI DSS v4.0.1 req 6.4.3 and 11.6.1, in force since 2025-03-31. Inventory every script on the payment page, authorize each one, justify why it's there, and detect unauthorized changes to page headers and content. The catch: the scripts in question are usually marketing tags nobody in engineering added. PCI now has an owner problem — the tag manager on checkout needs a named owner and an approval step.
European Accessibility Act, Art. 2(2)(f), applying after 2025-06-28 to ecommerce services sold to EU consumers. Measured against EN 301 549, which points at WCAG. "We'll do accessibility after launch" is now a statement about legal exposure, not backlog priority.
GDPR / CCPA / CPRA, unchanged but load-bearing. Consent, access, and deletion paths wired into the customer data model rather than bolted on. If a deletion request can't be executed against the customer record without an engineer, it doesn't scale past the first hundred.

Accessibility written as an acceptance criterion in phase 3 costs days. Discovered in phase 6, it costs weeks. Same work, different bill.

Checkout: the causes are business decisions, not design problems

The average documented cart abandonment rate is 70.22% (Baymard, across 50 studies, updated 2025-09-22). Among shoppers who intended to buy, the top causes:

Extra costs shown too late — shipping, tax, fees — at 40%.
Slow delivery, 20%.
Distrust of the payment form, 19%.
Forced account creation, 18%.

Three of those four are decisions no redesign can make on your behalf. In order of what actually abandons carts: show the full total on the cart page, offer guest checkout, give a real delivery date rather than a shipping-method name, make the payment step look like a payment step, and cut every optional field.

Run A/B tests only after those five are in place. Below a few thousand checkouts a month you don't have the traffic to detect a one-point conversion change without weeks per variant, and teams routinely call the result on day four because the chart looks decided. Testing earns its place above the checkout — category layouts, product page structure, price and delivery presentation — where there's more traffic per decision.

Where AI actually earns its place

The test: does the model's output change a decision somebody was already making on worse information?

Demand forecasting and replenishment. Output is a purchase order quantity. Error is countable against what sold.
Fraud and chargeback scoring. Output is a score with a threshold. Both false positives and false negatives show up in the finance report — which means the review interface is in scope, not a follow-up.
Search and merchandising, but above roughly ten thousand SKUs, where manual merchandising already stopped covering the tail.

Where it doesn't: personalization on a 200-SKU catalog has too little behaviour to learn from and the engineering cost lands anyway. Dynamic pricing on consumer goods invites a customer to screenshot two prices and post them. Generated product copy at scale reads like generated product copy at scale, on exactly the pages you wanted indexed.

Separately — AI in the build is a different question from AI in the store. Coding assistants compress phases 3 and 4. They don't touch 1, 5, or 6, which is where the schedule risk lives, because those are constrained by other people's systems and by evidence someone has to sign.

Composable, honestly

Headless decouples the front end and is a configuration option on several hosted platforms. Composable replaces the engine with separate services. Headless is a weekend of architecture; composable is a program of work. The terms get used interchangeably and shouldn't be.

On a composable build, expect the first feature to take four to six weeks longer than on a coupled platform, and expect payback to start around the third. If your roadmap has three changes in it, the arithmetic doesn't work.

You also inherit a service layer somebody maintains, and six services that fail in more ways than one platform does. "Wanting to be modern" is not a reason. Multi-channel selling from one inventory, or business rules no vendor has productized, are.

If the reason is real, migrate incrementally: routing layer in front of the existing store, move one capability behind it, run both, move the next. Search and product content first — read-heavy, low-risk. Checkout near the end, because it handles the money and the compliance scope. Teams that move checkout first take the program's largest risk before learning anything.

The three-year number

Build cost is the number in the quote and the smaller half of the decision. A $70,000 build with $12,000 annual maintenance and $1,500 a month in platform fees is a $160,000 commitment before a single new feature ships.

Price every platform over three years, because the fee structures behave differently: revenue-percentage platforms get more expensive as you succeed, self-hosted charges you in engineering hours whether you grow or not, licence-plus-build front-loads and holds steady. Ask each vendor what the bill looks like at three times current volume, and get it in writing. That belongs in phase 2, not the post-launch review.

And then the question underneath all of it. Under roughly $5M in annual online revenue, with a standard catalog and standard checkout, buying wins on price, time, and risk. Custom development is the answer to a named business rule you can attach a cost to living without — never to a general preference for control.

Full breakdown — all seven phases, the platform comparison table, and the build-vs-buy signals: see the full guide

Written by Bohdan Varshchuk, CTO at Teamvoy. More engineering writing at teamvoy.com/blog.

Nobody Budgets for the Redirect Map

Teamvoy — Wed, 26 Aug 2026 20:07:58 +0000

A $5,000 template refresh and a $150,000 rebuild both get called "a website redesign." Then someone requests three quotes without scoping the project and compares numbers that don't describe the same work.

That's how a redesign that started on a small-business budget discovers halfway through that it needed enterprise scope. Not because anyone lied. Because "redesign" is a word that covers two completely different engineering jobs, and the person buying it usually can't tell which one they're in until the content migration starts.

Broad shape for 2026: roughly $3,000 at the small-business end, past $250,000 for complex enterprise builds, with most mid-market work landing between $40,000 and $100,000. Those ranges are almost useless on their own. What matters is which drivers your project actually has.

What actually moves the number

Unique page templates. Not pages — templates. A homepage, product page, service page, blog post, and landing page are five separate design-and-build efforts, each wireframed, built, made responsive, tested, and wired into the CMS. Small sites might need 3 to 5. Large ones run past 15. Figure $1,500 to $5,000 per additional template depending on complexity.
Custom features and third-party integrations. Every integration means research, API configuration, testing, and ongoing support. The ones that hurt are the ones needing custom middleware or two-way data sync, because those keep costing after launch.
Content migration. The most underestimated line by a wide margin. If content is inconsistently structured or spread across platforms, migration alone eats 10 to 20% of the project budget. Automated tools help with extraction and reformatting; manual review is still where the quality comes from. New copy runs $100 to $300 per page.
Accessibility. WCAG 2.1 AA means semantic HTML, keyboard navigation, contrast checks, ARIA attributes, and testing every template. It's real hours. Skipping it saves money up front and buys legal risk later.
Timeline. Rushed deadlines and vague scope both inflate cost, for opposite reasons — one buys overtime, the other buys rework.

The one that quietly costs the most

SEO migration. It is the cheapest thing to skip and the most expensive thing to have skipped.

You need a URL map with 301s from every old path, meta and schema carried across, header structure and alt text preserved, and performance that doesn't regress on mobile. Get it wrong and you launch a beautiful site that lost its organic traffic, which nobody notices for six weeks because everyone is looking at the design.

Build the redirect map during development, not during launch week. It's the artifact that makes the difference between a redesign and an outage with better typography.

What does AI actually take off the bill?

Real, but narrower than the pitch suggests. It compresses the repetitive middle of the project:

Scanning an existing site, extracting content, reformatting it for new templates — genuinely useful on large blogs and news archives.
Resizing, compressing, and tagging images. Auto-generating alt text saves dozens of hours on an image-heavy migration.
Automated passes for broken links, accessibility issues, and performance bottlenecks.

Where it stops: brand narrative, custom integrations against proprietary systems, and interpreting compliance requirements. Automate deliberately — not everywhere at once. On a small project the savings are meaningful. On a complex custom build, automation optimizes the process rather than replacing the expensive part of it.

The number nobody puts in the quote

Ongoing maintenance runs 15 to 20% of build cost, annually. Security patches, CMS updates, performance monitoring, bug fixes.

On a $60,000 build that's $9,000 to $12,000 a year, every year, and it is almost never in the proposal you're comparing. Neither is custom photography ($3,000 to $15,000), video ($5,000 to $25,000 per video), hosting, or premium plugin licensing.

Two things that make a quote comparable: insist on itemized proposals that include content migration, maintenance, and media production as explicit lines, and reserve 10 to 15% for contingency. Scope creep is not a sign anyone failed. It is the default state of a project where the content turns out messier than the audit suggested.

The design is the part everyone argues about and the part that costs the least to change. The redirect map, the integration surface, and the annual maintenance line are the ones that decide whether the number you were quoted resembles the number you pay.

Full breakdown — pricing by business size, all seven cost drivers, and a budgeting checklist: teamvoy.com/blog/website-redesign-cost

Written by Zhanna Yuskevych, CPO at Teamvoy. More engineering writing at teamvoy.com/blog.

Your Main Branch Has Code Nobody Read. Whose Name Is On It?

Teamvoy — Tue, 25 Aug 2026 12:58:20 +0000

Who is accountable for AI-generated code? The engineer who merges it. That answer hasn't changed since 2023. What changed is the volume behind it.

Google's own number: just over a quarter of new code AI-generated in October 2024, roughly half by late 2025, 75% by April 2026 per Sundar Pichai. Almost nobody's review gates, provenance records, or audit evidence moved in that same window. So the accountability gap is the distance between how much of your codebase a model wrote and how much your governance assumes a human wrote.

It stays invisible until the worst possible moment — an incident postmortem, a security review, a customer questionnaire, an examiner asking about your SDLC.

Three failure modes

The review bottleneck. Agents produce diffs faster than humans read them. Review becomes either the constraint on delivery or a rubber stamp. Watch for the tell: approval latency stays flat while actual scrutiny drops.
Provenance blindness. Someone asks "was this AI-generated, and what checked it?" and nothing in the commit history can answer. This is cheap to record now and impossible to reconstruct later.
Shadow tooling. Policy says no, engineers use personal accounts and local models anyway. The company keeps the risk and loses the visibility. We ran this exact movie with unsanctioned SaaS a decade ago.

On the review bottleneck, there's a recurring r/ExperiencedDevs thread shape worth reading: a senior engineer describes review queues doubling after agent adoption, and the replies split between "review harder" and "trust the tests." Both miss the same move. The review standard has to change shape, not intensity. A heroic reviewer and a green CI run are equally poor substitutes for deciding which changes deserve which gate.

Why banning makes it worse

Three postures, and only one of them produces a defensible answer to "who signs this?"

Ban — you get a clean-sounding policy. You pay in shadow usage with zero visibility, a senior hiring penalty, and the loss of the audit trail you were trying to protect.
Laissez-faire — you get speed and goodwill. You pay in no provenance, rubber-stamped review, and unanswerable audit questions.
Governed adoption — you get speed plus evidence that survives an audit. You pay in real setup work, roughly four to eight weeks of process and tooling change.

The ban threads on r/cscareerquestions read identically every time: the policy says no, the replies describe exactly how people route around it, and nobody believes the codebase is cleaner for it. When your engineers can narrate the workaround in public, the ban isn't a control. It's a blindfold.

The five moves

Name the owner, per merge. One sentence of policy: the engineer who merges a change owns it, regardless of what produced the draft. The model proposes, the engineer signs. No autonomous merges.
Tag provenance at commit time. Commit trailers, PR labels, or tool-native attribution. Record which changes are substantially AI-generated and by which tool.
Harden the gate where it counts. Risk-tier your repos. AI-drafted changes to payment logic, authz, and data migrations get a second reviewer and a required test diff. AI-drafted changes to internal tooling get the standard gate. Uniform strictness is how review becomes theater.
Give non-deterministic code a CI gate. An eval harness is a CI suite for code whose output isn't reproducible. If your agents write code that calls models, evals gate the merge the way unit tests gate deterministic changes.
Write the evidence file as you go. Tool inventory, policy, provenance stats, gate configs, exception log — one folder. When the auditor asks, you hand over a folder instead of a promise.

The objection we hear on move 2: provenance tagging slows engineers down and they'll skip it. In practice the tag is a commit trailer the tooling adds, not a form someone fills in. That objection describes the policy-PDF version of governance, not the tooling version.

What it looks like six weeks in

Composite from rollouts inside client deliveries — fintech scaleup, 30 to 80 engineers, high agent adoption, no provenance, a customer security questionnaire sitting unanswered because nobody could describe the AI share of the SDLC.

Six weeks later: a bot tags model-drafted changes at commit time, so the AI-drafted share of merges is a dashboard number instead of a guess. Repos are risk-tiered. Anything calling a model at runtime runs an eval job in CI. The evidence file exists.

The outcome worth copying is counterintuitive. Review time per diff spikes in week one, then settles at or below baseline — because reviewers stop treating every change as equally suspect. The stalled security questionnaire gets answered from the evidence file in a day. And the tooling debate changes tone, because arguments about impressions become arguments about a dashboard.

None of it required slowing the agents down.

What this is not

It is not a case for treating model-drafted code as suspect by default. Model-drafted code that passed a real review and a real test suite is production code. The vibe-coding failure mode — shipping unread model output — is a review failure, not a tooling failure.

The tools are already in your building and the volume is already in your main branch. The only open question is whether the signature on the merge still means anything. Governance that lives in a document drifts. Governance that lives in the pipeline holds.

Full breakdown — the three postures compared, the rollout timeline, and the legal accountability question: teamvoy.com/blog/ai-generated-code-accountability-2026

Written by Zhanna Yuskevych, CPO at Teamvoy. More engineering writing at teamvoy.com/blog.

Your GenAI Pilot Didn't Fail. Your Data Layer Did.

Teamvoy — Fri, 21 Aug 2026 10:29:44 +0000

MIT found that 95% of enterprise GenAI pilots returned no measurable P&L impact, and only 5% reached production at scale. Almost none of those failures were model failures.

The stalled pilots we get called into look identical from the inside. Someone spent three months on prompts and model selection. Nobody looked at the pipeline feeding it. When we trace a bad output back, it is a brittle query pulling stale or wrong records — the model was working correctly on garbage. Fix the pipeline and the same model suddenly looks smart.

The model is the kernel. The integration layer is the operating system around it, and that is where pilots die. Gartner forecast $644 billion in global GenAI spend for 2025, up 76.4% in a year, against that 95% figure. Big spend is not safe spend.

The demo-to-production cliff

A demo runs on clean, hand-picked data. Production does not. That is the entire gap, and it has three concrete failure modes once an agent gets write-access:

Runaway cost. An agent stuck in a retry loop runs unattended for hours and burns thousands in API spend before anyone wakes up.
Prompt injection. The top agentic risk, and not hypothetical — one arXiv study cut attack success from 73.2% to 8.7%, but only after layering multiple defenses. A single successful injection exfiltrates a secret in minutes.
Quadratic token growth. Cost can scale with the square of context length, not linearly. "Just add more context" gets expensive fast, and past roughly the 40% context-fill mark many models get less accurate anyway. More context is not free, and it can make the system dumber.

None of these are model-quality problems. They are control problems, and NIST's GenAI Profile (AI 600-1) names all of them.

Three questions before anything gets write-access

Ask these of any partner, internal team, or yourself:

Where is the circuit breaker that stops a runaway agent, and who set the spend cap?
Who owns the audit trail when the agent writes to production data?
What requires human approval before the agent executes a sensitive action?

A clean answer to those three tells you more than any benchmark. Build the circuit breaker before you grant write-access, not after the first incident.

Name your pattern

A demo says "we use AI." A production system names its pattern, and each one has a caveat you inherit:

Self-RAG — the model rewrites its query and checks its own retrieval. Good for messy queries. Extra model calls add latency.
Corrective RAG — filters or rejects weak retrieved chunks. Use when wrong context is costly. Needs a tuned relevance scorer.
Adaptive RAG — routes simple and complex queries differently. Routing logic is its own complexity.
GraphRAG — retrieves over a knowledge graph instead of flat text. Strong on relational data. Graph build and upkeep is real, ongoing work.
Agentic orchestration — bounded tool calls with control flow. Untrusted output must never call tools directly.

These stack into retrieve, rank, generate, verify. Latency compounds at every layer, so caching and tight chunking are not optimizations you defer.

How do you know it works, rather than sounds confident?

Evals. This is the phase teams skip and the reason pilots die on real data.

Automation is safe when you can specify the objective and verify the output against it. No verification, no safe automation — you are shipping on vibes. The phased path that survives contact with production data:

Scope one narrow workflow with a measurable outcome.
Prove it on messy production data, not a clean sample, so you find out where it actually fails.
Add evals and guardrails, so failures get caught before users do.
Harden for latency, cost, and edge cases under real load.
Hand over documented, with rollback in place.

Step five matters more than it sounds. A model has no memory of your system between sessions, so the knowledge has to live in documentation and code rather than in one engineer's head. "Done" means it still works in six months.

What it costs, and where the budget actually leaks

Roughly $50K for a proof of concept, $2M+ for a production system at scale with compliance. But the build is rarely what blows the budget — the runtime is. Quadratic token billing and cloud shock, the penalty for running elastic AI infrastructure with a static data-center mindset. Right-size compute before you scale, not after.

The most expensive AI code is not the broken kind. It is the code that almost works — it runs, it looks fine in review, and it writes to the wrong data while the screen looks correct. That is also true of the pilot itself. It demos beautifully and returns nothing, and the reason is never the part everyone spent three months tuning.

Full breakdown — integration patterns, NIST AI RMF mapping, and how to evaluate a delivery partner: teamvoy.com/blog/generative-ai-implementation-services

Written by Taras Voytovych, Founder & CEO at Teamvoy. More engineering writing at teamvoy.com/blog.

Claude Code vs Copilot Is the Wrong Question

Teamvoy — Mon, 17 Aug 2026 12:59:51 +0000

The most expensive mistake in AI tooling procurement right now is treating Claude Code and GitHub Copilot as substitutes and picking one.

We watched a team standardize on autocomplete alone, then hand it a service restructure. The code compiled, read fine in review, and shipped with the wrong invariants — the kind of plausible-looking output you get when a tool designed to finish a line is asked to make a design decision. Nobody was misusing the tool. They just bought one product and assumed it covered both jobs.

They don't overlap the way the comparison posts imply. Copilot is an IDE assistant that helps a human type faster, with an agent layer bolted on top. Claude Code is a CLI agent that reads your repo, edits files, runs your tests, and commits. Different category, different workflow, different budget line.

Where each one actually wins

Per-line autocomplete and IDE chat: Copilot. Years of telemetry-driven tuning and native reach across VS Code, JetBrains, Visual Studio, Neovim, and Xcode. Claude Code's canonical interface is a terminal, not a cursor in a buffer. The trade-off: the autocomplete surface is shallow by design. Good at finishing a line, mid-quality on a function, wrong tool for restructuring a service.
Multi-file refactors and long autonomous tasks: Claude Code. A 1M-token context window, real shell access, and Agent Teams coordinating through a shared task file and git worktrees. We've run multi-service migrations with one instance on API contracts, one on migrations, one on the test suite, all through a shared TASKS.md. The trade-off: agent loops burn tokens. A documented Express.js refactor came in roughly ten times more expensive than a cloud-sandboxed equivalent. On a narrow, well-scoped issue, Copilot's coding agent often gets there for less.
Code quality on hard diffs: Claude Code. In blind review — reviewers shown unlabeled diffs from Claude Code and leading cloud-agent peers — Claude Code's output was preferred 67% of the time. The trade-off: on easy, well-scoped work the gap is small and closing.
Per-seat cost for a broad rollout: Copilot, not close. Roughly five to ten times cheaper at the daily-driver tier. Claude Code's daily driver sits at $100 per seat per month after the April 2026 reset, $200 for parallel agent workflows. The trade-off: Copilot's headline seat price excludes the agent surfaces, which meter separately. A team living in Workspace closes the gap.

Security and audit: this one genuinely splits

Claude Code wins source-code residency. Code stays on the developer's machine, so a CISO at a bank or insurer isn't signing a data-handling agreement to let source leave the network.

Copilot wins day-one governance. Content exclusions, IP indemnity, audit logs in the GitHub admin surface, and the compliance alignment procurement already accepts.

For a NYDFS-regulated bank or a DORA-scoped insurer, the split we deploy: Claude Code on hardened dev environments for source-bearing work with explicit egress controls and audit hooks through MCP servers, Copilot inside GitHub for review, draft PRs, and the long tail of non-source-bearing work.

What doesn't split: prompt injection through code comments, README files, and dependency metadata hits both tools identically. Same mitigations — confidence thresholds, sandboxed test environments, human-in-the-loop gates.

So how do you actually spend the budget?

Split the seats instead of picking a winner.

Copilot for every engineer, on the tier your governance review clears in under a month.
Claude Code Max layered on top for the ~20% who do refactors and platform work.
Both behind one guardrails layer — audit log routing, secrets access, human-in-the-loop gates — so the tool choice stays reversible.

That third point is the one teams skip, and it's the one that determines whether you can change your mind in a year without re-litigating the whole rollout.

The head-to-head framing is comfortable because it produces a decision. It just produces the wrong one. Pick the lane each tool wins, instrument both, and stop pretending an autocomplete engine and an autonomous agent are competing for the same job.

Full category-by-category breakdown, pricing tables, and the decision matrix: teamvoy.com/blog/claude-code-vs-github-copilot-cto-guide

Written by Bohdan Varshchuk, CTO at Teamvoy. More engineering writing at teamvoy.com/blog.

Your CI Agent Should Not Have Write Access Yet

Teamvoy — Mon, 17 Aug 2026 09:43:19 +0000

An agent in your pipeline does not fail the way a script fails. A script fails the same way every time. An agent fails creatively, and then tries again.

We watched one retry the same broken fix eleven times against a failing build. No retry limit, no memory of the previous attempt, so every loop looked like a fresh problem to it. It burned compute for most of an afternoon and left the build exactly as broken as it found it. The pipeline never errored. It just never finished.

That is the thing tech leads underestimate: agentic CI/CD is not automation with better judgment. It is a non-deterministic system with commit rights, and it needs guardrails that traditional CI never needed. Confidence thresholds, sandboxes, retry limits, and observability on the agent's own decisions.

The four failure modes scripts don't have

Looping. Without a retry ceiling and memory of prior attempts, the agent repeats itself indefinitely. Cost and wall-clock time, no progress.
Hallucinated fixes. On an unfamiliar error it will invent a fix — a flag that doesn't exist, a config key from a different version. This breaks the pipeline further, or worse, passes and creates a subtle bug downstream.
Non-determinism. Same input, different action across runs. Reproducibility was the whole basis of trusting CI, and probabilistic agents remove it. Debugging gets much harder.
Prompt injection through your logs. This is the one that should stop you. If the agent reads error logs and has execution permissions, a crafted string in a log line is an instruction. Destructive commands, leaked API keys. Your logs are now untrusted input.

Maturity is genuinely low here. Only a small fraction of agent-driven pipeline changes land successfully today. Treat agents as assistants with a review queue, not as replacements for the engineer who owns the deploy.

What to put in place before the agent can commit

Read access and proposals first. Write access after you have evidence.

A confidence threshold. Above roughly 90%, the agent acts. Between 60 and 70%, the action is flagged and waits for a human. Nothing uncertain executes silently.
A sandbox. Let it try to fix the broken build in full isolation. Even a failed attempt produces logs and debugging context an engineer can use. Nothing it does can touch production data.
Retry limits and prior-attempt memory. The cheapest fix for the most expensive failure mode.
Strict input validation on anything the agent reads. Logs included. Especially logs.
A dashboard on the agent's own behaviour. Error rates, rollback frequency, resource utilisation, and drift in decision patterns. Correlate agent decisions against build events so anomalies surface early instead of in the invoice.

Where agents actually earn their place

The wins are real, and they cluster in the same area: deciding what to run rather than running everything.

Test selection. Analyse the change, run the tests it can actually affect, skip the rest. Shorter feedback loops without dropping coverage where it matters.
Flaky test prediction. Historical failure patterns tell you which tests fail for no reason, which is maintenance work nobody wants to do manually.
Risky commit flagging. Build history and error logs predict failures before code reaches production.
Deploy timing and rollback. Pick the window based on load, auto-trigger rollback on post-deploy signals.
Incident triage. Root-cause insight and a suggested corrective action before a human opens the dashboard.

Continuous learning is the headline benefit, but it has a precondition: the agent can only improve if the evaluation harness and observability exist to teach it. That instrumentation is the part most teams skip, and they find out they skipped it when the agent makes a decision nobody can explain.

How do you scope the first agent?

In this order, and skipping either of the first two is how you ship something that works perfectly and changes nothing:

Name the business outcome — release frequency, change failure rate, mean time to recover. Not "adopt AI."
Identify the specific bottleneck, and state how an agent removes it.
Sequence the work so the first agent ships against one measurable bottleneck, not five.
Track the KPI you named in step 1.
Feed what you learn back into scope and guardrails.

The rule we hold: no agent gets write access to a repo before it has spent a sprint proposing changes that a human approved or rejected. The rejection rate is the only honest readiness signal you get.

None of this shows up in a demo, where the agent fixes the broken build on the first try and everyone claps. It shows up on the afternoon it tries the same fix eleven times, or the afternoon a log line convinces it to do something nobody asked for. Guardrails are not the exciting part of agentic CI/CD. They are the part that decides whether you keep the agent.

Full playbook — architecture, benefits, pitfalls, and the guardrail set in detail: teamvoy.com/blog/building-ai-agents-into-your-ci-cd-pipeline

Written by Alyona Kakora, Project Manager at Teamvoy. More engineering writing at teamvoy.com/blog.

Your Rails App Isn't Too Old for AI. It's Too Untested.

Teamvoy — Thu, 06 Aug 2026 14:25:16 +0000

When an AI feature fails to land on a legacy Rails app, it is almost never the model. It is that nobody could safely change the code underneath it.

We picked up an app running Rails 5.2 with 11% test coverage and a Sidekiq queue nobody had looked at since 2021. The team had already built an LLM classifier. It worked on a laptop and had been sitting in a branch for five months, because merging it meant touching a controller that three people were afraid of.

That is the whole argument: modernization is not a rewrite and it is not a version bump. It is the sequence that makes a legacy Rails app changeable again, and then puts AI in the places where the numbers move. Skip the sequence and you get a prototype that lives in a branch forever.

The sequence, and why you don't get to reorder it

Audit, upgrade, test, refactor, then integrate. Every step exists because the next one is unsafe without it.

Audit. Map dependencies, security posture, and tech debt against business priorities. You are deciding what not to touch as much as what to fix.
Upgrade, one major at a time. 4.2 → 5.0 → 5.2 → 6.0 → 6.1 → 7.0 → 7.2 → 8.0. Resolve deprecations at each stop and enable new defaults gradually through config.load_defaults. Skipping a version to save a sprint buys you a month of breaking changes.
Test. Unit, integration, and system coverage, running on every push in CI. This is not hygiene. It is the precondition for everything after it.
Refactor. Extract business logic into service objects and concerns. This is where the seams for AI actually get cut.
Integrate. Only now.

Incremental work also wins on cost - rebuilds have been measured at up to three times the feature development cost of upgrading in place. And there are still over 560,000 live Rails sites, so the population of teams making this decision is not small.

What the AI layer actually looks like

Four components, and none of them are optional if the feature is going to survive a production week.
A provider client. Wrap OpenAI, Anthropic, Hugging Face, or your self-hosted model behind one abstraction. Swap vendors without touching business logic. Swap in a mock during CI so tests are fast and deterministic.
A service object. All AI logic lives here. Testable, replaceable, out of the controller.
A background job. Sidekiq or equivalent. Model calls never sit in the request path. A recommendation engine can chew through user data without adding a millisecond to page load.
**An audit table. **Inputs, outputs, and confidence scores in JSONB on Postgres. When a classification is wrong, you pull the exact pair, fix the prompt, and version the change.

The trade-off nobody puts in the proposal: that audit table grows fast and someone has to own retention on it.

How do you keep the AI layer from quietly costing you money?

Set confidence thresholds before launch, not after the first invoice. When confidence drops below the line, fall back to deterministic logic, a cached prior answer, or a human review queue. Enforce a per-tenant daily cost cap at the job layer, where you can actually stop the work.
Then treat prompts as versioned, testable code and mock model responses in the suite. An untested prompt change is a production change with no rollback story.

When to bring someone in

In-house works when the team ships weekly, has bandwidth for a 20% side program, and the AI use case is scoped tightly. Bring in help when:

Rails is two or more majors behind and no engineer on the team has run an upgrade this decade.
A live prototype never got past staging because nobody owns operations for it.
A board or regulator has set a date, and missing it has a number attached.
The one senior Rails engineer who understood the plan just gave notice.

What to push back on in a vendor quote: any multi-month discovery phase that ends in a slide deck, any proposal naming juniors on delivery, any pricing without a locked scope.
None of this shows up in a demo. It shows up five months later, when the classifier still works perfectly on someone's laptop and the branch has 400 commits of drift behind main. The upgrade sequence is not the interesting part of adding AI to Rails. It is the part that decides whether the AI ever ships.

Full breakdown of the assessment, upgrade path, and integration patterns: teamvoy.com/blog/how-to-transition-legacy-ruby-on-rails-apps-to-ai-enabled-architectures

Written by Bohdan Varshchuk, CTO at Teamvoy. More engineering writing at teamvoy.com/blog