DEV Community: Samay Thakkar

AIOps for IT Operations: What It Is

Samay Thakkar — Thu, 12 Nov 2026 14:35:52 +0000

AIOps for IT operations means applying machine learning to your operational data (metrics, logs, traces, alerts and change records) to group related alerts, flag unusual behaviour and suggest likely causes, so on-call engineers can triage incidents faster. It doesn't replace monitoring or SRE practice. It can only be as good as your SLOs, alerts and telemetry.

What does AIOps mean for IT operations?

AIOps is a set of machine learning techniques that sit between your monitoring tools and your on-call engineers. They turn a flood of signals into a short list of incidents with context attached.

Gartner, which coined the term, says AIOps "combines big data and machine learning to automate IT operations processes, including event correlation, anomaly detection, and causality determination". In day-to-day work, that comes down to four jobs:

Deduplication: collapsing 200 copies of the same alert into one.
Correlation: recognising that a database latency alert, an API error alert and a queue backlog alert are all one incident.
Anomaly detection: spotting behaviour that breaks a learned pattern, such as traffic that's normal on a Monday but strange on a Sunday.
Cause suggestion: pointing at the likely trigger, often a deploy, config change or failing dependency, and sending the incident to the team that owns it.

Some platforms add a fifth job, which is running a runbook step automatically. Treat that as a separate decision with its own controls.

How signals reach an on-call engineer through an AIOps layer, and how outcomes feed back

The last arrow matters most. If engineers never record whether a grouping was correct or what actually caused the incident, the system has nothing to learn from.

Which problems does AIOps solve and which does it not solve?

AIOps helps when you have too many alerts. It can't help when the alerts you need don't exist, or when nobody owns what they point at.

The noise problem is real. In a 2026 survey of IT production teams reported by Carrier Management:

77% of on-call teams get at least ten alerts a day.
57% say fewer than 30% of those alerts are actionable.
83% of engineers ignore or dismiss alerts at least occasionally.
44% had an outage in the past year linked to suppressed or ignored alerts.
78% had at least one incident where no alert fired at all.

Splunk's State of Observability 2025 report, which surveyed 1,855 ITOps and engineering professionals, found that 13% of respondents globally admit to ignoring or suppressing alerts.

Alert fatigue figures from a 2026 survey of IT production teams (Carrier Management)

Look closely at those numbers. AIOps can work on the first four, because grouping and ranking cut how many pages engineers get and how often they ignore one. It does nothing about the fifth. If no alert fired, there was no signal to correlate. That's an instrumentation gap, and you close it with better telemetry and symptom-based alerts, not a smarter model.

What AIOps handles well:

Alert storms during a cascading failure
Linking incidents to recent deploys or config changes
Routing to the right team when ownership data is accurate
Seasonal patterns that fixed thresholds handle badly

What it doesn't fix:

Services with no instrumentation
Alerts on causes (CPU at 80%) rather than user impact
Services with no clear owner
Missing or outdated runbooks
Unreliable releases that cause the incidents to begin with

Not every AIOps feature carries the same risk. Grouping duplicate alerts is cheap to get wrong, because an engineer can still see the underlying alerts. Automatically restarting services or closing tickets is expensive to get wrong. Start in the top-left of this chart.

AIOps uses placed by value and by the cost of a wrong answer

What data does AIOps need before it can help?

AIOps needs consistent telemetry, change events, an accurate list of services and owners, and labelled incident history. Without all four, correlation is just guessing.

Telemetry signals. OpenTelemetry currently supports traces, metrics, logs and baggage, with events and profiles still in development or at the proposal stage. A vendor-neutral standard lets you change tools later without instrumenting everything again.
Consistent labels. Every alert needs the same service name, environment, severity and owning team. If one tool calls a service "checkout" and another calls it "checkout-svc-prod", correlation fails before any model runs.
Change events. Deploys, feature-flag flips, config pushes and infrastructure changes, all with timestamps. Most incidents begin with a change, so this is often the most useful data you can give AIOps.
Topology and ownership. Which service calls which, and who owns each one. This drives both correlation and routing.
Incident history with outcomes. Past incidents tagged with the real cause and with which alerts were noise. That's the ground truth you'll use to judge any tool.

Keep enough history to cover your normal cycles. If traffic follows a weekly pattern, a model trained on four days of data will flag every Monday as an anomaly.

Why should SLOs and clean alerts come before AIOps tools?

SLOs tell you which alerts matter to users. If you feed AIOps alerts without that, it learns to rank noise.

Google's SRE guidance is direct about this. The Monitoring Distributed Systems chapter says effective alerting has "good signal and very low noise". It recommends putting far more effort into catching symptoms than causes, and says alerting configuration that's rarely exercised (less than once a quarter for some teams) should be up for removal.

The SRE Workbook chapter on alerting on SLOs shows how to put this into practice. Alert on how fast you're using up your error budget (the burn rate), not on raw thresholds. With a 99.9% SLO over 30 days, a constant 0.1% error rate uses up the budget exactly by the end of the window, which is a burn rate of 1. Spending 5% of the budget in a single hour takes a burn rate of 36. The workbook calls multiwindow, multi-burn-rate alerts the most appropriate approach in most cases.

Do this before buying anything:

Define one or two SLOs per critical service, in terms users would recognise (successful checkouts, search latency).
Replace cause-based pages with burn-rate alerts. Turn cause alerts into dashboards or tickets.
Delete alerts that haven't fired, or haven't led to any action, in a quarter.
Make sure every remaining alert has an owner and a runbook link.

This often cuts paging by more than any tool would. It also gives you a clean definition of "an incident that mattered", which you'll need to judge an AIOps pilot. This is where our Observability & SRE work starts: SLOs users would recognise, OpenTelemetry instrumentation, and cutting alerts down to the ones that matter.

How do you pilot AIOps on one service with a measured baseline?

Pick one noisy service with a clear owner. Measure its on-call load for a few weeks, clean up its alerts, then run AIOps in suggest-only mode and compare against the baseline.

Pick the service. Choose one that pages often, has a single owning team and already sends telemetry. Avoid your most critical system for the first round.

Record a baseline. Tag every page for two weeks:

Total alerts and total pages per week
Share of pages that were actionable
Time to acknowledge and time to restore
Incidents found by users or other teams before any alert fired

Clean up first. Add the SLO and burn-rate alerts and remove dead ones. Measure again. Some of the improvement will come from this step alone, and you should credit it there, not to the tool.

Run in shadow mode. Let the AIOps tool group incidents and suggest causes, but not act. Engineers mark each suggestion right or wrong.

Decide with criteria set in advance. For example: pages per week down by an agreed share, no drop in actionable rate, no missed SLO breaches, and cause suggestions correct often enough that engineers check them first. Agree a stop rule too. Our guide to why AI automation projects fail covers the usual ways pilots without a baseline or owner stall.

A nine-week plan to pilot AIOps on one service against a measured baseline

Read recovery time together with change failure rate, not on its own. Our post on DORA metrics when AI writes your code explains why one metric improving can hide another getting worse.

What should you check when a vendor sells AIOps?

Ask the vendor to prove the claims on your own past incidents, show why it grouped what it grouped, and keep a person in charge of any action it takes.

Check	Good answer	Red flag
Proof on your data	Replays your last 90 days of incidents and shows grouping accuracy	Only shows benchmarks from other customers
Explainability	Shows which signals and changes linked alerts together	"The model decided"
Noise reduction	Separates deduplication from suppression and reports both	One headline percentage
Data standards	Takes in OpenTelemetry and exports your data and rules	Proprietary agents only, no export
Actions	Suggest-only by default; approval and audit trail for risky steps	Auto-remediation switched on out of the box
Data terms	Clear on retention and whether your telemetry trains shared models	Vague or missing answers
Cost	Pricing tied to a unit you can forecast (hosts, GB, events)	Costs that grow faster than your traffic

One test is worth more than the rest. Hand the vendor three past incidents you already understand and see whether the tool reaches the same conclusion. If it can't explain incidents you've already solved, it won't help with the ones you haven't.

Conclusion

AIOps is useful for one specific job: cutting a stream of alerts down to a few incidents with likely causes attached. It works best on top of good telemetry, SLO-based alerts, clear ownership and labelled incident history, and it can't stand in for any of them. Do the SRE groundwork first, then pilot one service against a baseline and let the numbers decide.

At Calidad, DevOps and AIOps are part of our Platform Modernization work. We start with a paid pilot: one pod, one outcome, and success criteria agreed up front.

Talk to an engineering lead →

Frequently asked questions

Is AIOps the same as observability?

No. Observability is your ability to understand a system from its telemetry: traces, metrics and logs. AIOps is an analysis layer that runs on that telemetry to group alerts, find anomalies and suggest causes. Without observability, AIOps has nothing reliable to work with.

How long should an AIOps pilot take?

Plan for about two months on one service: two weeks of baseline, two weeks of alert cleanup and SLO setup, then four weeks in shadow mode. Weekly traffic patterns mean shorter pilots tend to produce false anomalies. Set success criteria and a stop rule before the pilot starts.

Should AIOps be allowed to fix incidents automatically?

Not at first. Start with suggest-only grouping and cause hints, where a wrong answer costs little. Add automated actions one runbook step at a time, with an approval step, scoped permissions and an audit trail for anything that's hard to undo.

Can we get AIOps benefits without buying a platform?

Often, yes. A lot of the noise reduction comes from replacing cause-based alerts with SLO burn-rate alerts, deleting alerts nobody acts on, and adding consistent service and owner labels. Do that first. It shows whether a tool would add enough on top to justify its cost.

Does a GCC in India need its own AIOps setup?

Usually not. A capability center works best on the same telemetry, SLOs, alert routing and runbooks as headquarters, so incidents are handled the same way across time zones. Separate tooling tends to split ownership and makes correlation harder.

This article first appeared on Calidad Infotech.

Recovering LLM Costs in SaaS Pricing

Samay Thakkar — Thu, 12 Nov 2026 14:35:51 +0000

Yes, you can recover LLM token costs from SaaS customers. First you need to measure those costs for each tenant and each feature. Most SaaS teams do best with a hybrid model: a subscription that includes an AI allowance sized for typical use, plus metered overage or higher tiers for heavy users. Caps on top of that protect your margin.

Why do LLM costs break a flat SaaS price?

Every AI request costs you money each time it runs, while a flat price stays the same however much a customer uses.

In classic SaaS, serving one more user costs almost nothing. With LLM features, every prompt, retrieved document and generated answer is billed by the token. The bill also varies a lot by model. CloudZero's breakdown of OpenAI's rate card lists the current flagship at $10 input and $50 output per million tokens, while the cheapest model in the same table costs $0.20 for input. Output tokens cost several times more than input, so long answers and agent loops add up quickly.

Here's a worked example using those flagship rates. Say a feature sends 3,000 input tokens and gets 800 output tokens back on each call:

Input: 3,000 × $10 / 1M = $0.03
Output: 800 × $50 / 1M = $0.04
Cost per call: $0.07

A user who runs it 50 times a working day makes about 1,050 calls a month and costs you about $73. If the seat sells for $40, every heavy user loses you money, and light users can't make up the gap if heavy users are common.

Many vendors have moved away from flat pricing for this reason. Metronome's field report found the move to usage-based or hybrid pricing was often a defensive move to deal with real, variable costs rather than a strategy.

How do you measure token usage per tenant and per feature?

Send every model call through one gateway. The gateway tags each call with the tenant, the feature and the model, and records the token counts the provider returns.

Set it up in this order:

One path to the model. Product code shouldn't call provider SDKs directly. Use a single gateway or wrapper, so nothing goes unmetered.
Tag every request. Add tenant ID, plan, feature, model and user. Without the feature tag, you can't tell which feature is losing money.
Record each token type separately. OpenTelemetry's generative AI conventions define separate counters for input, output, cache-read, cache-write and reasoning tokens. The recorded value should match the count the provider returns, not your own estimate.
Price each call at the time it runs. Multiply each token type by the rate card in force on that date and write the result to a cost ledger. Rates change, so totals recalculated later at new rates will be wrong.
Add the costs that aren't tokens. Count embeddings, vector search, retries and failed calls. Retries are easy to miss and can be a real share of spend.
Look at the spread, not the average. Report cost per tenant at the median and the 95th percentile, and cost per completed task for each feature.

Collect at least one full billing cycle of this data before you change any prices. The spread decides your pricing model more than the average does. We cover the setup in more detail in our guide to cutting cloud and LLM costs at a SaaS company.

Each request is tagged with its tenant and feature before its cost reaches the ledger.

Which pricing models can recover LLM costs: quotas, metering or tiers?

All three can recover costs. Which one fits depends on how uneven usage is across tenants and how much your buyers need predictable bills.

Hybrid models are now the most common choice. Metronome reports that most enterprise AI deals in 2025 still rely on usage-based or hybrid pricing. Orb, citing Stripe, says that among AI company leaders, 56% use hybrid pricing and 38% use pure usage-based pricing.

Model	How it works	Recovers cost when	Main risk
Absorbed (built into the price)	AI is part of the plan price	Usage is light and even	Heavy users erode margin
Included quota + overage	A set allowance, then a charge per unit	Usage has a long tail	Bill surprises if alerts are weak
Pure metering	A charge per request, task or token	Usage is driven by the customer and is spiky	Buyers can't budget for it
Tiered plans	Higher tiers raise AI limits	Usage tracks customer size	Customers bunch up just under a tier limit
Prepaid credits	Customers buy credits up front and spend them down	Features vary in cost	Credits feel abstract to buyers
Outcome-based	A charge per resolved task	The outcome can be measured cleanly	Arguments over what counts as done

Several large vendors already use these models. GitHub Copilot business plans include a set number of premium requests, and an admin policy decides whether to allow charges beyond that or block usage. Intercom sells annual credit buckets with optional pricing per resolution.

Charge business buyers in a unit they recognize, such as a request, a document, a report or a resolution. Use tokens only to set that price internally. That way, when you switch to a cheaper model, your margin improves and the customer's price list doesn't change.

The chart places each model on two criteria. One is how predictable the bill is for the buyer. The other is how well the model protects your margin when usage spikes.

Pricing models placed by buyer predictability and margin protection

How do you set token budgets and caps that protect margin?

Start from the gross margin you want on each plan, work out how much AI cost each tenant can carry, and enforce that budget in the gateway.

Set the AI share of revenue for each plan. For example, a $500-a-month plan that may spend 15% on AI has a $75 budget per tenant.
Convert the budget into units. At $0.07 a call, $75 covers about 1,070 calls. Set the included quota a bit lower, say 1,000, to leave room for retries and fallbacks.
Add thresholds. Warn the customer's admin at 80% of the quota. At 100%, either move the tenant to a cheaper model or ask them to approve overage. The hard cap is whatever the contract allows.
Budget expensive features separately. Long-context analysis and multi-step agents can use up a whole quota in a few runs. Give them their own limits.
Cut costs before you raise prices. Under Anthropic's pricing, cache reads cost 0.1x the base input rate, a 90% discount on repeated system prompts. Sending simple requests to smaller models lowers costs further. In our generative AI builds, each feature gets latency and cost targets per request, with model routing and fallbacks.
Recheck every month. Provider prices change, so recalculate how many units the budget covers whenever the rate card moves.

Lifecycle of one tenant's AI budget within a billing period

How should you explain usage charges to customers without losing trust?

Show customers their usage before you bill it, let them set their own limits, and charge in units they understand.

Surprise bills are common. Orb cites a survey in which 78% of IT leaders reported unexpected charges from consumption-based or AI pricing. To avoid adding to that:

Put a usage meter in the product that shows units used, units left and the date the allowance resets.
Alert admins, not only users, at 80% and 100% of the quota.
Let customers choose what happens at the limit. Like GitHub's overage policy, admins should decide whether usage continues at a charge or stops.
Offer a trial period so buyers can estimate their usage before they commit. Intercom uses generous or extendable free AI trials for this.
Don't charge for your own failures. Leave retries, timeouts and answers that failed validation off the bill.
Give notice before changes take effect. Let existing contracts run to renewal on their current terms, and publish the new units well in advance.

When should you absorb LLM costs versus pass them on?

Absorb LLM costs when the cost per tenant is small and stable compared with the price. Pass them on when usage is uneven, heavy or controlled by the customer.

Absorbing tends to fit when:

Even your 95th-percentile tenant stays within the AI budget for the plan.
The feature is now something buyers expect, and competitors include it at no extra charge.
Use is limited by how the product works, such as one summary per record.

Passing costs on tends to fit when:

A small group of tenants causes most of the spend.
Customers trigger the volume themselves, such as with bulk jobs, agents or API access.
The feature is a new premium capability that buyers will pay more for.

Deciding whether to absorb LLM costs or pass them on

Conclusion

Price LLM costs only after you can see them. Meter every call by tenant and feature, then look at how unevenly usage is spread. Most SaaS companies end up with a hybrid: an included allowance, then overage or tiers, budgets enforced in the gateway, and usage that customers can see before they're billed.

Calidad's cloud cost optimization work covers this for teams whose cloud and AI bills grow faster than revenue. We show where the money goes, cut the waste and set guardrails so the savings last. Every engagement starts with a paid pilot, and there are no long-term contracts.

Talk to an engineering lead →

Frequently asked questions

Should I show customers raw token counts on their invoice?

Usually not for business buyers. Tokens are hard to predict and depend on which model you use. Bill in a unit customers recognize, such as requests, documents or resolved tasks, and set its price from tokens internally. Developer-facing API products are the exception, because their buyers often expect to see tokens.

How much markup should I add on top of LLM costs?

No single markup is right. Start from the gross margin you want on each plan, and allow for retries, embeddings, infrastructure and support as well as tokens. Check what the price of your chosen unit looks like to the buyer against the value they get from it, not only against your cost.

What should I do when model prices drop?

If you charge in your own units rather than tokens, a cheaper model raises your margin without any change to the price list. Decide whether to keep the savings, add more included usage, or lower prices for competitive reasons. Recheck how many units each budget covers whenever a rate card changes.

Can I add AI usage charges to existing customer contracts?

Doing it mid-term risks breaking trust and the contract itself. A safer approach is to keep current terms until renewal, give clear notice and show customers their actual usage beforehand. That way the new pricing reflects usage they've already seen.

How long should I meter usage before choosing a pricing model?

Collect at least one full billing cycle of data tagged by tenant and feature, and longer if usage changes with the season. You need enough data to see the 95th-percentile tenant, not only the average. Pricing on a short sample tends to set quotas too generously or too tightly.

This article first appeared on Calidad Infotech.

LangGraph vs CrewAI vs AutoGen Production

Samay Thakkar — Thu, 12 Nov 2026 08:26:22 +0000

The right choice of LangGraph, CrewAI or AutoGen for production depends on your workflow, not on a ranking. Pick the one that passes your own tests for saved state, approval steps, tool permissions and tracing. Note that Microsoft has stopped adding features to AutoGen, so today the real comparison is LangGraph vs CrewAI vs Microsoft Agent Framework. Run all three through the same tasks before you commit.

This guide is for CTOs and lead engineers whose agent workflow needs an audit trail and human sign-off. It gives you the criteria, a way to test each framework fairly, and a check for when you don't need a framework at all.

What should a production agent framework need to support?

A production agent framework has to save its state, pause safely for a person, limit what each tool can do, and leave a trace you can audit.

A demo only has to work once. Production has to keep working through restarts, timeouts, rejected approvals and misleading tool outputs. Check every candidate against these six things:

Saved state. A run can survive a process restart and pick up from the last completed step.
Explicit control. You can see and change the path a run takes, including branches and loops.
Approval steps. A run can wait hours or days for a person, then continue with their decision.
Tool permissions. Each step can reach only the tools and data it needs.
Tracing. Every model call, tool call and decision shows up as structured telemetry.
Exit cost. You can leave without rewriting your business logic.

Tool permissions come before features. In OWASP's 2026 ranking of LLM application risks, Excessive Agency rose from sixth to third. OWASP traces it to "excessive functionality, overly broad permissions, or too much autonomy." A framework that makes broad access easy is a risk, however good its demos look.

How do LangGraph, CrewAI and AutoGen differ in state, control and tooling?

LangGraph gives you a graph you control directly, with saved state at each step. CrewAI adds Flows for control and approval on top of its agent crews. AutoGen is frozen, and new projects go to Microsoft Agent Framework instead.

LangGraph. A checkpointer saves the graph's state at each step, grouped into threads. Approval uses interrupt(): "When an interrupt is triggered, LangGraph saves the graph state using its persistence layer and waits indefinitely until you resume execution." The docs say to use a durable checkpointer in production. You also choose a durability mode, and the trade-off matters. In "async" mode, there is a small risk that a checkpoint isn't written if the process crashes.

CrewAI. Flows are the part of CrewAI that matters for control. CrewAI's @human_feedback decorator can pause a flow, show a person the output, collect their feedback, and route the flow based on the answer. Check the defaults: by default, @human_feedback stops and waits for input at the console. A non-blocking async mode exists. CrewAI's paid platform adds email-based review, but those features need version 1.8.0 or later and apply only to Flows, not to Crews.

AutoGen. Microsoft no longer develops AutoGen. The repository has shown a maintenance-mode notice since 2 October 2025, the last Microsoft release was autogen-agentchat 0.7.5 on 30 September 2025, and new users are sent to Microsoft Agent Framework, which reached 1.0 on PyPI on 2 April 2026. In Agent Framework, approval works by request and response: a workflow step sends a request to a person or system and waits for the answer before going on.

Criterion	LangGraph	CrewAI	AutoGen → Agent Framework
Control model	Graph you define (nodes, edges)	Agent crews, plus Flows for step-by-step control	Group chats (AutoGen); workflows (Agent Framework)
Saved state	Checkpointers, with a choice of durability mode	Test how Flow state survives a restart	Checkpointing in workflows
Approval step	`interrupt()` and resume	`@human_feedback` in Flows	Request and response
Project status	Actively developed	Actively developed	AutoGen frozen; build on Agent Framework

Take every cell as something to test, not a verdict. Each feature exists, but you won't know how it behaves in your setup until you run it.

How do you test each framework against the same task set?

Build 10–20 tasks from your real workflow. Run them on every framework with the same model, prompts and tools. Then score the results against criteria you agreed before the first run.

Keep the framework as the only thing that changes. Put your tools behind one shared interface. An MCP server works well for this, because each framework then calls the same tool code. Pin the model version too, the same way you would when comparing LLMs.

Your task set should include runs that go wrong on purpose:

Normal run: the standard case finishes correctly.
Approval rejected: the run stops cleanly and records why.
Approval after 48 hours: the run resumes with the right state, on a different worker.
Process killed mid-run: the run resumes from its checkpoint and doesn't repeat finished side effects.
Tool error: the run retries a set number of times, then hands off to a person.
Injected instruction: a tool result tells the agent to call an off-limits tool, and the call is blocked.

The worker test is a real risk, not a theory. In a GitHub discussion from November 2025, an Agent Framework user found that with several approval rounds, holding checkpoint state inside the framework, with no option to store it in a distributed cache, looked like a hard limit. That may have changed since. Your multi-worker test is how you find out.

The states one agent run must pass through in testing, including crash recovery and rejection

Score each framework on:

task success rate
correct resumes
approval bypasses (must be zero)
how complete the traces are
how much connecting code you had to write
how long it took to add one new step

The last two show what owning the framework will cost you over time.

Which criteria matter most for audit trails and human approval steps?

The criteria that matter most are where approval is enforced, whether it is tied to an exact action, and whether the record lives in a store you control.

Enforce approval in the tool layer. A prompt that says "ask before refunding" is not a control. The tool gateway should refuse a risky call unless it comes with a valid approval.
Tie approval to the exact action. Store a hash of the tool name and its arguments with each approval. If the agent changes the amount after approval, the call fails.
Record who, what, when and why. Log the approver, the timestamp, the evidence they saw and the decision. Write it to an append-only store, not just to the framework's checkpoint tables.
Make side effects safe to repeat. Find out whether resuming re-runs any code before the approval point. Use an idempotency key so a resumed run can't send a payment twice.
Use standard trace names. The OpenTelemetry generative AI conventions define names for model calls, tool calls and agent operations (agent spans). Use them so your traces don't depend on one framework's dashboard.

How a risky tool call is paused, approved, checked and logged before it runs

Calidad builds AI agents and automation this way: scoped permissions, a full audit trail, and a person on every risky action. Checking what the agent says before users see it is a separate job, covered in our guide to validating AI responses in production.

What are the risks of vendor lock-in and how do you limit them?

The main lock-in risks are state formats tied to one framework, features available only on a paid platform, and the project being abandoned. You limit them by keeping business logic, tools and audit records outside the framework.

AutoGen shows that being abandoned is a real risk, not a theory. A framework with strong backing and a large user base still went into maintenance mode within two years. Plan for leaving on day one:

Keep business logic in plain modules. Graph nodes or flow steps should only call your own functions.
Keep tools behind your own interface or MCP, so switching frameworks doesn't mean rewriting the tools.
Own the audit log. Checkpoint tables are internal to the framework. Your compliance record shouldn't be.
Know which features need a paid platform. Email approvals or a hosted runtime may be worth paying for. Just write down which parts of your design depend on them.
Keep the task set. It doubles as your migration test. A new framework is ready when it passes the same tasks.

How do you decide between a framework and custom orchestration code?

Use a framework when the workflow branches, loops or pauses for long periods. Use plain code when it is a short, fixed sequence with one approval step.

A three-step pipeline with one approval can run on a job queue, a database row for state and a small approval service. That's less to learn and less to upgrade. Frameworks earn their keep when you have many branches, retries, long waits and several agents, because then you would end up rebuilding checkpointing and resuming yourself.

Deciding between an agent framework and custom orchestration for one workflow

Conclusion

No framework wins in general. Write down the six criteria, build a task set from your real workflow, and run LangGraph, CrewAI and Microsoft Agent Framework through it with the model and tools held fixed. Choose based on zero approval bypasses, correct resumes and complete traces. Then keep your tools, logic and audit log outside the framework so you can change your mind later.

Calidad runs this kind of comparison as a paid pilot: one pod, one outcome, with success criteria agreed up front. Talk to an engineering lead →

Frequently asked questions

Is AutoGen still safe to use in production?

Existing AutoGen systems can keep running, but Microsoft is no longer adding features, and new users are directed to Microsoft Agent Framework. For new production work, test Agent Framework instead. If you already run AutoGen, plan a migration and use your task set to prove the new version behaves the same.

Can we combine frameworks, for example LangGraph for control and CrewAI for agents?

Yes, but every extra framework adds upgrades to track and another place where state can live. If you combine them, have one of them own the saved state and approvals, and keep tools behind a shared interface so neither framework controls access directly.

How long should a framework comparison take?

Long enough to run every task in your set, including crash and delayed-approval cases, on each framework. Keep the scope to one real workflow, not a platform-wide review. A narrow comparison with agreed pass/fail criteria tells you more than a broad one.

Does the framework handle agent security for us?

No. Frameworks give you building blocks such as approval pauses and checkpoints. Permissions, approval checks tied to exact actions and audit records are your design decisions. Enforce them in the tool layer and test them with injected instructions.

Which LLM should we use when comparing frameworks?

Use the model you expect to run in production, and pin its version so it is the same across all frameworks. If you haven't chosen a model yet, run a separate model evaluation first. Otherwise you can't tell whether a difference in results comes from the framework or the model.

This article first appeared on Calidad Infotech.

CI/CD Security Gates for AI-Written Code

Samay Thakkar — Thu, 12 Nov 2026 08:26:22 +0000

AI-written code needs four security gates before it merges: secrets scanning, static analysis (SAST) on the changed code, dependency scanning for new packages and a required senior review. Run all four on every pull request. Block the merge only when a finding is both high-confidence and high-severity, warn on everything else, and keep every result as release evidence.

What security gates does AI-written code need before merge?

It needs the same gates as code a person wrote, but they have to be enforced, not left optional. AI coding assistants write code that compiles and looks clean, yet Veracode's 2026 GenAI Code Security Report found that about 44% of AI code generation tasks introduced a risky vulnerability. The average security pass rate across models was 56%, almost unchanged from the first report. A later update covered in the trade press still found vulnerabilities in 38% of tasks.

The risk is not the same in every language. In the same Veracode report, Java code passed security tests far less often than Python, JavaScript or C#.

Security pass rate of AI-generated code by language, Veracode 2026 GenAI Code Security Report

Secrets are the second problem. In a GitGuardian sample of about 20,000 repositories where Copilot was active, 6.4% leaked at least one secret. The minimum gate set is:

Secrets scanning, before the code reaches the remote and again in CI.
SAST on the diff, so injection, weak crypto and unsafe deserialisation are caught in the lines the assistant wrote.
Dependency scanning (SCA) for every package added or upgraded, including a check that the package exists in your approved registry.
Required human review from a code owner who understands the module.

Our Quality Engineering work follows the same rule: tests, security checks and senior review on every change, whether a person or an AI wrote it.

Which checks belong on every pull request and which only on release?

Run fast, diff-scoped checks on every pull request. Save the slow checks that need the whole system for the release pipeline or a nightly run.

A pull request gate should finish while the author still has the context. A release gate can take longer, because it looks at the built artefact and the running system.

Check	Pull request	Release	Default action
Secrets scan (pre-commit + CI)	Yes	Full history scan	Block on verified secret
SAST on changed files	Yes	Full repo scan	Block new critical/high
Dependency scan (new/changed packages)	Yes	Full SBOM scan	Block critical CVE with a fix available
Infrastructure-as-code scan	Yes, if IaC changed	Yes	Block policy violations
Required code owner review	Yes	—	Block until approved
Container image scan	—	Yes	Block critical with a fix available
DAST against staging	—	Yes	Warn, triage before release
Licence compliance	Warn	Yes	Route to owner
SBOM generation	—	Yes	Store as evidence

If your pipeline also deploys infrastructure, our guide to infrastructure as code security in CI/CD covers scanning plans on each pull request.

How do you configure SAST, dependency and secrets scanning in CI/CD?

Run each scanner as its own named CI job. Each one scans only what changed, writes results in a standard format and reports a pass or fail status that the branch rules can check.

The pull request flow, from the engineer's commit to the merge:

How a change written with an AI assistant passes the hook, CI scanners and code owner before merge

SAST setup

Run the scanner in diff-aware mode on pull requests and report only new findings. Pre-existing findings go into a baseline and a backlog, so authors aren't blocked for code they didn't touch.
Output SARIF so findings appear inline on the pull request.
Add rules for the patterns assistants tend to repeat in your codebase, such as string-built SQL queries or disabled TLS checks.

Dependency setup

Scan the lockfile diff, not just the manifest.
Fail a new dependency that isn't on your internal registry or allow-list. Assistants can suggest packages that don't exist or that nobody has vetted.
Pin versions and let an update bot open the upgrade pull requests, so each upgrade goes through the same gates.

Secrets setup

Add a pre-commit hook on developer machines. Then turn on server-side protection: GitHub's push protection blocks pushes that contain secrets and raises an alert whenever someone bypasses the block.
Scan the full history again in CI on release branches.
When a secret is found, rotate it first. Removing the line doesn't make a leaked key safe.

Name every job uniquely. GitHub warns that reusing job names across workflows can produce ambiguous status results and block merges.

How do you enforce policy so a failing scan blocks the merge?

Make the scanner jobs required status checks on the protected branch, so that the platform blocks the merge, not the author.

On GitHub, rulesets can require status checks to pass before merging. A code scanning rule can also block a pull request when a required tool finds an alert at or above a severity you set. The same page lists a rule that requires secret scanning alerts to be resolved, which is currently in public preview. Other platforms have equivalent merge checks. Turn on "require a pull request", "require code owner review" and "require branches to be up to date" alongside them.

Then decide, for each finding, whether to block or warn:

Decision path that turns a scan finding on a pull request into block, warn or a logged exception

Three rules keep the gate credible:

Block on high confidence only. A gate that fails on noise teaches engineers to bypass it.
Nobody bypasses alone. Admin overrides should need a second person, and every override is recorded.
Treat AI reviewers as advisory. An AI review bot is useful as an extra pair of eyes, but it shouldn't count as the required approval. We explain why in what AI code review catches and misses.

Who owns exceptions and how are they logged as release evidence?

The code owner of the affected module and a named security lead own each exception together. Every exception is time-boxed and recorded in the same place as the scan results.

An exception record should include:

the finding ID and severity
why it can't be fixed now
the compensating control, such as a WAF rule or a feature flag that's turned off
the two approvers
an expiry date

Store it as a file in the repository or as a ticket linked from the pull request, not in a chat thread.

Lifecycle of one security exception from request to fix, rejection or renewal

For each release, put together one evidence bundle tied to the commit SHA:

SAST, dependency and secrets results (SARIF or JSON)
the SBOM
container and DAST results
review approvals
open exceptions with their expiry dates

This maps cleanly to NIST's Secure Software Development Framework (SP 800-218). NIST has also finalised SP 800-218A, which adds practices specific to generative AI. When an auditor or customer asks how a release was checked, you hand over the bundle instead of rebuilding the story.

How do you tell whether the gates are slowing delivery too much?

Track gate time, false positives and bypasses alongside delivery and stability metrics. The gates are too slow when lead time rises and change failure rate doesn't fall.

This matters because AI speed alone doesn't make delivery better. The 2024 DORA report found that AI adoption raises individual productivity but hurts delivery stability and throughput. It concludes that fundamentals like small batches and robust testing still matter. RedMonk's summary of the report puts the estimate at a 7.2% drop in stability for every 25% increase in AI adoption.

Watch five signals each month:

PR gate duration (p50 and p90). Set a time budget and treat going over it as a defect in the pipeline.
False positive rate per rule. Tune or remove rules that are mostly dismissed.
Bypass and exception count. A rising count means a gate is too strict or the code is getting worse. Find out which.
Lead time for changes. If it rises after a gate goes live, find the slowest job.
Change failure rate and escaped vulnerabilities. These show whether the gates are catching anything.

If gates are slow, the usual fixes are:

scan only the diff
run scanner jobs in parallel
cache dependency databases
move full scans to nightly or release runs

Don't make the block threshold looser to save time. Our guide to DORA metrics when AI writes your code shows how to read these numbers together.

Conclusion

Start with the four pull request gates: secrets, diff-scoped SAST, dependency checks and code owner review. Make them required status checks. Block only on verified secrets and new high-severity findings, record every exception with an owner and an expiry date, and bundle the results as evidence for each release. Then measure gate time next to stability, so security stays a check on speed rather than a brake.

Our Security & DevSecOps practice finds the gaps in your pipeline, ranks them by risk and builds these checks into it, starting with a paid pilot.

Want these gates in your pipeline? Talk to an engineering lead →

Frequently asked questions

Should AI-written code have stricter gates than code a person wrote?

Use the same gates and thresholds for all code, and enforce them without exceptions for AI output. Separate rules for AI code are hard to apply, because most changes mix human and assistant edits. What changes is volume: more code per engineer means the automated gates and the required review carry more of the load.

Can an AI code reviewer replace the human approval gate?

No. Treat AI review as an extra advisory check that comments on the pull request. The required approval should come from a code owner who understands the module and can answer for the change.

What should we do with the thousands of findings already in our codebase?

Baseline them so pull request gates report only new findings in changed code. Then rank the existing backlog by severity and exposure, and fix it in small, gated changes. Blocking every pull request on old findings mostly teaches people to bypass the gate.

How long should a pull request security gate take?

Set an explicit time budget for your team and track p50 and p90 gate duration against it. If the gate goes over budget, fix the pipeline: scan only the diff, run jobs in parallel, cache scanner databases, and move full scans to nightly or release runs.

What counts as release evidence for a security audit?

A bundle tied to the release commit: SAST, dependency and secrets results, the SBOM, container and DAST results, review approvals, and any open exceptions with owners and expiry dates. Keeping it per release lets you show exactly how a given version was checked.

This article first appeared on Calidad Infotech.