DEV Community: Mark Nguyen

What Five Months of AI Code Review Actually Found

Mark Nguyen — Sat, 14 Nov 2026 01:05:21 +0000

Originally published on the SLIDEFACTORY blog.

Since February 27, 2026, every pull request in our active repositories has been reviewed by CodeRaven, the AI code-review platform we built and operate. That gives us 148 days of production history. It isn’t a benchmark or a vendor demo. It’s the complete record of what an AI reviewer actually did across our own and our clients’ codebases, including the parts that flatter nobody.

Most claims about AI code review come from the companies selling it, measured on curated examples. We sell it too (CodeRaven is our product), so read this with that in mind. What we can offer is the full dataset, published with its limitations attached. And honestly, the most interesting findings aren’t the ones we would have picked.

The headline numbers

1,164 pull requests reviewed over 148 days
2,812 reviews, about 2.4 per PR, since PRs get re-reviewed as they change
14,552 findings posted as review comments
~7 million lines touched (+5,477,927 added / −1,478,062 removed)
76.0% of AI Code Review approved; 24.0% requested changes
Severity distribution across reviews: 1,441 low · 1,134 medium · 237 high · 0 critical
Total AI spend: $161.20

One in four PRs was asked to change

If you want one number that answers whether AI Code Review actually pushes back or just rubber-stamps, it’s the 24% changes-requested rate. One PR in four got stopped and revised. That rate wasn’t constant. It started much rougher, 57.9% approval in the first partial week, and settled into a band between 72% and 81% as the codebases absorbed feedback and authors got used to being reviewed.

237 high-severity catches, zero critical ones

Over five months, 237 reviews flagged high-severity issues. The critical count is zero, and we’re reporting that as-is rather than dressing it up: either nothing catastrophic entered a pull request in 148 days, or things that would have become catastrophic were caught while they were still merely “high.” The data can’t distinguish those two stories, and neither can we.

What the number does tell you is that real problems worth blocking a merge showed up at a rate of about 1.6 per day, every day, across three active repositories. Code review, human or AI, exists for that steady drip, not for the dramatic save.

The AI Code Review cost: $161.20. Total.

Not per month. The entire 148-day, 14,552-finding history cost $161.20 in AI spend. That works out to roughly 1.1 cents per finding and about 68 cents per high-severity catch.

For comparison, price out fifteen minutes of a senior engineer’s time reading a diff, spotting the issue, and writing it up. This isn’t “cheaper than a human reviewer.” It’s a different category of expense. The whole AI Code Review layer costs less per month than the coffee at the standup where you’d debate turning it off. At that price, the question stops being whether AI review is worth it and becomes what the argument is for not running it.

Month by month: volume nearly doubled, scrutiny held

Month (2026)	PRs	Reviews	Low / Med / High	Approval
Feb (27–28)	16	19	10 / 8 / 1	57.9%
Mar	200	420	269 / 123 / 28	75.5%
Apr	227	431	244 / 160 / 27	81.2%
May	194	559	258 / 240 / 61	76.9%
Jun	283	775	385 / 316 / 74	72.3%
Jul (1–24)	244	608	275 / 287 / 46	77.3%

Review volume nearly doubled from March (420) to June (775) as re-review activity grew: more iterations per PR, not just more PRs. The approval rate stayed in a narrow band while that happened, which is the pattern you’d want. Throughput scaled without the reviewer getting more lenient. May and June also carried the heaviest high-severity load (61 and 74), the same months with the deepest re-review cycles.

The clearest pattern in the author data: churn predicts pushback

CodeRaven reviews PRs from eight contributors. Three data points describe the spread:

Our highest-volume contributor by PR count: 364 PRs, 83.3% approval
Highest approval rate in the dataset: 84.8%
Our highest-churn contributor: 196 PRs, 2.63 million lines added, and the lowest approval rate: 62.3%

The contrast worth noticing: PR count alone doesn’t predict pushback. The contributor with the most PRs holds an 83.3% approval rate, close to the best in the dataset. The contributor who moved the most code, 2.63 million lines across 196 PRs, sits at the bottom. GitClear has been documenting this churn-quality correlation industry-wide as AI-assisted coding pushes churn and duplication up. We can see it reproduced at the level of individual contributors in our own repos. The more code you move per unit time, the more often review says stop.

The honest reading is that approval rate mostly measures blast radius per change. Small, contained changes sail through AI Code Review; big sweeping ones get stopped. When AI helps you generate more code faster, independent review gets more necessary, not less. The throughput is real, and so is the debt.

What this dataset can’t tell you about AI Code Review

Every “state of” report should have this section; here’s ours.

Severity is the only structured category. CodeRaven doesn’t currently tag findings by issue type, so we can’t rank “most common mistake” from structured data alone. A text-mining pass over the 14,552 review comments is the planned follow-up to this report.
Three active repositories. Eight repos are monitored, but the volume comes from three internal and client repositories. This is one team’s production reality, not an industry sample.
Month boundaries are approximate. Monthly rows come from rolling-window differencing and can be off by one or two at the edges. All-time totals are exact.
We built the reviewer. We’ve published the raw counts so you can discount our interpretation and keep the data.

What we’d do with these numbers

If you’re running AI-assisted development (Copilot, Claude, Cursor, agents), here are the takeaways we act on ourselves:

Put review where the churn is. Our data says high churn predicts changes-requested. If AI is multiplying your line count, that’s precisely where an independent reviewer belongs.
Treat sub-penny review as default-on. At 1.1 cents per finding, the cost argument against a second set of eyes no longer exists.
Watch approval-rate drift, not just bug counts. A reviewer that gets more lenient as volume grows is decorative. Ours held between 72% and 81% while volume nearly doubled, and that band is the health metric we track.

If you run AI review on your own repos, what approval rate are you seeing? I’d like to compare notes.

AI-Native SDLC: The Agency Reality Check

Mark Nguyen — Sat, 14 Nov 2026 01:04:49 +0000

Originally published on the SLIDEFACTORY blog.

Anthropic published two pieces this year that together describe what happens to an engineering organization once agentic coding stops being a tool somebody uses and becomes the way the whole team works.

The first, published in June, was Fiona Fung’s talk writeup from Code w/ Claude SF. She runs engineering for Claude Code, and her piece is about people and process: the four team norms that stopped working once her team went agentic, and what replaced them. The second, published in August, was Louis Claxton’s AI-native SDLC playbook, which is the more prescriptive counterpart. It lays out six stages, a chain of committed files that runs through all of them, and a governance layer underneath.

Read together, they’re probably the best public description of this shift I’ve seen. They also share an assumption neither one really states: you own your roadmap. Anthropic can change direction when its own product, tooling, or team changes what makes sense. Agencies can’t. We’re an eleven-person agency, and most of our work arrives with a signed scope, a fixed date, and a client who agreed to a specific list of deliverables. That constraint changes which parts of the AI-native SDLC you can actually adopt, and most of what’s being written about this shift isn’t written for people working under it.

One disclosure before the rest. We build and sell CodeRaven, an AI code-review platform, so we have a commercial interest in one specific stage of this process. The data further down is published with its limitations attached, and you’re welcome to keep the numbers and discard our framing.

What an AI-native SDLC actually changes

The software development lifecycle is just the path a change takes from someone’s idea to something running in production. You plan it, design it, build it, test it, deploy it, and maintain it. Traditionally, each step belongs to a different role, and work moves between them through documents, tickets, meetings, and sign-offs.

A lot of that ceremony was built around one assumption: writing the code was the slow, expensive part. Requirements documents, estimation rituals, and design review existed to force agreement before anyone spent three months building the wrong thing. When building the thing takes an afternoon, a lot of that structure is guarding against a cost that no longer exists.

An AI-native SDLC keeps the six stages but changes what travels between them. Instead of relying mainly on tickets and meetings, each stage ends by committing a plain markdown file that the next stage reads. An idea becomes an intent file. The intent becomes a spec. The spec becomes an implementation plan naming which files change and which tests prove it worked. Then come the actual diff, the pull request and its review findings, and an incident record if production disagrees with all of it.

The files are readable by both a person and an agent, they live in git next to the code, and the chain of commits becomes an audit trail without anyone having to build a separate one. You’ll also see the same idea called the agentic SDLC or just the AI SDLC. Different labels, same basic structure.

The bottleneck moved, and three stages absorbed it

Fung’s point is straightforward. On her team, writing code, writing tests, and refactoring stopped being what slowed anyone down. The bottleneck didn’t disappear when the typing did. It moved to verification, review, and security, because those still run much closer to human speed regardless of how fast the diff shows up.

Claxton makes the same point structurally. Build collapses to hours while the stages on either side keep much of their original length. Your review queue was sized for the amount of code humans could write. Your security review was staffed around roughly the same assumption. Your approval gates still run on their existing cadence. None of that gets ten times faster because generation does, so either the queue backs up or work starts shipping under-reviewed.

The real question isn’t which AI tools to buy. It’s which existing processes were solving a problem you no longer have, and which ones just had a lot more work dropped on them.

The four norms Fung’s team rewrote

The tooling is only part of the change. Fung’s other point is that the way the team works has to change with it.

Planning went just-in-time. Her team wrote a solid six-month roadmap and it was obsolete by month three, specifically because of what their own tooling did to throughput. Planning moved away from long design documents and toward prototypes and discussion in pull requests. Build it, put internal users on it, and act on what they say.

Context gathering stopped starting with the author. When a person wrote the code, finding that person was usually step one when someone had a question. When nearly every pull request is agent-assisted, “who made this change” doesn’t get you very far. The better question is what you actually need to know. Ask the agent first, then ask whether that answer should be captured somewhere so the next person doesn’t have to go looking for it again.

Review became trust but verify. The agent handles style, linting, obvious bugs, and test coverage. Humans stay where judgment still matters: legal risk tolerance, trust boundaries, security-sensitive code, and product taste. Fung’s caveat here is important. The right split keeps moving as models improve, so the review policy needs to be revisited instead of written once and treated as finished.

Roles blurred. Product managers on her team write code now. Engineers pick up design and content work. What she’s stopped hiring for is raw throughput, because the models increasingly provide that. The value moves toward creative builders with product sense and engineers with deep systems knowledge.

Obsolete processes don’t remove themselves. Somebody has to have permission to kill them, and on her team that permission is an explicit principle rather than something everyone is expected to infer.

Where this breaks against a signed scope of work

Just-in-time planning is the right answer when the roadmap belongs to you. It’s a much harder sell when a client signed a statement of work that lists the deliverables and the launch date. We can’t always prototype our way toward whatever the project turns out to want. We’ve already committed to a scope, and the commercial relationship is built around delivering against it.

The same tension runs through the rest of the process. Fung’s advice to kill your noisiest process assumes you own the process. A lot of ours belong to clients: their approval cycles, brand review, change boards, security requirements, release windows, procurement rules, accessibility checks, and internal ticketing systems. Those aren’t processes an agency can simply decide are obsolete.

Claxton’s playbook has a sidebar about legacy systems and naming one source of truth per artifact, but for an agency that is the main event rather than a footnote. The cleanest version of the playbook has the repository holding the authoritative record. On client work, that option frequently isn’t ours to choose. Their Jira may be the system of record because their auditors, compliance team, or internal process already depends on it. We don’t get to replace it with a markdown file in our repo because the architecture would be cleaner.

None of that makes the AI-native SDLC unusable for agencies. It just means the split runs differently than it does at a product company. The stages where you control your own tooling port over almost completely. The stages where the client’s process is the gate often don’t, and pretending otherwise usually produces a plan nobody follows for very long.

What the AI-native SDLC gives you regardless of who owns the roadmap

Review does, and we have data on that one rather than opinion.

Since February 27, we’ve run every pull request in our active repositories through CodeRaven. Over the first 148 days, that came to 1,164 pull requests, 14,552 findings, and a total AI spend of $161.20. The full breakdown is in our five-month AI code review dataset.

Two findings matter here. First, 24% of pull requests got changes requested, and that rate stayed within a 72% to 81% approval band even while review volume nearly doubled from March to June. A reviewer that gets more permissive as the pile grows isn’t very useful, and we didn’t see that drift.

Second, across eight contributors, what predicted getting stopped wasn’t how many pull requests someone opened. Our highest-volume contributor by count had an 83.3% approval rate, near the top of the group. The contributor who moved the most code, 2.63 million lines added across 196 pull requests, had the lowest approval rate at 62.3%. Approval rate mostly tracked how much surface area a change touched.

That’s the sequencing argument. Agentic coding raises the amount of code a person can move in a week, and the amount of code moved is the thing that correlates with review pushing back. Turning on generation before the independent check means accelerating exactly the input most likely to create problems.

Two files port over too, and both cost very little to put in place. A CLAUDE.md at the repo root holds build, test, and lint commands, the conventions that actually matter, and a running list of what the agent keeps getting wrong. A simple rule works well: if the same mistake happens twice, it goes in the file. Anthropic’s memory documentation covers the mechanics.

Alongside it, a review policy defines which passes you care about, what counts as important versus a nit, how many nits one review should be allowed to report, and what should be skipped entirely. That last part matters because AI review can bury a team in low-value style comments until people stop paying attention to the reviewer altogether. Reliable agent workflows need more than capable models. They also need clear instructions, evaluation criteria, review policies, and defined boundaries for human oversight.

Plan mode ports over too, and for agency work it’s probably the highest-leverage thing on this list. The agent reads the codebase without touching it and produces a written plan naming which files should change and which tests should prove the work is correct. The engineer corrects the plan before code exists.

Catching a bad decision while it’s still a paragraph costs a text edit. Catching the same decision in a finished diff costs a rework cycle, and when the diff itself took four minutes to generate, that rework cycle becomes one of the most expensive parts of the change. Waterfall front-loaded planning because building was expensive. AI-native engineering front-loads planning because building is cheap and reviewing isn’t.

There’s also a second benefit that matters specifically when you bill clients. A committed plan becomes a scope artifact. When a client asks for something the plan doesn’t cover, the conversation about whether it’s in scope has a document attached to it.

What to measure, and the one metric we’d keep

Fung suggests tracking three things as these changes roll out: onboarding ramp time going down, pull request cycle time going down, and the share of agent-assisted commits going up. Her caveat is right: throughput is a metric, not the goal.

Her evidence for those three is the weaker part of an otherwise strong piece. Ramp time is described as much faster than a year ago. The commit metric is an anecdote about not having seen a non-assisted commit in four months. Those are useful directional signals, but they’re surprisingly directional for an article telling engineering leaders to start measuring.

If we kept one, it’d be approval rate, or whatever your equivalent measure of review strictness is. Ramp time and cycle time both improve naturally when generation speeds up, so they mostly confirm that the tool is doing what you expected. Approval rate is the one that can quietly go the wrong way while every other number looks great, and you won’t see that drift unless you record a baseline before changing anything.

What this can’t tell you

We have production data for one stage out of six. The 1,164 pull requests are review data. Our use of plan mode, context files, and scope artifacts is experience rather than measurement, and those sections should be read at that weight.

The sample is three active repositories and eight contributors on a single team. That’s our production reality, not an industry pattern. We also sell an AI code reviewer, and the section arguing you should sequence review first is the one place where our data and our commercial interest point in the same direction.

We also don’t have a clean baseline, because these repositories were never run without AI review during this dataset. We know that 24% of pull requests received changes requested, but we can’t tell you what that number would’ve been under human-only review.

What we’d do with this

Sequence review ahead of generation. Churn predicts pushback, and agentic coding is a churn multiplier. The other order accelerates the problem before installing the check.
Write the two files before anything else. A context file and a review policy change agent output immediately, and most of the remaining practices assume one or both already exists.
Treat plan mode as both a control and a scope artifact. On client work, the second function can be worth as much as the first.
Sort your processes by who owns them. The ones you own are fair game for Fung’s advice about killing what no longer serves you. The ones a client owns are a negotiation, not a decision, and planning as though they’re yours is how these initiatives stall.
Record one number before you change anything. Ours is approval rate. Most of the speed metrics improve on their own and tell you less.

The practical version of the AI-native SDLC for an agency is narrower than the version you’d implement inside a product company. The file chain and review layer port over almost completely, the organizational norms port where you actually control the process, and just-in-time planning mostly doesn’t port when someone else signed for the scope.

That’s a narrower adoption than either Anthropic article describes, but it’s still most of the value.

If you’ve adopted parts of this on client work, which pieces held up against a fixed scope, and which didn’t?

Claude Code Removed from Anthropic’s Pro Plan. And It Reveals a Bigger AI Pricing Problem

Mark Nguyen — Sat, 14 Nov 2026 01:04:01 +0000

Originally published on the SLIDEFACTORY blog.

On April 21, 2026, developers noticed that Claude Code had been removed from Anthropic’s $20 Pro plan, silently, with no announcement, no email, and no entry in the changelog. Just an updated pricing table with a red X where a checkmark used to be. By the next morning, after a wave of developer backlash, the page had been quietly restored. Anthropic’s Head of Growth, Amol Avasare, called it a test affecting roughly 2% of new Pro signups. Existing subscribers, he said, were not impacted.

So everything’s fine. Move along.

Except this story isn’t really about a pricing page glitch. The Claude Code Pro plan removal, however brief, is a signal that the economics of flat-rate AI subscriptions are quietly falling apart, and Anthropic isn’t the only company that’s going to feel it.

Update, November 2026: Where Claude’s Plans Ended Up

Claude Code stayed on the Pro plan. Anthropic’s pricing page (checked November 13, 2026) lists Pro at $20 a month, Max at $100 and a new $200 tier called Ultra, with Claude Code included on all three. Max and Ultra now also come with monthly API credits, $100 and $200, for building on the Claude Platform. The rest of this article is as published in April.

The Gym Membership Problem: How AI Subscriptions Were Always Built on an Assumption

To understand what’s happening with Anthropic’s pricing, it helps to think about gyms.

A gym charges you $40 a month. But that gym doesn’t have physical capacity for all of its members to show up simultaneously. Most gyms sell three to ten times more memberships than they can actually accommodate at peak hours. They do this because they’ve studied behavior: the average member goes twice in January, once in February, and barely at all by spring. The business model works because most people don’t use what they’re paying for.

This isn’t dishonest. It’s a rational model built on realistic usage assumptions. You get cheap access to a great facility. The gym gets predictable revenue. Everyone wins, as long as the assumption holds.

AI subscriptions work the same way. When Anthropic priced Claude Pro at $20/month, that price was calibrated around how humans use AI: you ask a question, read the answer, think for a moment, ask something else. That cycle has natural friction: attention limits, work hours, meetings, sleep. Even a heavy human user generates a finite volume of compute in a month.

The gym membership model works until too many members actually show up. And in 2026, AI agents are flooding the gym.

AI Agents Are the Friends Who Never Leave the Gym

Here’s where the analogy gets precise. Imagine the gym started allowing members to lend their access cards to friends. One friend becomes two. Two becomes a group chat. And unlike you, these friends don’t go home. They’re on the treadmill at 3am. They bring more friends. They run every machine simultaneously, indefinitely, without stopping to check their phone.

That’s what AI agents do on a flat-rate subscription.

An agent doesn’t sleep. It doesn’t pause between prompts. It doesn’t get distracted. You set it loose on a task at midnight and it runs in autonomous loops, firing requests continuously, chewing through context windows the size of novels, generating output that would take a human days to produce. And it does this on the same $20/month membership as the person sending three chat messages a day.

One developer tracked 10 billion tokens over eight months of daily Claude Code use, compute that would have cost over $15,000 at API rates, running on a subscription that cost $800 total across that period.

The Register reported that some Claude Max subscribers ($200/month) were running between $1,000 and $5,000 worth of agent compute every single month. The math isn’t complicated. And it isn’t sustainable.

“Engagement per subscriber is way up. We’ve made small adjustments along the way, but usage has changed a lot and our current plans weren’t built for this.” – Amol Avasare, Anthropic Head of Growth

That’s a polite way of saying: we sold you a gym membership, and you’re living there. And you brought friends.

OpenClaw Was the “Lend Your Card” Problem Made Literal

The most vivid example of the friends-at-the-gym problem was OpenClaw, an open-source AI agent framework created by Austrian developer Peter Steinberger. By early 2026, OpenClaw had accumulated 247,000 GitHub stars, making it one of the fastest-growing open-source projects in history. The appeal was simple: connect your Claude Pro subscription to OpenClaw and suddenly you had a persistent personal AI agent running continuously through WhatsApp, Telegram, email, and anything else you pointed it at.

In gym terms: OpenClaw was a mechanism for lending your membership card to an entity that genuinely never left the building. It ran at 6am, at midnight, on weekends. And it brought more friends, with multi-agent setups where one $20 subscription powered several parallel agents, each consuming full compute resources simultaneously.

On April 4, 2026, more than two weeks before the Claude Code Pro plan removal, Anthropic shut it down. As PYMNTS reported, Claude Pro and Max subscribers could no longer run third-party agent frameworks on their flat-rate plans. For developers who had built operations around the tool, the cost increase was reported at up to 50 times their previous monthly spend.

Anthropic’s April 2026 Timeline:

April 4: Anthropic blocks third-party agent frameworks, including OpenClaw, from running on Pro and Max subscription plans. API billing required for all external agent usage going forward.

April 21: Claude Code quietly removed from the $20 Pro plan pricing page with no public announcement. Discovered by developers comparing current documentation against the April 10 archived version.

April 22: Pricing page restored after developer backlash. Anthropic confirms it was an A/B test on ~2% of new Pro signups. Head of Growth signals broader plan restructuring is coming.

Read those three events together and a pattern is obvious. This isn’t accidental. Anthropic is methodically separating human-pattern usage from agentic usage, and figuring out how to charge accordingly.

Claude Code Isn’t the Only Subscription Feeling This Pressure

It would be easy to frame the Anthropic Claude pricing changes as a communication problem, and the communication has genuinely been poor. Developers who built workflows around Claude Code Pro plan access deserved more than a silent pricing page update. But blaming Anthropic’s process misses the structural problem underneath it.

Every AI company that sold flat-rate subscriptions before agentic AI became mainstream is facing the same math.

Cursor, the AI coding editor, went through its own version in mid-2025. After switching from fixed request allotments to credit-based usage pools, some developers reported $350 in overages in a single week on a $20/month plan. Cursor issued a public apology and refunded affected users. They hadn’t priced for agents either.

GitHub Copilot and Google are managing the same capacity constraints. OpenAI publicly took aim at Anthropic the day Claude Code’s Pro plan removal surfaced, announcing that Codex would remain on their free and $20 tiers, but OpenAI is operating on a fundamentally different compute cost structure. That math will catch up with them too.

The gym analogy has a hard limit: when too many members actually show up, the gym either gets bigger, sells fewer memberships, or changes the pricing terms. There’s no fourth option. AI companies are in the same position, and right now they’re all quietly working toward the same destination: usage-based pricing that reflects what agents actually consume.

What the Claude Code Pro Plan Situation Means for Your Business

If you’re running AI tools at any scale, as an agency, a development team, or a business that’s embedded these tools into real workflows, a few things are worth acting on now rather than waiting until pricing forces your hand.

Audit the difference between chat usage and agent usage

There’s a meaningful practical difference between using Claude as a conversational assistant and running Claude Code as an autonomous agent across a codebase. The first matches what $20 subscriptions were designed for. The second is the gym membership problem, and it’s exactly what Anthropic is moving to price differently. Know which category your usage falls into, because that distinction is increasingly what tier structures will be built around.

The pricing gap between tiers is going to widen

The $80 jump from Claude Pro ($20) to Claude Max ($100) already felt like a significant leap. If Anthropic restructures plans to formally separate human-pattern usage from agentic usage, which Avasare’s comments strongly suggest is coming, expect that gap to grow. The tool you’re using on a $20 plan today has a realistic chance of becoming a $100 tool within 12 months. Budget for it now.

Track your actual token consumption

Most businesses using AI tools have no visibility into what they’re actually consuming. That data matters a lot when pricing shifts. Start tracking usage now, not because you need to cut back, but because you need a real baseline when you’re deciding between plan tiers, API billing, or building your own infrastructure.

Don’t build single-vendor dependency into critical workflows

The Claude Code situation is a useful reminder that any AI tool running on a subscription plan can change under you, sometimes overnight and sometimes without warning. A pricing restructure, a policy shift, or a capacity crunch can disrupt workflows you’ve started treating as infrastructure.

This is something we navigate constantly, both in our own products and when integrating AI into client work. When we built CodeRaven, one of the core decisions was designing around model portability from the start, not because we assumed any single provider would pull the rug, but because we’d seen enough of how this industry moves to know that locking into one provider’s pricing model is a liability, not a convenience.

The practical hedge isn’t complicated: know which parts of your stack would break if a pricing change hit tomorrow, and have at least a rough answer for what you’d do if it did. That might mean keeping API access as a fallback alongside a subscription, maintaining some familiarity with alternative models, or simply avoiding deep automation built on top of features that haven’t been around long enough to feel permanent. Claude Code on a $20 plan, as it turns out, was one of those features.

Consider whether usage-based pricing might already make more sense for you

For some workflows, paying per token through the API is actually cheaper than a flat subscription, especially if your usage is irregular rather than continuous. If you’re doing burst-heavy, intermittent work, API billing gives you more control. If you’re running agents continuously, a subscription at the right tier is usually still more efficient. The point is to do the math intentionally rather than defaulting to a plan because it was right six months ago.

The Gym Eventually Has to Charge More for 24/7 Access

Gyms that oversell memberships don’t collapse because members use too much. They collapse when too many members actually show up, when the implicit deal stops working for everyone simultaneously.

What Anthropic is navigating right now is exactly that moment. Usage patterns changed faster than pricing could follow. Claude Code shipped. Claude Cowork landed. OpenClaw exploded to a quarter million GitHub stars. Long-running async agents became everyday workflows practically overnight. As Ed Zitron noted, the Max plan was designed before any of this existed. It was built for heavy chat usage, not for agents running around the clock.

The uncomfortable truth about the flat-rate AI subscription model is that it was always priced on the assumption that most users wouldn’t fully use what they were paying for. That assumption made AI tools accessible to a mainstream audience, which was genuinely valuable. But as agentic workflows become the default rather than the exception, that assumption is obsolete.

What comes next, whether that’s tiered agent pricing, compute credit pools, hybrid models, or something we haven’t seen yet, will define which AI platforms win the next two years. Not necessarily the companies with the best models, but the companies that figure out how to price access to those models in a way that’s sustainable for both sides of the transaction.

The gym needs to either get bigger, sell fewer memberships, or start charging more when you move in permanently and bring your friends. Anthropic is actively testing which of those it’s going to be. So is everyone else in the industry. Pay attention to how they answer, because the structure they land on will shape how your business plans around AI tools for years.

How are you handling this on your own team: subscriptions, API billing, or a mix?

Retrieval-Augmented Generation (RAG) Systems with OpenAI

Mark Nguyen — Sat, 14 Nov 2026 00:52:38 +0000

Originally published on the SLIDEFACTORY blog.

Exploring Retrieval-Augmented Generation (RAG) Systems with OpenAI

Recently, I had a conversation where the term RAG (Retrieval-Augmented Generation) came up. While it’s becoming more common in AI discussions, many people still have questions about what it actually means and how it can be implemented. A lot of interest revolves around how RAG works in practical terms, especially when paired with tools like OpenAI. So, in this article, we’ll dive into what RAG systems are, how they function, and how they can be used to tackle real-world challenges such as answering complex queries and generating context-aware content.

Update, August 2026: The OpenAI Landscape Has Changed

If you built (or are about to build) RAG on OpenAI’s Assistants API, act now: OpenAI deprecated the Assistants API and is shutting it down on August 26, 2026. Its replacement, the Responses API, reached feature parity and is where all new RAG work should happen. The concepts in this article are unchanged (retrieval plus generation is still the architecture), but the implementation path is now the Responses API’s built-in file_search tool: you upload documents to a vector store, OpenAI handles parsing, chunking, and embeddings, and retrieval runs a hybrid of vector and keyword search automatically. For custom pipelines, the pattern below (embed, retrieve, augment the prompt, generate) still applies verbatim; only the API surface has moved. If your production system still says assistants anywhere in its code, migrating is this month’s job, not this quarter’s.

What is a Retrieval-Augmented Generation (RAG) System?

At its core, Retrieval-Augmented Generation (RAG) is a hybrid method that combines the power of information retrieval and natural language generation (NLG). RAG allows AI systems to retrieve relevant information from external databases, knowledge repositories, or other data sources, and then generate contextually accurate responses using an NLG model like OpenAI’s GPT. This combination is particularly useful in domains where large-scale models alone may not have up-to-date or specific enough information. By leveraging external sources, the RAG system ensures that generated content is accurate and based on real-time, relevant data.

How RAG Systems Work

RAG systems operate in two primary stages:

Retrieval Stage: The system searches for relevant information or documents based on a user’s input. The retrieval process can pull from web databases, internal knowledge repositories, or structured data sources like SQL databases. The goal is to identify and rank the most relevant pieces of information based on the user’s query.
Generation Stage: Once the relevant information has been retrieved, it’s passed to a generative model like OpenAI’s GPT. The model uses the retrieved data to generate a response, ensuring that the output is grounded in accurate, up-to-date information. This makes the generated content richer and more context-aware, as it reflects real-time or specific data rather than relying solely on the model’s training data.

Real-World Application: Using RAG for Search Query and Article Retrieval

In the partial example below, we’ll breakdown how a Retrieval-Augmented Generation (RAG) system might improve the quality of answers to a users question by:

Searching for relevant articles,
Using that information to augment a user’s original query,
and then generating a more detailed response with a LLM completion API.

The Process

1. User Query

The user asks a question in natural language:

“Can you summarize the most recent research on climate change impacts?”

2. Retrieval Stage (Search Query)

The system first performs a search for relevant, up-to-date articles on the impacts of climate change. It uses a search engine or a pre-built database to retrieve the most relevant articles related to the user’s query.

For example, the system retrieves the following articles:

Article 1: “Impact of Climate Change on Global Agriculture in 2024”
Article 2: “Rising Sea Levels and Their Effect on Coastal Communities”

3. Augmentation Stage (Modifying the User Query)

Once the articles are retrieved, the system can scan the article, grab the relevant information and augment the original user query with the key findings or highlights from the retrieved articles. This can provide additional context for OpenAI (or any external LLM)to generate a more informed and up-to-date response.

Here’s an example of the augmented prompt that could be sent to OpenAI:

“Recent research shows significant impacts of climate change, including disruptions in global agriculture and rising sea levels affecting coastal communities. Can you summarize these effects and suggest mitigation strategies?”

4. Generation Stage (OpenAI’s Response)

With the augmented query, OpenAI’s model generates a more detailed and contextually aware response. Here’s an example of what OpenAI might return using the Chat Completion API:

{
    "role": "assistant",
    "content": "Recent research highlights two major impacts of climate change. First, global agriculture is facing disruptions due to changing temperatures and extreme weather patterns, leading to reduced crop yields. Second, rising sea levels are threatening coastal communities, causing displacement and increasing the risk of floods. To mitigate these impacts, strategies include investing in sustainable farming practices and developing better flood defense systems for vulnerable coastal areas."
}

‍

5. Final Output to the User

The final output displayed to the user is the response from OpenAI, which now includes both a summary of recent research and actionable suggestions for addressing the impacts of climate change.

“Recent research highlights two major impacts of climate change. First, global agriculture is facing disruptions due to changing temperatures and extreme weather patterns, leading to reduced crop yields. Second, rising sea levels are threatening coastal communities, causing displacement and increasing the risk of floods. To mitigate these impacts, strategies include investing in sustainable farming practices and developing better flood defense systems for vulnerable coastal areas.”

Benefits of Using RAG Systems

The combination of retrieval and generation in RAG systems offers significant advantages for many applications:

Access to Updated Knowledge: Large language models like OpenAI’s GPT are trained on static datasets, which means they may not have access to the most current information. A RAG system augments this by retrieving real-time data, whether from databases or live web sources, allowing for more up-to-date responses.
Domain-Specific Expertise: Whether it’s healthcare, finance, or any other specialized field, RAG systems can retrieve highly specific, domain-relevant information to generate accurate and reliable content that adheres to the necessary guidelines or terminologies.
Enhanced Performance on Complex Queries: For questions requiring detailed, multi-step responses, like generating SQL queries or providing legal advice, the retrieval component ensures the generated content is grounded in factual, relevant data.
Improved Accuracy and Fact-Checking: Since the model bases its generation on retrieved information, the risk of hallucination (the model generating incorrect information) is minimized. This makes the system more reliable for use cases requiring high accuracy, such as SQL query generation or complex research questions.

Implementing RAG Systems with an LLM

The actual set-up of a RAG system can vary in complexity, depending on the actual needs of the application. Here’s a high-level view of how you might go about setting up an implementation:

Retrieval Module: The first step is setting up a retrieval system that can pull in relevant data. This could be something as simple as indexing documents with ElasticSearch or as complex as integrating with proprietary APIs.
Preprocessing Retrieved Data: Once relevant information is retrieved, it might need preprocessing, such as summarizing or filtering out irrelevant parts, before feeding it into the model for generation.
Generation: The retrieved data is passed to the generative model (OpenAI’s GPT, for instance), which uses this context to generate a response tailored to the user’s query.
Post-Processing: After generation, you may need additional steps, such as fact-checking, formatting, or refining the generated output, depending on the complexity of the user’s request.

A Working Example: RAG with the Responses API (November 2026)

The walkthrough above runs retrieval yourself. If your documents can be stored with OpenAI, the Responses API can do the retrieval step for you: index the files once in a vector store, and the file_search tool pulls the relevant chunks into each answer. This is the path for anything that used to run on the Assistants API.

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment
MODEL = "gpt-6-astra"  # the model OpenAI's file search guide uses (Nov 2026)

# 1. Index the documents once. A vector store holds their chunks and embeddings.
store = client.vector_stores.create(name="research-articles")
for path in ["agriculture-2026.pdf", "sea-levels-2026.pdf"]:
    with open(path, "rb") as f:
        uploaded = client.files.create(file=f, purpose="assistants")
    client.vector_stores.files.create_and_poll(
        vector_store_id=store.id, file_id=uploaded.id
    )

# 2. Ask. file_search retrieves the relevant chunks and the model answers
#    from them: retrieval, augmentation and generation in one call.
response = client.responses.create(
    model=MODEL,
    input="Summarize the most recent research on climate change impacts in these articles.",
    tools=[{
        "type": "file_search",
        "vector_store_ids": [store.id],
        "max_num_results": 5,
    }],
    include=["file_search_call.results"],  # also return the retrieved chunks
)
print(response.output_text)

# 3. List the files the answer cited.
for item in response.output:
    if item.type == "message":
        for part in item.content:
            for note in getattr(part, "annotations", None) or []:
                if note.type == "file_citation":
                    print("cited:", note.filename)

Three things to check before you rely on it:

Model: the example uses the model named in OpenAI’s file search guide in November 2026. Use whichever current model your account has access to.
max_num_results trades answer quality for speed and tokens. Start around 5 and measure.
include=["file_search_call.results"] returns the chunks the model saw. Keep it on while testing so you can see why an answer came out the way it did.

If your data can’t leave your own systems, keep retrieval on your side (a database or search index you control) and send only the retrieved text in the prompt, as in the walkthrough above.

Conclusion

While OpenAI and large language models (LLMs) are incredible tools, Retrieval-Augmented Generation (RAG) systems provide a powerful way to enhance these platforms, making it easier to access relevant, real-time data and generate more contextually accurate responses. Whether you’re answering customer queries, augmenting existing data sets with up-to-date information, or optimizing token usage for API requests, RAG systems offer an efficient solution by combining data retrieval with advanced language generation. The result? Smarter, more informed responses that save time and effort while improving the overall quality of interactions.

By integrating OpenAI’s capabilities with retrieval techniques, you can build applications that not only understand your needs but also act on them quickly and accurately. For businesses, this means more efficient workflows and improved decision-making, without needing deep technical expertise.

‍Sources & further reading:

Building an AI Video Development Workflow: Bigfoots Digital Survival Guide

Mark Nguyen — Sat, 14 Nov 2026 00:52:08 +0000

Originally published on the SLIDEFACTORY blog.

72 Hours with AI: Creating Our Bigfoot Commercial

As a Portland-based AI agency, we’re always looking for new ways to push the boundaries of what’s possible with technology. At SLIDEFACTORY, our roots are in digital innovation, but today, that means more than just building websites or apps. We’re focused on the future, leveraging AI and emerging technologies to reimagine creative production, workflows, and storytelling.

Curious about the limits of generative AI in an AI video production process, we decided to put our capabilities to the test. The challenge? Create a complete commercial, from concept to final cut, using only AI tools. No traditional cameras, no live crews. Just prompts, models, and our creative instincts.

We gave ourselves exactly 72 hours.

What followed was an intense, fast-paced sprint that was equal parts exhilarating and eye-opening. The result wasn’t just a finished piece of content. It was a firsthand look at how far AI-powered production has come, and where it’s heading.

For us, this wasn’t just an experiment. It was a real-time case study in how a nimble, creative AI agency can move quickly, adapt new tools, and deliver results that would have taken weeks in a traditional pipeline.

Update, November 2026: How Our AI Video Workflow Has Changed

This post covers a 72-hour sprint from July 2025, and the tools have moved since. Our current process is laid out step by step in our AI video production case study. It starts the same way, with Midjourney stills and an ElevenLabs voice, but the voice is now a custom voice built from a human read and converted with speech-to-speech. For animation we choose the model by content type: Veo 3, Seedance 2.5, Runway or HeyGen, and a finished video often mixes shots from more than one. The edit still happens in Adobe Premiere Pro.

The Spark: Why Bigfoot?

Every great project starts with that moment when someone throws out a seemingly random idea that somehow clicks. We were brainstorming concepts for our own promotional video when someone in the office mentioned Bigfoot. Not as a joke, but as a genuine creative direction. There’s something universally appealing about the myth: mysterious, larger than life, and surprisingly relatable when you think about it. Plus, what Portland, Oregon based Digital Agency doesn’t love a good cryptid story?

The more we discussed it, the more it made sense. Bigfoot could represent the elusive “perfect client” or embody the mysterious creative process itself. The concept had legs (big, hairy legs), and we decided to run with it. With our tight 72-hour deadline looming, we dove headfirst into the AI production pipeline.

Building the Foundation: Script and Concept Development

Initially, we considered building out a complex N8N workflow to automate our content generation process. However, we quickly realized this would be overkill for our specific use case. When working under tight deadlines, we’ve found that the simplest approaches often work best, and ChatGPT’s direct interface would serve us just as well without the additional setup time.

So, after discarding our initial fancy ideas, ChatGPT became our first stop. To benign, we fed it detailed information about SLIDEFACTORY. Our services, our studio personality, our target audience. Then we asked it to generate multiple concept outlines. The AI didn’t disappoint. Within minutes, we had a dozen different approaches, from documentary-style mockumentaries to sleek corporate presentations featuring our mythical protagonist.

After reviewing the options, we found a direction that felt authentically us while leveraging Bigfoot’s inherent charm. But having a concept was just the beginning. We needed a script that could actually work within our technical constraints and timeline.

This brought us back to ChatGPT, this time asking for multiple script variations based on our chosen concept. The AI generated everything from snappy 30-second spots to longer narrative pieces. We refined, combined, and iterated until we had something that felt right, a script that balanced humor with genuine information about our services, all delivered through our unlikely spokesperson.

The collaborative process with AI felt surprisingly natural. Rather than replacing our creative instincts, ChatGPT became a brainstorming partner that never ran out of ideas. It helped us explore directions we might not have considered while keeping our core vision intact.

Designing Our Star: The Great Bigfoot Character Hunt

With our script locked, we needed to bring our character to life visually. This is where MidJourney entered the picture, and honestly, this phase became one of the most entertaining parts of the entire project.

We started broad: What kind of Bigfoot were we creating? The classic forest-dwelling cryptid? A modern, tech-savvy influencer version? Someone in an obvious costume winking at the camera? MidJourney helped us explore every possibility, generating image after image as we refined our prompts.

The iterations were fascinating. We saw Bigfoot in business suits, Bigfoot holding smartphones, Bigfoot that looked convincingly real, and Bigfoot that was obviously theatrical. Each generation taught us something about our character and helped us narrow down the personality we wanted to convey.

Eventually, we landed on a design that struck the perfect balance: clearly Bigfoot, but with a contemporary, approachable quality that would work for our brand. The character had personality in his eyes and a presence that suggested he’d be comfortable on camera discussing slide presentations and creative services.

Finding the Voice: From Pixels to Personality

Once we had our visual locked down, we faced an interesting challenge: What should Bigfoot sound like? We took our final character image back to ChatGPT and asked it to describe the voice that would match this particular Bigfoot. The AI analyzed the visual elements and suggested characteristics: warm but authoritative, slightly rustic but intelligent, with just enough gravitas to be taken seriously.

Armed with this description, we headed to ElevenLabs to find our voice actor. The platform’s V3 Alpha compatible voices offered impressive options, and we auditioned several candidates against our character description. When we found the right match, it was immediately obvious, the voice fit our Bigfoot like a perfectly tailored suit.

Using ElevenLabs V3 Alpha, we were able to generate our voiceover with remarkable control over emphasis and delivery. The technology allowed us to direct the performance, adding natural inflections and emotional beats that brought the script to life. It was like having a voice actor who never got tired and could nail every take.

Scene by Scene: The Visual Production Pipeline

With our character and voice established, we needed to create the world around him. ChatGPT helped us break down our script into detailed scene descriptions, providing the kind of specific visual direction that would translate well to our various AI tools.

This is where things got really interesting from a production standpoint. We experimented with multiple platforms: ChatGPT for conceptual development, Veo3 for video generation, and MidJourney for static scenes. Each tool had its strengths, and finding the right application for each became part of our workflow optimization.

MidJourney consistently delivered the strongest starting points for our scenes. The image quality was exceptional, and the platform’s ability to interpret our detailed prompts gave us reliable results. However, we quickly discovered that MidJourney’s animation features didn’t provide the level of control we needed for professional work.

Veo3 offered better animation control, but our licensing restrictions meant we couldn’t feed in our custom images as starting points. This led to inconsistent results that didn’t match our established visual style. We needed a solution that could bridge the gap between MidJourney’s excellent stills and the motion we required.

The Animation Solution: Runway Gen-4 Takes Center Stage

Runway Gen-4 became our secret weapon for bringing static scenes to life. By feeding our MidJourney images into Runway, we could generate sophisticated animations while maintaining visual consistency with our character design. The platform excelled at creating atmospheric background movement, subtle character animations, and environmental effects that elevated our scenes from static to cinematic.

While Runway was undeniably powerful, we quickly learned that AI animation comes with its own set of challenges. Results were often unpredictable, some animations turned out stiff and unrealistic, while others simply didn’t match our input images or creative vision. It became a process of iteration, requiring multiple attempts to achieve shots that truly worked for our story. What looked promising in our static image didn’t always translate effectively into motion.

Still, for scenes without dialogue, Runway Gen-4 proved to be our go-to solution despite these hurdles. The tool’s ability to interpret and extend our visual concepts while maintaining quality made it indispensable for our tight timeline. We could generate multiple animation options quickly, then select the best results from several attempts for our final cut, a workflow that became essential given the unpredictable nature of the output.

However, speaking scenes required a different approach entirely.

Bringing Bigfoot to Life: HeyGen Avatar IV

For the portions of our AI commercial where Bigfoot needed to speak directly to the camera, we turned to HeyGen Avatar IV. This platform allowed us to upload our ElevenLabs-generated voiceover and create synchronized character animations that felt natural and engaging.

The level of control HeyGen provided was impressive. We could direct our character’s performance, adjusting gestures, expressions, and timing to match our creative vision. The technology seamlessly integrated our custom voice with believable character animation, creating moments that felt genuinely conversational rather than obviously AI-generated.

Watching our static Bigfoot character come to life and deliver our script with personality and charm was one of those magical production moments that reminded us why we love this work.

Post-Production: Bringing It All Together

With our visual elements generated, we entered familiar territory: post-production. Using AI-powered tools in Photoshop, we refined our MidJourney-created images: expanding edges, enhancing details, and adjusting composition to ensure consistency across scenes.

An interesting discovery emerged during this phase: certain elements, particularly signs with text or written messages, were actually easier to create using ChatGPT than MidJourney. Our workflow evolved into a hybrid approach: we’d generate the base image in MidJourney, then describe the scene and desired edits to ChatGPT for clearer, more contextually accurate text elements. These ChatGPT-generated components were then seamlessly integrated back into our Photoshop workflow, where we blended everything together for a polished final look.

Refining and Integrating Elements

Once all assets were refined, we moved into Premiere Pro, where traditional editing skills became crucial in tying our AI-generated elements into a cohesive final cut. The assembly process felt remarkably similar to conventional production workflows, just with different source materials. Our AI-generated elements cut together naturally, and the familiar rhythm of editing allowed us to craft the pacing and emotional beats essential for an effective commercial.

For the soundtrack, time constraints led us away from AI music generation and toward Envato’s proven library. This decision reinforced an important lesson: sometimes the hybrid approach, combining AI innovation with established, reliable resources, produces the strongest results.

The Power of Hybrid Workflows

What struck us most was how naturally our AI-assisted workflow integrated with traditional post-production techniques. The technology enhanced our creative process without replacing the fundamental storytelling skills that make commercial production effective.

The Verdict: 72 Hours Well Spent

Did we succeed in creating a compelling commercial entirely with AI tools in just three days? Absolutely. But more importantly, we learned that AI isn’t about replacing traditional production skills, it’s about augmenting them in exciting ways.

The technology allowed our Portland AI agency to explore creative directions we might never have considered, iterate rapidly without the usual resource constraints, and produce high-quality content on an aggressive timeline. However, success still required human creative judgment, technical problem-solving, and the kind of storytelling instincts that come from years of production experience.

Our Bigfoot commercial proved that AI tools like MidJourney, ElevenLabs, Runway, and HeyGen are ready for professional creative work, but they’re most powerful when wielded by people who understand both the possibilities and the limitations. Here in the Pacific Northwest, we’re seeing more agencies embrace this hybrid approach, combining AI innovation with traditional creative expertise.

As we wrapped up our 72-hour sprint, one thing was clear: Bigfoot might be mythical, but the potential of AI tools like MidJourney, ElevenLabs, Runway, and HeyGen in creative production is very real. For agencies ready to embrace this technology, the future isn’t human versus AI: it’s human with AI, and that future is incredibly exciting.

‍

Why Your AI Automation Keeps Failing (And What LangGraph Does Differently)

Mark Nguyen — Sat, 14 Nov 2026 00:51:17 +0000

Originally published on the SLIDEFACTORY blog.

You’ve seen the demo. An AI agent that drafts emails, pulls data, researches competitors, and drops a clean report into Slack, all without anyone touching a keyboard. It’s impressive. It’s fast. It works perfectly under the conditions it was built in.

Then it goes live.

Users give it messy inputs. Something times out. The AI produces a confident wrong answer and nobody catches it until a client does. The workflow that wowed the room six weeks ago now needs a person hovering over it full-time, which defeats the whole point.

This isn’t a niche problem. MIT’s 2025 study on AI implementation found that 95% of AI pilots deliver zero measurable business impact. Not low ROI, zero. And in almost every case, the model wasn’t the problem. The architecture underneath it was.

So what’s actually breaking? And what does it look like when it’s built to hold up? That’s what this article is about.

What’s Actually Failing

Most AI workflow failures aren’t random. They cluster around the same handful of problems.

The workflow forgets where it was. When a multi-step AI process gets interrupted (a timeout, an error, a server restart) most basic implementations start over from scratch or fail silently. There’s no memory of what already happened. A seven-step process that fails at step five has to begin again at step one, or it just stops.

Real processes don’t move in straight lines. A customer support workflow might need to check a knowledge base, draft a response, escalate if the issue is sensitive, wait for a manager’s approval, and then send. Most early AI builds model this as a simple chain: A then B then C. When reality requires a branch (“if X, do Y instead of Z”) the chain breaks.

Errors have nowhere to go. A basic AI pipeline that hits a problem either crashes or silently produces bad output. There’s no built-in retry logic, no fallback, no way to route an exception to a human for review. So either the whole thing stops, or wrong work gets done without anyone knowing.

You can’t see what’s happening. When something goes wrong in an AI workflow, figuring out which step failed, why, and what the AI was working with at that moment is often impossible. You know the output was wrong. You don’t know where it went wrong.

The demo environment wasn’t real. Demo data is clean and predictable. Demo users are patient. Demo load is low. Production is the opposite of all three, and most initial AI builds aren’t designed for it.

These aren’t technology failures. They’re architecture failures. The AI could do the job, but nobody built the scaffolding required to make it reliable.

What “Built Right” Actually Looks Like

The tools that address this most directly are LangChain and LangGraph, two frameworks from the same team that solve different parts of the problem.

LangChain is the toolkit. It handles the connective tissue: linking an AI model to your data sources, your APIs, your databases. It gets a working prototype built fast, which is genuinely useful. You want to see something running early so you can evaluate whether the approach is right before committing to a full build.

LangGraph is what you add when the prototype needs to become a real system. It’s an orchestration layer that controls how the AI workflow actually runs: how it moves between steps, what it remembers, what it does when something goes wrong, and where humans need to be involved.

The practical difference looks like this:

Situation	LangChain Alone	With LangGraph
Simple chatbot or document Q&A	Works fine	Overkill
Multi-step workflow with decision points	Gets complicated fast	Built for this
Workflow needs to pause for human approval	Requires custom workarounds	Built-in
Server restarts mid-task	State is lost	Picks up exactly where it left off
Multiple AI agents working together	Brittle to coordinate	Handled natively
Something breaks and you need to know why	Hard to diagnose	Full execution history available

Most workflows that are simple in a demo end up in the right column once they hit real conditions. That’s not a reason to over-engineer from day one, it’s a reason to know what you’re getting into and plan accordingly.

The One Problem That Breaks Everything: Memory

If there’s a single thing to understand about why AI workflows fail in production, it’s this: most of them have no persistent memory between steps.

Think of it like an employee who forgets everything the moment they look away from the screen. Every step in the workflow starts from zero. The AI doesn’t remember what the previous step produced, what decisions were made, or where in the process it was when something interrupted it.

According to the LangChain State of Agent Engineering report, durability and state management are the top production challenges teams face in 2026, and it’s exactly what LangGraph’s automatic checkpointing is built to address. Every time the workflow moves from one step to the next, the current state is saved. If anything interrupts the process (a timeout, a crash, a deliberate pause) the workflow can resume from exactly that point. Same context. Same position. Nothing lost.

For workflows that run for minutes or hours, or that span multiple sessions with a user, this isn’t optional. It’s the difference between a system that works in the real world and one that only works in ideal conditions.

When the AI Should Stop and Ask

One of the most common architecture mistakes we see is building AI workflows that are designed to run to completion without any human input. That works fine for low-stakes, fully automatable tasks. It’s the wrong design for almost anything that touches a customer, a contract, or a decision with real consequences.

Real business processes have checkpoints. A workflow that researches a lead and drafts an outreach email probably shouldn’t send that email automatically, a person should review it first. A workflow that processes invoices might handle 90% of them automatically, but flag the outliers for human review before anything gets paid.

LangGraph has built-in support for this pattern. The workflow can pause at any point, hand off to a person for review or approval, and then resume from exactly that point once the human has weighed in. The AI handles the volume. The human handles the judgment calls.

This pattern shows up consistently in well-built marketing and operations workflows, systems that run autonomously most of the time, but route edge cases to a person automatically rather than guessing. The result is teams that actually trust the output, because they know the AI isn’t making calls it shouldn’t be making on its own.

When One AI Isn’t the Right Answer

For straightforward workflows, a single AI handling everything is fine. For more complex processes, it’s often the wrong architecture.

A single AI handling research, quality review, drafting, routing, and approval logic all at once ends up with a bloated context window, inconsistent results, and debugging that’s nearly impossible. The more you ask one AI to juggle, the worse it does at each individual thing.

The better pattern for complex processes is to break the work into specialized agents, each with a clear job. One handles research. One drafts. One reviews for quality before anything leaves the system. LangGraph coordinates them: when the research step finishes, the output gets routed to the drafting step automatically. If the review step flags a problem, it goes back for revision instead of moving forward.

This produces better outputs, makes the system easier to debug, and lets you improve one part of the workflow without touching the rest. LangGraph’s open-source repository on GitHub includes working examples of multi-agent patterns you can reference directly.

So Where Does That Leave You?

If you’ve tried AI automation and hit a wall: tools that worked in the demo but required constant babysitting in practice, workflows that failed in ways nobody could diagnose, results too inconsistent to trust, the problem almost certainly wasn’t the AI model. It was the architecture.

The teams getting durable results from AI automation in 2026 aren’t the ones using the most tools. They’re the ones that built their workflows with state management, human checkpoints, and real error handling, the infrastructure that makes AI reliable rather than just impressive.

LangGraph is a powerful framework, but it’s not a drag-and-drop tool. Getting it right takes architectural thinking alongside the technical implementation, and knowing which tool is actually the right fit before you start building.

How are you handling state and human review in your own agent workflows? I’d like to hear what’s worked and what hasn’t.