DEV Community: Samcorp

Our Accessibility Audit Found 340 Issues Here's the Pattern

Samcorp — Fri, 04 Sep 2026 09:44:02 +0000

We expected the accessibility audit to produce a long list of unrelated problems.

It found 340 issues.

But the interesting part was not the number.

It was the pattern.

A small group of recurring mistakes accounted for most of the failures.

340 findings
     ↓
A handful of repeated patterns
     ↓
A much smaller number of root causes

That changed how we approached the fixes.

Instead of repairing 340 problems individually, we started fixing the components and design decisions creating them.

Note: The counts in this post are representative examples based on a typical mixed automated/manual audit. Replace them with your actual audit results before presenting them as project data.

What the Audit Found

We grouped the 340 findings by root cause.

Issue Type	Findings	Share
Color contrast	112	32.9%
Form labels	63	18.5%
Image alternative text	49	14.4%
Keyboard and focus	38	11.2%
Links and button names	31	9.1%
Heading and page structure	27	7.9%
Other issues	20	5.9%
Total	340	100%

The distribution surprised us.

Nearly two-thirds of the findings came from just three areas:

Contrast
+
Forms
+
Images

This was not unusual.

The 2026 WebAIM Million study found low-contrast text on 83.9% of analyzed home pages, missing image alternative text on 53.1%, and missing form labels on 51%. WebAIM also found that six common error types represented 96% of all automatically detected errors.

Our audit reinforced the same lesson:

Accessibility problems often repeat because the underlying implementation repeats.

Pattern #1: Contrast Problems Were Mostly Design-System Problems

The largest category contained 112 findings.

At first they looked like individual color mistakes.

.card-description {
  color: #a6a6a6;
  background: #ffffff;
}

But the same colors appeared across cards, tables, helper text, placeholders, secondary buttons, and empty states.

Fixing every page individually would have been the wrong approach.

The real problem lived in the design tokens.

Instead of changing dozens of components, we changed the shared semantic color definitions and then retested the components that depended on them.

Fixing shared components and design tokens is also part of building scalable web platforms, because one accessible component can improve dozens of pages at once.

The lesson was:

When the same accessibility issue appears everywhere, look for the shared system creating it.

WCAG 2.2 includes minimum contrast requirements for text as well as requirements for visual information used to identify interface components and states.

Pattern #2: Forms Looked Labeled but Were Not Programmatically Labeled

The next 63 findings involved forms.

Visually, many fields looked perfectly understandable:

Email Address
[________________]

But some implementations were closer to:

<span>Email Address</span>
<input type="email">

A sighted user could understand the relationship.

Assistive technology could not necessarily determine it.

We replaced patterns like that with explicit labels:

<label for="email">Email Address</label>
<input id="email" type="email">

Icon-only controls needed accessible names too:

<button aria-label="Clear search">
  <svg aria-hidden="true">...</svg>
</button>

This category taught us that accessibility cannot be judged from screenshots alone.

The visual interface and the accessibility tree are different representations of the same product.

WCAG's Name, Role, Value requirement exists so user interface components can expose information that assistive technologies can understand.

Pattern #3: `alt` Text Was Not Just Missing — It Was Often Wrong

We found 49 image-related issues.

Some images had no alt attribute.

Others technically had one but used values like:

<img src="chart.png" alt="image">

or:

<img src="team-photo.jpg" alt="team-photo.jpg">

Those technically contain text.

They do not necessarily communicate anything useful.

We started classifying images by purpose.

Informative images received meaningful descriptions.

Decorative images used:

alt=""

Functional images needed text that described the action or destination rather than simply describing how the image looked.

This distinction mattered more than simply asking:

Does every image have an alt attribute?

The better question was:

Does the text alternative provide the information this image contributes?

WebAIM's 2026 analysis found that 16.2% of images on the million home pages lacked alternative text, and it also identified large numbers of questionable or repetitive alternative text.

Pattern #4: Mouse Testing Hid Keyboard Problems

Thirty-eight findings appeared only when we stopped using the mouse.

We pressed Tab.

Suddenly we found:

Dropdowns that could not be opened

Dialogs that trapped focus

Clickable divs that keyboard users could not activate

Focus indicators that disappeared

Focus moving somewhere unexpected

One common anti-pattern looked like this:

<div onclick="openMenu()">
  Menu
</div>

The better solution was often simply:

<button type="button" onclick="openMenu()">
  Menu
</button>

Native HTML already provides keyboard behavior, semantics, and focus support.

We had occasionally rebuilt those features ourselves and produced worse results.

WCAG 2.2 requires functionality to be operable through a keyboard, and its Focus Visible criterion requires users to be able to see where keyboard focus currently is.

Pattern #5: Icon Buttons Were Easy to See but Hard to Understand

Thirty-one findings involved empty links, empty buttons, or controls without useful accessible names.

Our interfaces contained many controls like:

<button>
  <svg>...</svg>
</button>

A sighted user saw a trash icon.

A screen reader could encounter an unnamed button.

The fix might be:

<button aria-label="Delete project">
  <svg aria-hidden="true">...</svg>
</button>

But the larger lesson was not “add ARIA everywhere.”

It was:

Every interactive control needs a meaningful name.

Where native visible text could do the job, we preferred that.

ARIA was useful when the visual design genuinely required an icon-only control.

Pattern #6: Heading Problems Came From Styling Decisions

We found 27 structural issues.

Some pages jumped from:

H1
 ↓
H3
 ↓
H5

because developers chose heading elements based on their default visual size.

That is backwards.

We changed the approach:

<h2 class="small-heading">
  Account Settings
</h2>

instead of choosing <h4> simply because it visually looked smaller.

CSS controls appearance.

HTML should describe structure.

This matters because headings are an important navigation mechanism for screen-reader users. The 2026 WebAIM study found skipped heading levels on 41.8% of analyzed home pages.

The Most Important Finding Was Duplication

Once we grouped the 340 issues, something became obvious.

Many findings were not 340 separate bugs.

For example:

21 contrast issues
→ one incorrect text token

17 missing labels
→ one shared form component

12 unnamed buttons
→ one icon-button component

9 focus problems
→ one modal implementation

Suddenly the remediation strategy changed.

Instead of:

Find issue
   ↓
Fix page
   ↓
Find next issue

we moved toward:

Find pattern
   ↓
Find shared component
   ↓
Fix root cause
   ↓
Regression test
   ↓
Remove many findings at once

That was the real value of the accessibility audit.

It showed us where accessibility debt was being generated.

Automated Testing Was Useful — but It Was Not the Audit

Automated tools found many obvious problems quickly:

Contrast

Missing labels

Missing alt attributes

Empty links

Empty buttons

Some ARIA problems

But automation could not answer every important question.

It could not reliably tell us whether:

The keyboard flow made sense

Alternative text was actually useful

A modal announced itself correctly

Focus moved to the right place

Error messages were understandable

A workflow was usable with a screen reader

That is why our audit combined automated checks with keyboard testing and manual review.

WebAIM makes the same limitation explicit: automated tools detect only a subset of accessibility failures, and having zero automatically detected errors does not prove that a page conforms to WCAG.

What We Changed After the Audit

The biggest improvement was not fixing those 340 findings.

It was changing how new findings were prevented.

Accessibility checks moved earlier into:

Design
   ↓
Component development
   ↓
Code review
   ↓
Automated testing
   ↓
Manual keyboard testing
   ↓
Release

Shared components became particularly important.

If we could make one button, form field, dialog, or navigation component accessible by default, dozens of future screens benefited automatically.

Accessibility stopped being something we checked at the end.

We started treating accessible UI/UX design as part of the product system itself, so contrast, component behavior, hierarchy, and WCAG requirements were considered before development reached QA.

It became part of how the component system was built.

The Data Changed Our Priorities

Before the audit, accessibility felt like a huge collection of rules.

After grouping the findings, the work became more concrete.

We did not need to solve every possible accessibility problem simultaneously.

We needed to stop repeatedly shipping the same problems.

The data showed where to start:

Fix contrast tokens.

Fix form components.

Fix image handling.

Fix keyboard interaction.

Fix accessible names.

Fix document structure.

That was much more actionable than saying:

Make the website accessible.

The Biggest Lesson

The most useful number from the audit was not:

340 issues

It was:

A few patterns created most of them.

That changes accessibility work from an endless cleanup exercise into an engineering problem.

If one component creates 25 failures, fix the component.

If one design token creates 40 contrast problems, fix the token.

If one modal pattern breaks keyboard navigation across the application, fix the pattern.

Accessibility scales when the correct behavior becomes the default behavior.

Final Takeaway

An accessibility audit should not finish with a spreadsheet containing hundreds of rows.

That spreadsheet is the beginning.

Group the findings.

Look for repetition.

Trace repeated failures back to shared components, design tokens, and engineering conventions.

Then fix the systems creating the issues.

Our 340 findings looked overwhelming at first.

Once we analyzed the data, the problem became much simpler:

We did not have 340 independent accessibility problems. We had a small number of accessibility patterns repeated across the product.

And repeated problems are exactly the kind of thing engineering systems are good at fixing.

What Broke When We Adopted React Server Components

Samcorp — Fri, 04 Sep 2026 09:30:29 +0000

We adopted React Server Components expecting a fairly simple win:

Less client JavaScript
        ↓
More server-side data fetching
        ↓
Faster pages

Some of that happened.

But the migration also broke assumptions that had quietly existed across our frontend for years.

Components that used to be interchangeable suddenly belonged to different environments. Shared objects could no longer always cross component boundaries. Data fetching moved closer to the database but sometimes became slower because we created waterfalls.

The biggest lesson from our React Server Components issues was simple:

RSC was not just a rendering optimization. It changed the architecture of the application.

Note: This is a representative engineering post-mortem. The examples describe common RSC migration problems rather than a specific customer deployment.

What We Expected

Our previous mental model was straightforward:

Server
  ↓
HTML + JavaScript
  ↓
React in the browser
  ↓
API calls

With Server Components, the architecture became closer to:

Server Components
       ↓
RSC Payload
       ↓
Client Components
       ↓
Browser interaction

The benefit was obvious.

Code that only needed to run on the server no longer needed to become browser JavaScript.

But we underestimated how important the boundary between those two environments would become.

Break #1: `"use client"` Spread Further Than Expected

Our first problem looked harmless.

A component needed state:

import { useState } from "react";

export default function Filters() {
  const [open, setOpen] = useState(false);

  return (
    <button onClick={() => setOpen(!open)}>
      Filters
    </button>
  );
}

Server Components cannot maintain client-side state or register browser event handlers, so the component needed:

"use client";

That part was expected.

What surprised us was what happened to its dependencies.

Once a module becomes part of the client component graph, code imported beneath that boundary can also become client-side code.

So this:

Page
 ↓
Dashboard
 ↓
Filters
 ↓
Utilities
 ↓
Formatting library

could accidentally turn more code into browser JavaScript than we intended.

React's documentation explicitly describes "use client" as a boundary in the module dependency tree, not merely a marker for one component.

This is where scalable web development becomes less about component count and more about defining clean boundaries between server-rendered logic and client-side interactivity.

What We Changed

We stopped placing "use client" high in the tree.

Instead of:

Dashboard
  "use client"
      ↓
Everything below it

we moved the boundary downward:

Dashboard        Server
├── Header       Server
├── Report       Server
└── Filters      Client

The rule became:

Keep the interactive island as small as practical.

Break #2: Browser Assumptions Started Failing

Years of React development had left browser assumptions everywhere.

Components accessed:

window
document
localStorage
navigator

Some third-party libraries also expected a browser environment immediately during module evaluation.

Then those components moved into server-rendered code.

Suddenly we saw errors around unavailable browser APIs.

The solution was not:

Add "use client" everywhere.

That would have removed much of the reason for adopting Server Components.

Instead, we separated concerns.

For example:

ProductPage
   │
   ├── ProductDetails      Server
   ├── Recommendations    Server
   └── RecentlyViewed     Client

Only the feature that genuinely needed localStorage remained client-side.

Lesson

A component needing browser APIs is not a failure.

But it needs an explicit client boundary.

React's current documentation makes the same distinction: Server Components cannot use interactive browser behavior or most stateful Hooks, while Client Components handle that work.

Break #3: Our Shared Objects Stopped Crossing the Boundary

Before RSC, we passed rich JavaScript objects around freely.

For example:

class Money {
  constructor(amount, currency) {
    this.amount = amount;
    this.currency = currency;
  }

  formatted() {
    return `${this.currency} ${this.amount}`;
  }
}

Then we tried:

<ProductPrice price={new Money(99, "USD")} />

across a server-to-client boundary.

That exposed another rule.

Values passed from Server Components into Client Components need to be serializable.

Plain data works well.

Custom class instances do not.

React's documented serializable types include primitives, arrays, maps, sets, dates, plain objects, and several other supported values—but ordinary class instances are not supported across that boundary.

What We Changed

Instead of passing domain objects:

new Money(99, "USD")

we passed DTO-style data:

{
  amount: 99,
  currency: "USD"
}

Then the client handled presentation.

This pushed us toward a cleaner rule:

Server domain model
      ↓
Serializable view model
      ↓
Client component

It was annoying at first.

Architecturally, it ended up being healthier.

Break #4: Moving Data Fetching to the Server Created Waterfalls

One reason we adopted Server Components was cleaner data fetching.

Instead of:

useEffect(() => {
  fetch("/api/orders");
}, []);

we could write:

async function Orders() {
  const orders = await getOrders();

  return <OrderList orders={orders} />;
}

Much nicer.

Then we accidentally wrote pages like this:

const user = await getUser();
const orders = await getOrders();
const recommendations = await getRecommendations();

If those requests were independent, we had created a waterfall.

getUser()
   ↓
wait
   ↓
getOrders()
   ↓
wait
   ↓
getRecommendations()

The code looked clean.

The request was slow.

Next.js documentation explicitly warns about accidental sequential data fetching and recommends parallelizing independent work when possible.

The Fix

Where requests were independent:

const [
  user,
  orders,
  recommendations
] = await Promise.all([
  getUser(),
  getOrders(),
  getRecommendations()
]);

And where parts of the page could arrive later, we used Suspense and streaming rather than blocking the whole route.

Page shell
   ↓
Important content
   ↓
──────── streamed later ────────
   ↓
Recommendations

Streaming lets slower parts of a route arrive separately instead of blocking everything above them.

Lesson

Server-side data fetching does not automatically mean fast data fetching.

The dependency graph still matters.

Break #5: Caching Became Part of Application Correctness

This issue was framework-specific rather than an RSC rule itself.

Once more rendering happened on the server, our framework's caching and revalidation behavior suddenly mattered much more.

Previously, developers often thought:

Call API
   ↓
Receive current data

Now we needed to ask:

Was this rendered dynamically?

Was the result cached?

When does it expire?

What invalidates it?

Should this page be static at all?

The bugs were frustrating because stale data often looked valid.

A product page displaying yesterday's inventory does not throw an exception.

It simply lies quietly.

What We Changed

Every important data source received an explicit caching decision:

Product description
→ cache aggressively

Current inventory
→ fresher policy

User session
→ request-specific

Pricing
→ explicit business TTL / invalidation

We stopped treating caching as a performance detail.

It became part of data correctness.

Break #6: Server Functions Looked Safer Than They Actually Were

Server Functions made mutations elegant.

Conceptually:

"use server";

export async function updateOrder(orderId) {
  // update order
}

Then a Client Component could invoke that function without manually building another traditional API route.

The abstraction felt safe because the function lived on the server.

That is not enough.

React's documentation explicitly warns that arguments sent to Server Functions are fully client-controlled and that authorization must be performed inside the server-side operation.

This is wrong:

"use server";

export async function deleteInvoice(invoiceId) {
  await db.invoice.delete({
    where: { id: invoiceId }
  });
}

A better pattern is closer to:

"use server";

export async function deleteInvoice(invoiceId) {
  const user = await requireUser();

  const invoice = await db.invoice.findUnique({
    where: { id: invoiceId }
  });

  if (!canDeleteInvoice(user, invoice)) {
    throw new Error("Unauthorized");
  }

  await db.invoice.delete({
    where: { id: invoiceId }
  });
}

Lesson

"use server" tells React where code executes.

It does not mean:

trusted
authorized
validated
safe

Those are still our responsibilities.

Break #7: Debugging Moved Away From the Browser

Before the migration, developers were used to finding most problems in:

Browser DevTools
Console
Network tab
Client stack trace

After RSC, an error might occur during:

server rendering
database access
RSC serialization
Server Function execution
stream generation

The browser sometimes only showed the result of that failure.

The useful stack trace lived on the server.

That changed our observability requirements.

We added clearer logging around:

Route
Request ID
User/session context
Server component
Data fetch
Server Function
Duration
Failure

Our mental model changed from:

This is a frontend application.

to:

This is a distributed full-stack application that happens to use React.

Break #8: Our Tests Were Testing the Wrong Architecture

We had many tests built around client-rendered React.

They assumed components could:

Use browser APIs
Call mocked REST endpoints
Run entirely inside a DOM test environment

Server Components changed those assumptions.

A component like:

async function AccountPage() {
  const account = await db.account.findFirst();

  return <Account account={account} />;
}

is no longer just a UI function.

It touches a server-side dependency.

So we separated testing into layers.

Server Logic

Test:

queries
authorization
transformations
Server Functions

Client Components

Test:

interaction
state
browser behavior
accessibility

Integration

Test:

Server Component
      ↓
Client boundary
      ↓
mutation
      ↓
updated UI

Trying to force all three into the old frontend test strategy created unnecessary pain.

We also started treating the migration as a broader software testing and QA problem, with separate coverage for server logic, client interaction, integration behavior, performance, and regression risk.

One More Lesson: RSC Added a New Security Patch Surface

There was another operational lesson we could not ignore.

React disclosed a critical React Server Components vulnerability in December 2025, followed by additional denial-of-service and source-exposure issues. React recommended immediate upgrades to patched packages.

That changed our dependency policy.

RSC infrastructure was no longer something we could treat like passive frontend tooling.

It was server infrastructure exposed to requests.

We added:

React/RSC security advisories
       ↓
Dependency review
       ↓
Framework compatibility check
       ↓
Rapid patch rollout

The broader lesson was not that Server Components are inherently unsafe.

It was:

Once React becomes part of your server execution path, React security updates become production server updates.

What Actually Worked

After the rough migration, several rules made the architecture much easier to reason about.

1. Server by Default

Components stayed server-side unless they genuinely required:

state
effects
events
browser APIs

2. Small Client Islands

We pushed "use client" as far down the component tree as practical.

3. Serializable Boundaries

Server Components returned simple, intentional view models to Client Components.

4. Parallel Data Fetching

Independent server requests started together.

5. Suspense Around Slow Work

Slow data stopped blocking unrelated UI.

6. Explicit Cache Decisions

Every important data source had a freshness policy.

7. Authorization Inside Server Functions

Never at the button.

Never only in the UI.

8. Server Observability

Logs and tracing became part of frontend debugging.

The Architecture We Ended Up With

Before:

React Client
     ↓
API
     ↓
Backend
     ↓
Database

After:

                 Server
                   │
        ┌──────────┴──────────┐
        ▼                     ▼
 Server Components       Data Layer
        │                     │
        └──────────┬──────────┘
                   ▼
              RSC Payload
                   │
                   ▼
            Client Islands
                   │
                   ▼
            User Interaction
                   │
                   ▼
           Server Functions

Neither architecture is automatically better.

But the second requires much more intentional boundary design.

What We Would Do Differently

If we adopted RSC again, we would start with boundaries rather than components.

First: Map Interactive Areas

Identify everything requiring:

useState
useEffect
events
browser APIs

Second: Map Server-Only Data

Identify:

database access
secrets
internal APIs
authorization

Third: Define the Boundary

Decide what data is allowed to cross from server to client.

Fourth: Inspect Data Dependencies

Find sequential fetches before they become waterfalls.

Fifth: Define Cache Rules

Do not wait for stale-data bugs.

Sixth: Rebuild the Test Strategy

Treat server logic and browser interaction as different layers.

Seventh: Add Server Observability

Before production traffic arrives.

The Biggest Lesson

The biggest mistake was treating React Server Components as:

React
+
better server rendering

The better model was:

React
+
two execution environments
+
a network boundary
+
serialization
+
server data access
+
client interactivity
+
new security responsibilities

That explains most of the React Server Components issues we encountered.

The code still looked like React.

The architecture was no longer purely frontend.

Final Takeaway

React Server Components can be valuable.

They can reduce client-side JavaScript, keep server-only logic off the browser, and make server data access feel natural.

But adopting them successfully requires asking a different question.

Not:

Which components should we convert to Server Components?

Ask:

Where should this code execute?

Which code really needs the browser?

What data is allowed to cross the boundary?

Which requests can run in parallel?

What must stay fresh?

Where is authorization enforced?

Where will failures appear?

Once we started answering those questions explicitly, the migration became much easier.

The biggest lesson was simple:

React Server Components are not just a component feature. They are an application-boundary decision.

Five AI Features We Built and Then Deleted

Samcorp — Thu, 03 Sep 2026 06:21:08 +0000

We shipped five AI features that worked, Then we deleted them.

Not because the models crashed. Not because the APIs stopped responding. And not because AI suddenly became useless.

The features simply failed to create enough value to justify the complexity they introduced.

That distinction changed how we think about AI feature failure.

A feature can be technically impressive and still be the wrong product decision.

Note: The examples below are representative engineering scenarios intended to explain common AI product failures. They are not claims about a specific customer or production deployment.

The Features Looked Good in Demos

This was the dangerous part.

Every feature had a convincing demo.

Prompt
  ↓
Model
  ↓
Useful-looking output
  ↓
Everyone says, "That's cool."

The problems appeared later.

Real users brought incomplete data, ambiguous requests, unusual workflows, latency expectations, and very little patience for confidently wrong answers.

By the end, five features had taught us five different lessons.

Feature	Why We Removed It
AI Search Answers	Users trusted incorrect summaries too easily
AI Code Review Bot	Too much low-value feedback
Automatic Support Replies	Confidence was hard to measure
AI Dashboard Insights	It described correlation like causation
Universal AI Assistant	Cost and complexity grew faster than usage

1. AI Search Answers

The idea was straightforward.

Instead of making users search documentation manually, we would retrieve relevant documents and generate a direct answer.

Question
   ↓
Vector Search
   ↓
Relevant Documents
   ↓
LLM
   ↓
Answer + Sources

The prototype worked surprisingly well.

Then we tested messy queries.

Sometimes retrieval found the correct document, but the model summarized it incorrectly.

Sometimes the answer was correct, but the citation pointed to a document that only partially supported it.

Sometimes two documents contained conflicting information, and the model quietly chose one.

The dangerous failures were not obviously ridiculous.

They looked reasonable.

That made them harder to detect.

We added retrieval scores, citation validation, better prompts, and fallback behavior. Accuracy improved, but the product problem remained: users often treated a fluent answer as authoritative.

NIST specifically highlights the risk of people over-relying on AI output, sometimes referred to as automation bias.

We eventually removed the generated answer from the primary search experience.

Search stayed.

Retrieval stayed.

The AI-written conclusion did not.

Lesson

Sometimes the better AI feature is:

Find the right information

instead of:

Interpret the information for the user

2. The AI Code Review Bot

This feature was popular internally at first.

Every pull request received automated review comments.

The bot looked for:

Potential bugs
Security concerns
Missing edge cases
Naming problems
Complex functions
Test gaps

Technically, it worked.

Operationally, it became annoying.

One pull request might receive eight comments.

Perhaps one mattered.

Developers quickly learned to skim the entire review because most comments were suggestions rather than actual defects.

That created a strange failure mode.

The bot was producing more review activity while reducing attention.

After enough low-value comments, even the useful warning looked like noise.

We tried raising confidence thresholds.

Then we restricted the bot to security and correctness issues.

That helped, but another problem appeared: deterministic checks were often better handled by existing tools.

A linter should catch formatting.

Static analysis should catch known patterns.

Tests should verify known behavior.

We also started treating model behavior like software testing and QA, with clear test cases, edge conditions, regression checks, and measurable failure criteria.
This is a natural fit because the SDLC Corp testing page covers functional, automation, performance, security, API, and AI-powered QA

Using an LLM for problems that deterministic tools already solved added cost and uncertainty.

So we deleted the general-purpose review bot.

Lesson

AI should not replace reliable deterministic automation simply because AI can produce an answer.

Use models where judgment is genuinely required.

3. Automatic Customer Support Replies

This was the feature everyone expected to save the most time.

Incoming ticket:

"My invoice shows two charges."

AI:

"I can help with that. It appears..."

The first version generated drafts.

Agents reviewed them before sending.

That worked reasonably well.

Then came the obvious question:

If agents approve most responses, why not send them automatically?

That is where the risk changed.

A draft can be imperfect.

An automatically sent customer response needs a much higher reliability threshold.

The model occasionally misunderstood account context.

Sometimes it made promises the support team could not keep.

Sometimes the wording sounded confident despite incomplete information.

We tried confidence scoring.

But what exactly was confidence?

Confidence the model understood the question?

Confidence retrieval found the correct account data?

Confidence the policy was interpreted correctly?

Confidence the response was safe to send?

One number could not represent all of those things.

OpenAI's current evaluation guidance makes a similar broader point: teams need to define what success means and test against representative real-world conditions rather than relying on vague quality judgments.

We removed auto-send.

The draft assistant survived.

Lesson

Human review is not always temporary scaffolding.

Sometimes it is the correct product architecture.

4. AI-Generated Dashboard Insights

This feature sounded excellent in planning meetings.

Instead of showing users a chart, we would explain the chart.

For example:

Revenue decreased 14% this month.

Possible explanation:
Customer churn increased in the enterprise segment.

Users loved the idea.

The problem was the phrase:

Possible explanation.

The model was excellent at producing plausible narratives.

But dashboards mostly contained observational data.

A change happening beside another change did not mean one caused the other.

The feature could turn:

Metric A decreased.

Metric B increased.

into:

Metric A decreased because Metric B increased.

That sentence sounded useful.

It was often unsupported.

We changed the system so the model could only describe facts visible in the data.

That produced safer outputs such as:

Revenue decreased 14%.

Enterprise churn increased from 3.2% to 4.1%
during the same period.

Much better.

But users told us something unexpected.

They could already see those numbers on the dashboard.

We had built a sophisticated system to narrate information that was already visible.

The AI was no longer dangerous.

It was simply unnecessary.

So we removed it.

Lesson

Reducing risk can sometimes reduce an AI feature until very little unique value remains.

That is useful information.

Delete it.

5. The Universal AI Assistant

This was our most ambitious feature.

A floating assistant appeared throughout the application.

Users could ask:

"Find this customer."

"Explain this report."

"Create a ticket."

"Summarize these records."

"Update this status."

"Where is this setting?"

One assistant.

Everything connected.

It looked like the future.

It also created an enormous engineering surface.

Every new capability required:

Tool definition
Permissions
Prompt updates
Evaluation cases
Error handling
Observability
Security review
UI behavior
Fallback logic

Latency also became inconsistent.

A simple question might require one model call.

Another request might need:

LLM
 ↓
Tool call
 ↓
Database
 ↓
LLM
 ↓
Second tool
 ↓
LLM

Then came permissions.

The assistant needed to understand not only:

Can this tool update invoices?

but also:

Can this particular user update this particular invoice in this particular state?

Eventually, we looked at usage.

Users were not asking for everything.

Most requests clustered around a few narrow workflows.

We replaced the universal assistant with focused AI features inside those workflows.

Less magical.

Much easier to understand.

Much easier to test.

Lesson

A narrow AI feature that solves one frequent problem can be more valuable than a universal assistant capable of doing twenty things.

What These Failures Had in Common

At first, we blamed different technical problems.

Search had retrieval issues.

Code review had noise.

Support had reliability problems.

Analytics had grounding problems.

The assistant had complexity problems.

But underneath them was the same mistake:

We started with what the model could do instead of what the user actually needed.

The early question had been:

Can AI do this?

The better questions became:

Does this problem need AI?

Is AI better than the existing workflow?

How will we measure success?

What happens when the answer is wrong?

Will users understand the uncertainty?

Does the value justify the operational cost?

That changed our feature-development process more than any prompt technique did.

We Changed How We Evaluate AI Features

We no longer consider a convincing prototype strong evidence.

A prototype tells us:

This is possible.

It does not tell us:

This should become a product feature.

Now we define the evaluation before building too far.

For example, an AI support feature might need to improve response time without increasing incorrect policy answers.

An AI review feature might need to detect meaningful defects without producing excessive low-value comments.

An AI search feature might need answers that are both correct and supported by retrieved evidence.

A strong AI consulting process should define the use case, success metrics, validation approach, and business value before a prototype becomes a production feature.
This fits well because SDLC Corp’s AI consulting page specifically covers use-case planning, model validation, explainability, and deployment strategy.

This reflects a broader principle behind modern AI evaluation:

Specify the desired behavior, measure it on representative cases, and use failures to drive improvement.

Deleting the Feature Was Sometimes the Best Result

Engineering teams naturally dislike deleting work.

A feature may have taken weeks.

The architecture may be elegant.

The prompt may have gone through twenty versions.

The evaluation harness may be excellent.

None of that makes the feature valuable.

One of the healthiest decisions an AI team can make is:

This works, but it should not exist.

Deleting a feature does not necessarily mean the experiment failed.

Sometimes the experiment answered the most important question.

And the answer was no.

The Biggest Lesson

Our biggest AI feature failure was not hallucination.

It was confusing capability with usefulness.

Modern models can summarize, classify, generate, search, call tools, and interpret large amounts of information.

That makes it surprisingly easy to create impressive prototypes.

The difficult part is deciding which capabilities deserve to become permanent product features.

The standard became higher:

Not:
Can the model do this?

But:
Does this make the product meaningfully better?

That question deleted five features.

It also improved the ones we kept.

Final Takeaway

If you are building AI features, do not only keep a list of what shipped.

Keep another list:

What did we test?

What failed?

What did users ignore?

What produced too much risk?

What was technically successful but unnecessary?

What did we delete?

Those decisions are part of AI engineering too.

Sometimes the best evidence that an AI team is learning is not the number of features it launches.

It is the number it is willing to remove.

What Broke When We Moved From GPT to Self-Hosted

Samcorp — Thu, 03 Sep 2026 05:37:10 +0000

Moving from a hosted GPT API to a self-hosted model sounded straightforward.

The application already had one clean interface:

Prompt
  ↓
LLM API
  ↓
Response

So the migration plan looked simple:

Replace API endpoint
        ↓
Change model name
        ↓
Deploy

The endpoint worked.

The model answered.

And then almost everything around it started behaving differently.

The lesson from our self hosted LLM migration was simple:

Replacing the model was easy. Replacing the behavior we had built around the model was not.

Note: This is a representative engineering post-mortem. The failures and architecture below illustrate common migration problems rather than a specific customer deployment.

Why We Wanted to Self-Host

The reasons were reasonable.

We wanted more control over:

Model selection
Infrastructure
Deployment location
Data handling
Inference configuration
Capacity planning
Model upgrades

We also wanted the ability to optimize around our own workload instead of treating inference as an external black box.

So we introduced a self-hosted inference server behind the existing application.

Conceptually, the architecture changed from:

Before

Application
    ↓
Hosted GPT API

to:

After

Application
    ↓
LLM Gateway
    ↓
Self-Hosted Model Server
    ↓
GPU Infrastructure

The API looked familiar.

That created false confidence.

Break #1: API Compatibility Was Not Behavior Compatibility

One of our first assumptions was:

If the server supports an OpenAI-compatible API, our application should behave the same.

At the HTTP level, that was mostly true.

For example, servers such as vLLM expose OpenAI-compatible Chat Completions, Responses, embeddings, and other endpoints.

But receiving a familiar response structure such as:

{
  "role": "assistant",
  "content": "..."
}

does not mean two models will interpret the same conversation identically.

We saw changes in:

Instruction following
Response length
Formatting
Refusal behavior
JSON reliability
Tool selection
Edge-case reasoning

The integration compiled.

The product behavior changed.

That was our first major mistake:

We tested protocol compatibility before behavioral compatibility.

Break #2: Our Prompts Were More Model-Specific Than We Thought

We had system prompts that had been tuned over months.

For example:

Return a concise answer.
Do not include commentary.
Use the supplied context only.
Return JSON matching the requested format.

They worked consistently enough with the previous model that we started treating them as application logic.

Then we moved models.

Some responses became longer.

Some ignored preferred formatting.

Some interpreted instructions differently.

Others needed clearer examples.

The prompts were not really portable.

They were part of the model integration.

Then We Discovered Chat Templates

With many self-hosted chat models, the messages sent by the application eventually need to become a token sequence such as:

<system>
You are...
<user>
...
<assistant>

But different models may expect different special tokens and formatting.

Hugging Face explicitly recommends using the chat template associated with the model because formatting that does not match training can hurt performance.

That meant our architecture was actually:

Application messages
        ↓
Chat template
        ↓
Tokenizer
        ↓
Model

not simply:

Application messages
        ↓
Model

A wrong template could produce valid inference with worse behavior.

That is a particularly dangerous failure because nothing crashes.

The model just becomes less reliable.

Break #3: Structured Output Became Our Problem

Several application workflows expected objects like:

{
  "category": "billing",
  "priority": "high",
  "requires_human": true
}

With the hosted implementation, we relied heavily on schema-constrained responses.

OpenAI supports Structured Outputs and strict function schemas designed to make responses conform to supplied schemas.

Our first self-hosted implementation mostly relied on prompting:

Respond with valid JSON only.

It worked.

Until it did not.

We received responses such as:

Sure! Here's the JSON:

followed by JSON.

Or:

{
  "priority": "HIGH"
}

when the schema expected:

high

Occasionally, we also received incomplete objects.

The migration exposed an architectural mistake:

Parsing model output is not validation.

We Added Schema Enforcement

Our gateway eventually owned structured-output validation.

Conceptually:

Model output
    ↓
Schema validation
    ↓
Valid?
 ┌──┴──┐
Yes    No
 ↓      ↓
Return Retry / Fail safely

Modern inference servers can also support constrained or structured generation.

vLLM, for example, supports JSON Schema, regex, grammar, and other structured-output constraints.

That helped enormously.

But it also reinforced the larger lesson:

The hosted platform had been providing more than tokens. It had been providing behavioral infrastructure.

Break #4: Tool Calling Changed

Our application used tools for actions such as:

search_documents()
lookup_customer()
create_ticket()
get_order_status()

Before migration, we had logic built around a fairly predictable tool-call structure.

After migration, three things changed.

Sometimes the new model:

Answered directly instead of calling the tool
Chose the wrong tool
Produced arguments that were technically valid but semantically poor

Tool calling is partly an inference-server feature, but it is also heavily model-dependent.

For example, vLLM requires model-specific tool parsers and configurations for automatic tool selection in many setups.

So this:

tools=[...]

was not enough.

We had to test:

Should the model call a tool?

Which tool should it choose?

Are the arguments correct?

What happens after the tool output returns?

Can it handle multiple tool steps?

Tool-use quality became its own evaluation suite.

Break #5: Latency Looked Great Until Concurrency Arrived

Our first benchmark was encouraging.

One request:

User
 ↓
Model
 ↓
Fast response

Then production-like traffic arrived.

Now it looked like:

Request ─┐
Request ─┤
Request ─┤
Request ─┼──→ GPU
Request ─┤
Request ─┤
Request ─┘

Suddenly we were debugging:

Queue time
Batching
Token throughput
Long prompts
Long generations
Memory pressure
Concurrent requests

Single-request latency had told us almost nothing about production capacity.

The important metrics became:

Time to First Token
Tokens per Second
Queue Time
Requests per Second
P95 Latency
P99 Latency

A self-hosted model is not just a model.

It is a serving system.

Break #6: Context Length Became a Capacity Problem

Our application frequently supplied:

System instructions
Conversation history
Retrieved documents
Tool definitions
User input

A request could become large quickly.

Under a hosted API, we mainly thought about context as a model limit and usage cost.

Self-hosting added another concern:

Memory capacity.

Longer contexts meant more inference memory and lower effective concurrency.

So a request that worked perfectly alone could reduce throughput when many users arrived together.

We started treating context as an infrastructure resource.

Instead of automatically sending:

Entire conversation
+
10 retrieved documents
+
All tool definitions

we became much stricter about:

Document selection
History trimming
Prompt size
Tool exposure
Output limits

Prompt optimization became capacity optimization.

Break #7: Retries Became More Dangerous

Our hosted integration had retry logic.

Something failed?

Retry.

That behavior became dangerous when infrastructure was already overloaded.

Consider:

Server overloaded
      ↓
Request times out
      ↓
Client retries
      ↓
More load
      ↓
More timeouts
      ↓
More retries

Now we had created a retry storm.

We changed the gateway to use:

Bounded retries
Backoff
Timeouts
Concurrency limits
Overload handling

Inference failure had become distributed-systems failure.

Break #8: We Became the Operations Team

Before:

POST request
     ↓
Receive result

After:

Model weights
GPU drivers
Inference server
Container images
Autoscaling
Networking
Load balancing
Monitoring
Logging
Model versions
Security
Capacity planning
Failure recovery

None of those are reasons not to self-host.

They are simply costs that need to exist in the migration plan.

We had moved responsibility across a boundary.

The managed API provider was no longer operating inference for us.

Self-hosting also turns model deployment into a cloud infrastructure and scaling problem, because GPU capacity, monitoring, networking, and failure recovery now belong to your team.

The Most Important Fix: Put a Gateway in Front

Initially, parts of the application called the inference server directly.

That made every model difference leak into product code.

We replaced that design with an internal LLM gateway.

Application
     ↓
LLM Gateway
     ↓
 ┌───────────────┐
 │ Hosted Model  │
 │ Self-Hosted A │
 │ Self-Hosted B │
 └───────────────┘

The gateway owned:

Request normalization
Prompt versions
Timeouts
Retries
Structured-output validation
Tool schemas
Logging
Metrics
Model routing

Application code stopped caring which inference backend produced the answer.

A stable gateway also makes enterprise AI development easier because applications can switch models, validate outputs, monitor behavior, and scale inference without coupling every workflow to one provider.

That made the migration much safer.

We Also Stopped Doing Big-Bang Model Swaps

Our first approach was basically:

GPT
 ↓
New model

Later migrations used:

Current model
      ↓
Shadow traffic
      ↓
Evaluation
      ↓
Small percentage
      ↓
Compare
      ↓
Increase traffic
      ↓
Full migration

The new model initially received traffic without controlling the user-visible response.

We compared:

Answer quality
JSON validity
Tool selection
Latency
Token usage
Failure rate

That surfaced problems before customers saw them.

Our Evaluation Suite Became the Real Migration Plan

The most useful artifact was not the deployment script.

It was our evaluation dataset.

We collected representative requests for:

Simple Q&A
RAG
Extraction
Classification
Tool calling
Long context
Ambiguous instructions
Bad input
Edge cases

For every candidate model, we tested the same workload.

Conceptually:

Test	Existing Model	Self-Hosted
JSON schema compliance	PASS	PASS
Tool selection	PASS	FAIL
RAG answer quality	PASS	PASS
Long-context request	PASS	SLOW
Classification	PASS	PASS

Now model selection was based on product behavior.

Not leaderboard scores.

What Broke

Looking back, the failures fell into a few categories.

Prompt Behavior

Prompts that worked well with one model needed adjustment.

Formatting

Structured output required stronger enforcement.

Tool Use

Tool selection and arguments needed fresh evaluation.

Performance

Single-request benchmarks hid concurrency problems.

Context

Large prompts reduced serving capacity.

Reliability

Retries, overload, and failover became our responsibility.

Operations

Inference infrastructure became part of the product.

What Worked

Several changes made the migration much safer.

A Stable Internal API

Application code talked to our gateway, not directly to a specific model.

Model-Specific Prompting

We stopped pretending every model needed identical prompts.

Schema Validation

Structured outputs were validated rather than trusted.

Production-Like Load Tests

We tested concurrent traffic instead of one request at a time.

Shadow Evaluation

New models were compared before receiving full traffic.

Versioned Models and Prompts

Every production request could be traced to:

Model version
Prompt version
Gateway version
Inference configuration

That made regressions much easier to investigate.

The Architecture We Ended Up With

The final system looked closer to this:

                   Application
                       │
                       ▼
                   LLM Gateway
                       │
       ┌───────────────┼───────────────┐
       ▼               ▼               ▼
 Prompt Manager    Validation     Observability
       │               │               │
       └───────────────┼───────────────┘
                       ▼
                 Model Router
                  ┌────┴────┐
                  ▼         ▼
              Hosted     Self-Hosted
               Model       Models

The key abstraction was no longer:

callModel()

It was:

executeLLMWorkflow()

because production AI applications depend on much more than generation.

What We Would Do Differently

For the next self hosted LLM migration, we would use this order.

1. Build Evaluations First

Build the evaluation suite before choosing the replacement model.

2. Inventory Every Model-Dependent Feature

Include:

Structured output
Tools
RAG
Vision, if applicable
Context length
Streaming

3. Introduce an LLM Gateway

Put the gateway in place before changing providers or models.

4. Validate the Chat Template

Make sure the model receives messages in the format it was trained to understand.

5. Load-Test Realistic Prompts

Do not rely only on tiny benchmark requests.

6. Test Tool Behavior Separately

API support does not prove that the model uses tools correctly.

7. Run Shadow Traffic

Compare production workloads before switching user-visible responses.

8. Plan Operations Before Launch

GPU capacity, monitoring, failure handling, scaling, and upgrades belong in the original migration scope.

The Biggest Lesson

We originally thought we were replacing this:

GPT

with this:

Self-hosted model

What we were really replacing was:

Model
+
Serving infrastructure
+
Structured output behavior
+
Tool behavior
+
Scaling
+
Reliability
+
Operational responsibility

That is a much larger migration.

And it explains why an application can successfully return its first self-hosted response while still being nowhere near production-ready.

Final Takeaway

A self hosted LLM migration should not begin with:

Which open model looks closest to our current GPT model?

Start with:

What behavior does our application depend on?

How will we measure that behavior?

Which parts are currently handled by the provider?

Which of those responsibilities are moving to us?

Then migrate in layers:

Inventory dependencies
       ↓
Build evaluations
       ↓
Add gateway
       ↓
Deploy model
       ↓
Validate outputs
       ↓
Load test
       ↓
Shadow traffic
       ↓
Gradual rollout
       ↓
Monitor

Self-hosting gave us more control.

But that control came with responsibility.

The real migration was not from one model to another.

It was from consuming inference as a service to operating inference as a production system.

Post-Mortem: The Go-Live That Slipped Six Weeks

Samcorp — Wed, 02 Sep 2026 06:09:43 +0000

The ERP project looked close to finished.

Development was done. Integrations were connected. Most users had completed training. The production environment was ready.

Then we ran the final readiness review.

The go-live moved by six weeks.

Not because of one catastrophic bug.

Because several small risks we had accepted for months finally became impossible to ignore.

Note: This is a representative ERP implementation post-mortem. The timeline and examples are illustrative, not customer production data.

The Original Plan

The final weeks looked simple:

Finish UAT
   ↓
Final data migration
   ↓
Cutover rehearsal
   ↓
Business sign-off
   ↓
Go live

But each step depended on the one before it actually being complete.

That was our first mistake.

We planned around dates.

We should have planned around exit criteria.

Problem #1: UAT Was “Almost Done”

The dashboard showed:

UAT: 92% complete

That sounded good.

It was not.

The missing 8% included:

month-end finance scenarios,
partial order cancellations,
inventory adjustments,
credit processing,
exception workflows.

These were not edge cases we could safely ignore.

They were business processes.

The lesson was simple:

A test cycle is not complete because most test cases passed.

It is complete when critical business processes are tested, defects are resolved or formally accepted, and the business signs off.

Current ERP implementation guidance similarly treats UAT completion and stakeholder approval as explicit go-live readiness conditions.

Problem #2: The Data Migration Did Not Fit the Cutover Window

Our migration scripts worked.

That was not the problem.

The problem was time.

The cutover window allowed roughly:

10 hours

Our latest migration rehearsal took:

14 hours 40 minutes

Technically successful.

Operationally unusable.

We found several causes:

slow transformation jobs,
unnecessary historical records,
sequential loads that could be reorganized,
reconciliation happening too late,
failed records requiring manual investigation.

The next dry runs became timed rehearsals:

Extract
   ↓
Transform
   ↓
Load
   ↓
Reconcile
   ↓
Business validation

A migration is not ready because it works.

It must also work inside the real cutover window.

A strong ERP data migration strategy should include repeated trial loads, reconciliation, realistic-volume testing, cutover timing, and clear rollback criteria.

Repeated migration rehearsals and validation are specifically recommended before production cutover.

Problem #3: Integrations Passed Functional Tests but Failed at Volume

One integration worked perfectly when we sent ten transactions.

Then we tested production-like traffic.

Queues started growing.

Retries increased.

One downstream service slowed the entire chain.

ERP
 ↓
Integration Layer
 ↓
Warehouse
 ↓
Carrier
 ↓
ERP confirmation

The happy path worked.

The production workload did not.

This is why ERP system integration testing needs to cover realistic transaction volumes, queue behavior, retries, external dependencies, and failure recovery—not just basic connectivity.

We had tested:

Does the integration work?

We had not tested enough:

Does it still work when hundreds or thousands of transactions arrive together?

That difference contributed directly to the ERP go live delay.

Integration testing should include realistic peak volumes and external-system failure scenarios, not only basic connectivity.

Problem #4: Data Reconciliation Was Still Manual

Our migration technically loaded the expected number of records.

But record count was not enough.

For inventory, finance, and open transactions, we needed to prove:

Source
  =
Target

at the correct business level.

That meant comparing things like:

inventory quantity by SKU and location,
open purchase order value,
customer balances,
vendor balances,
journal totals,
open sales orders.

A migration could contain the correct number of records and still contain incorrect values.

We automated more of the reconciliation and turned unexplained differences into blockers.

Problem #5: The Cutover Plan Had Tasks but Not Enough Decisions

Our first cutover plan looked detailed.

It had timestamps.

Owners.

Tasks.

Dependencies.

But it was missing something critical:

decision points.

For example:

02:00 — Final migration complete
02:30 — Reconciliation
03:00 — Integration validation
03:30 — Business verification

But what happened if reconciliation failed?

Who decided whether to continue?

How much variance was acceptable?

When did rollback become mandatory?

The improved runbook included explicit gates:

Migration complete?
      ↓
Data reconciled?
      ↓
Critical integrations healthy?
      ↓
Business processes validated?
      ↓
GO / NO-GO

A cutover runbook needs verification, sign-off, roles, dependencies, and rollback criteria—not just a task list.

Why It Became Six Weeks

No single issue required six weeks.

The combination did.

The additional time went into roughly four areas:

Area	Work Required
UAT	Finish critical scenarios and resolve blockers
Data	Optimize migration and repeat reconciliation
Integrations	Test at production-like volume
Cutover	Rehearse the complete runbook again

We also used the time to finish:

production support procedures,
user access validation,
monitoring,
rollback planning,
business communication.

The delay was painful.

But going live with unresolved issues would have been worse.

The Second Cutover Rehearsal Changed Everything

We treated the next mock cutover like production.

Same sequence.

Similar data volume.

Same teams.

Same dependencies.

Same validation steps.

The result looked very different:

Migration within window       PASS
Reconciliation                PASS
Critical integrations         PASS
UAT blockers                  0
Production access             PASS
Support team ready            PASS
Rollback procedure            VERIFIED

Only then did the date become credible.

Mock cutovers are valuable precisely because they test the whole transition rather than individual technical components in isolation.

What We Got Wrong

The biggest mistakes were not deeply technical.

We Treated the Go-Live Date as Fixed

The date became more important than readiness.

We Counted Progress Instead of Risk

“92% UAT complete” sounded reassuring while critical scenarios were still untested.

We Tested Components Individually

Migration worked.

Integrations worked.

ERP worked.

But we had not rehearsed the complete operating system enough.

We Accepted “Almost Ready”

Almost ready multiplied across five workstreams becomes very far from ready.

What Changed

For the second attempt, go-live required explicit evidence.

UAT signed off
      ✓

Migration fits window
      ✓

Data reconciled
      ✓

Integrations tested at volume
      ✓

Critical defects closed
      ✓

Users and security validated
      ✓

Support team ready
      ✓

Cutover rehearsed
      ✓

Rollback tested
      ✓

No green check meant no go-live.

That made discussions much easier.

Instead of asking:

Do we feel ready?

we asked:

Which exit criterion is still open?

The Biggest Lesson

The six-week delay was not created during the final readiness review.

It was created gradually.

Every time we said:

We'll finish that later.

we borrowed time from cutover.

Every incomplete test.

Every unresolved data issue.

Every integration assumption.

Every undocumented rollback decision.

Eventually, all of that debt became visible at once.

Final Takeaway

An ERP go live delay is rarely caused by a single dramatic failure.

More often, it is the result of several workstreams reaching the finish line at 90–95% readiness.

And in an ERP implementation, 95% ready can still mean:

Finance cannot close.
Inventory does not reconcile.
Orders cannot ship.
Users cannot access the right functions.
Integrations fail under load.

The most important lesson from the six-week slip was simple:

Go-live is not a calendar date. It is a readiness state.

Set measurable exit criteria.

Rehearse the full cutover.

Test with realistic data and volume.

Make unresolved critical risks visible.

And let readiness determine the date—not the other way around.

Reconciling Inventory Across Three Systems During Cutover

Samcorp — Wed, 02 Sep 2026 05:55:38 +0000

The ERP migration looked ready. Customers, products, and purchase orders were loaded, and the interfaces were green.

Then we compared inventory across three systems—and the numbers did not match.

That was when the real ERP cutover problem began.

Three systems gave us three different answers.

Legacy ERP:     184,732 units
WMS:            183,941 units
New ERP:        185,106 units

Every number was technically explainable.

None was acceptable for go-live.

That became the most important lesson from the ERP cutover:

Inventory reconciliation is not about proving that one total matches. It is about proving that every system agrees on what each quantity means at the exact moment ownership changes.

Here is how we approached the problem.

Note: This is a representative engineering post-mortem. The systems, quantities, and examples are illustrative rather than customer production data.

The Three Systems

Our inventory existed across three operational systems.

Legacy ERP

The financial system of record.

It knew about:

item balances,
inventory valuation,
purchase orders,
transfers,
adjustments.

Warehouse Management System

The operational warehouse view.

It knew exactly where stock physically existed:

Warehouse
Bin
Lot
Serial
Inventory status

New ERP

The system we were preparing to activate.

It needed a correct opening inventory position before new transactions could start.

At first, the reconciliation looked straightforward:

Legacy ERP
     ↓
Compare
     ↓
WMS
     ↓
Load
     ↓
New ERP

A structured ERP implementation should include trial migrations, reconciliation, integration testing, business sign-off, and post-go-live monitoring—not just the final data load.
It was not.

Inventory Was Not One Number

Our first mistake was comparing total quantity.

Suppose all three systems reported roughly:

185,000 units

That looked encouraging.

But total quantity could hide serious errors.

For example:

SKU	Location	Legacy ERP	WMS	New ERP
A100	Chicago	500	500	500
A100	Dallas	300	250	300
B220	Chicago	200	250	200

The grand total still matched.

Operationally, it was wrong.

So the reconciliation grain became:

SKU
+
Warehouse
+
Inventory status
+
Unit of measure

For controlled items, we went deeper:

SKU
+
Location
+
Lot / Serial
+
Status

A matching grand total was no longer enough.

The First Variance: Timing

The biggest source of differences was not bad data.

It was timing.

Consider this sequence:

10:01  ERP export starts
10:03  Warehouse ships 20 units
10:05  WMS snapshot starts
10:08  ERP receives shipment confirmation

Now compare both systems.

They represent different moments.

Neither system is necessarily wrong.

This is why reconciliation without a defined cutoff timestamp is meaningless.

We introduced one rule:

All inventory comparisons must represent
the same business moment.

That meant defining:

when warehouse transactions stop,
when ERP posting stops,
when interfaces pause,
which transactions belong before cutover,
which transactions belong after cutover.

The timestamp became part of the reconciliation evidence.

We Needed a Real Freeze Window

Initially, we hoped interfaces could continue running while we reconciled.

That created a moving target.

Every time we fixed one variance, another transaction changed the balance.

The final cutover sequence became stricter:

Stop warehouse transactions
        ↓
Pause integrations
        ↓
Complete in-flight messages
        ↓
Freeze legacy inventory
        ↓
Take final snapshots
        ↓
Reconcile
        ↓
Load new ERP
        ↓
Reconcile again
        ↓
Go / No-Go

The freeze was inconvenient.

But a reconciliation against continuously changing inventory was worse.

A controlled ERP data migration strategy needs a defined freeze window, final delta handling, reconciliation rules, and clear go/no-go criteria before production starts.

The Second Variance: Inventory Status

The WMS did not simply know that 100 units existed.

It knew their status.

For one SKU:

Available:       70
Allocated:       15
Quality Hold:    10
Damaged:          5
-------------------
Physical:       100

The legacy ERP might display:

On Hand: 100

while the new ERP imported:

Available: 100

The totals matched.

The business meaning did not.

That could allow damaged or quarantined inventory to be sold.

So we created explicit status mappings.

WMS AVAILABLE      → ERP AVAILABLE
WMS QA_HOLD        → ERP QUALITY_HOLD
WMS DAMAGED        → ERP BLOCKED
WMS ALLOCATED      → ERP RESERVED

Unknown statuses were not defaulted.

They became cutover exceptions.

That decision prevented silent inventory corruption.

The Third Variance: Units of Measure

Then we found quantities that were mathematically correct but represented different units.

One system stored:

1 CASE

Another stored:

12 EACH

A third had:

without enough context.

The reconciliation needed both:

Quantity
+
Unit of measure

before comparison.

So we normalized quantities into the agreed base unit.

Conceptually:

function toBaseUnit(quantity, conversionFactor) {
  return quantity * conversionFactor;
}

Then:

2 CASES × 12 EACH
=
24 EACH

Only normalized quantities were used in reconciliation.

The Fourth Variance: In-Flight Transactions

Some inventory existed between states.

Examples included:

warehouse transfers,
receipts not yet posted,
shipments picked but not confirmed,
purchase receipts awaiting interface processing,
adjustments waiting in an integration queue.

These records were dangerous because every system could legitimately represent them differently.

Consider an inter-warehouse transfer:

Warehouse A
-100 units

In Transit
+100 units

Warehouse B
0 units until receipt

If the new ERP loaded only warehouse balances and ignored in-transit stock, those 100 units effectively disappeared.

We therefore created a separate reconciliation category for in-flight inventory.

It could not hide inside ordinary on-hand quantities.

We Built a Canonical Inventory Snapshot

Comparing System A directly with System B and then System B with System C became difficult.

Each system used different field names and definitions.

Instead, we transformed every extract into the same structure.

Conceptually:

{
  "sku": "A100",
  "warehouse": "WH-01",
  "status": "AVAILABLE",
  "uom": "EA",
  "quantity": 250,
  "snapshotTime": "cutover"
}

Then the process became:

Legacy ERP ──┐
             │
WMS ─────────┼──→ Canonical Format
             │          ↓
New ERP ─────┘      Reconciliation

That made comparison much simpler.

The reconciliation engine did not need to understand three ERP/WMS schemas.

It only needed to understand one canonical inventory model.

The Reconciliation Was Automated

Forty thousand inventory combinations cannot be reliably compared in spreadsheets during a stressful cutover weekend.

We generated a reconciliation dataset.

Conceptually:

SELECT
    sku,
    warehouse,
    status,
    legacy_qty,
    wms_qty,
    new_erp_qty,
    legacy_qty - wms_qty AS legacy_wms_variance,
    wms_qty - new_erp_qty AS wms_new_variance
FROM inventory_reconciliation
WHERE
    legacy_qty <> wms_qty
    OR wms_qty <> new_erp_qty;

The output looked more like this:

SKU	Warehouse	Status	Legacy	WMS	New ERP	Result
A100	WH01	Available	500	500	500	PASS
B210	WH01	Available	120	118	118	INVESTIGATE
C330	WH02	Hold	40	40	0	FAIL
D115	WH03	Available	72	72	72	PASS

Now the team reviewed exceptions rather than manually checking every record.

Every Variance Needed a Reason

We did not immediately “fix” every mismatch.

First, we classified it.

Our variance categories became:

TIMING
UOM_MAPPING
STATUS_MAPPING
SKU_MAPPING
IN_FLIGHT_TRANSACTION
MISSING_TRANSACTION
DUPLICATE_LOAD
MANUAL_ADJUSTMENT
UNKNOWN

This was extremely useful.

For example:

SKU: A-1044
Variance: +24
Reason: UOM_MAPPING
Action: Correct case-to-each conversion

versus:

SKU: B-2291
Variance: -3
Reason: TIMING
Action: Include receipt posted before cutoff

A number without a reason is just another number.

A classified variance tells you what to fix.

We Added Control Totals

Detailed reconciliation caught SKU-level problems.

Control totals told us whether the entire migration was structurally sound.

We checked totals such as:

Total SKU/location combinations
Total physical quantity
Total available quantity
Total blocked quantity
Total inventory value
Total serialized units
Total lot-controlled quantities

For example:

Expected inventory value:  $12,482,911.40
New ERP inventory value:   $12,482,911.40
Variance:                  $0.00

But we never treated a matching inventory value as proof that quantities were correct.

Two errors can cancel each other out.

Detailed reconciliation and control totals served different purposes.

We Rehearsed the Cutover

The final reconciliation logic was not created during go-live.

We ran mock cutovers first.

Each rehearsal tested:

extraction duration,
load duration,
transaction freeze,
integration shutdown,
inventory mapping,
reconciliation scripts,
exception handling,
business sign-off,
rollback timing.

Every rehearsal revealed something.

One exposed an unmapped warehouse.

Another found duplicate SKU aliases.

Another showed that an integration queue could still post transactions after the supposed freeze.

That was exactly why we rehearsed.

The purpose of a mock cutover is not to prove the plan is good.

It is to discover where the plan is wrong while failure is still cheap.

Our Go/No-Go Rule Became Explicit

One of the worst things a cutover team can hear at 2 AM is:

“The numbers look close enough.”

We defined acceptance rules before the final weekend.

Conceptually:

Critical inventory variance:
Must equal zero.

Unknown SKU mapping:
Must equal zero.

Unknown inventory status:
Must equal zero.

Unprocessed integration messages:
Must equal zero.

Inventory valuation:
Must reconcile to approved control total.

Some organizations may define tolerances for particular non-critical controls.

The important part is deciding them before the cutover.

The person under schedule pressure should not also be inventing the acceptance criteria.

The Final ERP Cutover Flow

The final runbook looked approximately like this:

1. Stop business transactions
             ↓
2. Drain integration queues
             ↓
3. Freeze legacy systems
             ↓
4. Capture final inventory snapshots
             ↓
5. Normalize all three datasets
             ↓
6. Run source reconciliation
             ↓
7. Investigate exceptions
             ↓
8. Load opening inventory
             ↓
9. Reconcile new ERP
             ↓
10. Business sign-off
             ↓
11. Go / No-Go decision
             ↓
12. Resume operations

The migration script was only one step.

The reconciliation process was what made the cutover trustworthy.

What Went Wrong

Looking back, our biggest mistakes were simple.

We Compared Totals Too Early

A matching grand total hid location- and status-level errors.

We Did Not Define the Cutoff Precisely Enough

Systems representing different moments cannot be reconciled meaningfully.

We Treated Inventory Status as a Label

It was actually business logic.

We Underestimated In-Flight Transactions

Inventory between locations or process states needed explicit treatment.

We Relied Too Much on Spreadsheets

Spreadsheets were useful for investigation, but automated reconciliation was safer for repeatable cutover validation.

We Started With Data Movement Instead of Reconciliation Design

Knowing how data would be proved correct should have come before writing the final migration process.

What Worked

The strongest decisions were:

define one cutoff timestamp,
freeze transactions before final reconciliation,
normalize every system into one inventory model,
compare at SKU/location/status level,
normalize units of measure,
isolate in-flight inventory,
automate variance detection,
classify every exception,
establish control totals,
rehearse the full cutover,
define go/no-go rules before go-live.

The architecture became:

Legacy ERP ──┐
             │
WMS ─────────┼──→ Normalize
             │       ↓
New ERP ─────┘   Reconcile
                     ↓
                 Exceptions
                     ↓
                  Sign-Off

Simple architecture.

Strict controls.

The Biggest Lesson

The hardest part of inventory migration was not copying inventory into the new ERP.

It was agreeing on what inventory meant.

Was stock:

On hand?
Available?
Allocated?
In transit?
On hold?
Damaged?
Received but not posted?
Picked but not shipped?

Three systems could all be correct according to their own definitions and still disagree with each other.

That is why reconciliation has to start with business meaning, not database columns.

Final Takeaway

A safe ERP cutover should not ask only:

Did the inventory load succeed?

It should ask:

Did we capture the same point in time?

Do SKU and location quantities agree?

Do statuses mean the same thing?

Were units normalized?

Are in-flight transactions accounted for?

Does inventory value reconcile?

Can every remaining variance be explained?

If those questions cannot be answered, the inventory is not reconciled.

The most important artifact from our cutover was not the migration script.

It was the evidence that allowed the business to say:

These opening balances are trustworthy.

That is what reconciliation is really for.

Migrating 40,000 Pages From WordPress to Astro

Samcorp — Tue, 01 Sep 2026 10:09:37 +0000

Migrating a small WordPress site to Astro is mostly a development task.

Migrating 40,000 pages is something else entirely.

It becomes a content, URL, SEO, media, routing, and deployment migration—all at once.

Our first plan looked simple:

Export WordPress
      ↓
Convert content
      ↓
Build with Astro
      ↓
Deploy

The real process looked more like this:

Inventory
   ↓
Extract
   ↓
Normalize
   ↓
Transform
   ↓
Generate
   ↓
Validate
   ↓
Deploy

The Astro components were not the hardest part.

Preserving years of WordPress history was.

Note: This is a representative large-scale migration post-mortem, not a customer case study.

The First Mistake: Building Before Inventorying

We started by creating Astro components:

Header.astro
ArticleLayout.astro
Footer.astro
RelatedPosts.astro

The first migrated pages looked perfect.

Then we reached older content.

We found:

legacy shortcodes,
page-builder markup,
inline CSS,
broken HTML,
old CDN URLs,
embedded forms,
custom blocks,
absolute internal links,
obsolete plugin output.

That changed the question.

It was no longer:

Can Astro render WordPress content?

It became:

What has 15 years of WordPress actually stored?

Start With Every Existing URL

The most useful artifact in the entire migration became a URL manifest.

For every public URL, we recorded something like:

{
  "source": "/guides/performance/",
  "target": "/guides/performance/",
  "type": "page"
}

This gave us a clear contract:

Old URL exists
      ↓
New URL exists
      ↓
Same path OR redirect exists
      ↓
Correct status
      ↓
Correct canonical

Without this inventory, there was no reliable way to answer:

Did all 40,000 pages actually migrate?

Treat WordPress Export Like an ETL Pipeline

Forty thousand records are not something you should treat as one giant API response.

WordPress REST API collections are paginated, so extraction needs to happen in batches.

Conceptually:

let page = 1;

while (true) {
  const response = await fetch(
    `${baseUrl}/wp-json/wp/v2/pages?per_page=100&page=${page}`
  );

  const records = await response.json();

  if (!records.length) break;

  await saveBatch(records);

  page++;
}

The important change was adding checkpoints.

Fetch
 ↓
Normalize
 ↓
Save
 ↓
Checkpoint
 ↓
Continue

If batch 287 failed, we restarted from batch 287.

Not from zero.

Do Not Couple Astro Directly to WordPress Data

Instead of passing raw WordPress API responses through the application, we created a normalized content model.

For example:

type MigratedPage = {
  sourceId: number;
  path: string;
  title: string;
  description?: string;
  content: string;
  canonical?: string;
  publishedAt?: string;
};

That gave us:

WordPress
   ↓
Extraction
   ↓
Normalized Content
   ↓
Astro

Now WordPress-specific decisions stayed in the migration layer instead of leaking into every Astro component.

Not Everything Should Become Markdown

We originally considered converting all content to Markdown.

That worked for clean pages.

It failed on years of legacy content.

For example:

[pricing_table product="enterprise"]

or:

<div class="legacy-callout">
  ...
</div>

A blind HTML-to-Markdown conversion could silently remove functionality or structure.

So we used three options:

convert clean content,
transform known legacy patterns,
preserve sanitized HTML when conversion would lose information.

The goal was not format purity.

The goal was content fidelity.

Shortcodes Need Explicit Replacement Rules

WordPress understood old shortcodes because plugins interpreted them.

Astro does not.

We therefore classified them as:

Static

Convert during migration.

Dynamic

Replace with a real Astro component or service.

Dead

Remove or rewrite intentionally.

Unknown shortcodes became migration failures.

That was better than discovering this in production:

[old_widget id="32"]

Preserve URLs Unless You Have a Good Reason Not To

A framework migration is already risky.

We did not want to combine it with a complete URL redesign.

If WordPress used:

/guides/astro-migration/

Astro normally kept:

/guides/astro-migration/

When a URL genuinely changed, the redirect was stored as migration data:

{
  "from": "/old-guide/",
  "to": "/guides/new-guide/",
  "status": 301
}

We also standardized:

trailing slashes,
canonical URLs,
internal links,
sitemap URLs,
redirect destinations.

Routing was not merely a frontend decision.

It was an SEO requirement.

Media Was More Than `/uploads`

Images existed in more places than expected:

page HTML,
featured images,
metadata,
srcset,
custom fields,
CSS,
old CDN URLs.

So assets received their own manifest:

{
  "source": "https://old.example.com/uploads/hero.jpg",
  "destination": "/media/hero.jpg"
}

This allowed us to detect:

missing files,
old WordPress URLs,
failed rewrites,
broken image references.

At 40,000 pages, guessing whether every asset migrated was not acceptable.

40,000 Static Pages Change Build Economics

Astro can generate large numbers of static routes.

But static pages are not free to build.

Every generated page requires work.

At this size, scalable web development also means thinking about build times, routing strategy, caching, and deployment economics—not only frontend rendering.

So we had to ask:

Does every page need to be regenerated on every deployment?

The answer depends on the site.

Possible approaches include:

Fully static

or:

Static important routes
+
on-demand rendering where appropriate

The key lesson was simple:

Fast runtime performance does not automatically mean fast builds.

Build time became an architecture constraint.

SEO Metadata Is Content Too

Moving the visible article was not enough.

We also needed to preserve:

titles,
meta descriptions,
canonical URLs,
publication dates,
authors,
structured data,
social metadata where needed.

A page could render perfectly and still become an SEO migration failure.

That made metadata part of our normalized content model rather than an afterthought.

A large migration needs an SEO-first web architecture so clean URLs, metadata, structured data, redirects, and crawlability survive the platform change.

Validation Became More Important Than Conversion

Eventually, most of our confidence came from automated validation.

For every expected URL, we checked:

Route exists?
      ↓
Correct status?
      ↓
Title present?
      ↓
Canonical correct?
      ↓
Content present?
      ↓
Images valid?
      ↓
Internal links valid?

We also made certain errors fail the build:

missing expected routes,
duplicate slugs,
unknown shortcodes,
broken internal links,
missing canonicals,
redirect loops.

Our rule became:

If a machine can prove the migration is broken, a human should not need to discover it after launch.

The Cutover Needed a Delta Migration

A large migration does not happen instantly.

Editors may continue publishing while the migration runs.

That means the original export quickly becomes outdated.

Our cutover flow looked roughly like:

Full migration
      ↓
Validate
      ↓
Sync recent changes
      ↓
Short publishing freeze
      ↓
Final delta sync
      ↓
Deploy
      ↓
Production validation

We also kept WordPress available during the rollback window.

A migration is safer when the previous production system remains recoverable.

What Went Wrong

The biggest mistakes were straightforward.

We started with components instead of inventory

Building UI felt productive, but understanding the existing site was more important.

We underestimated legacy content

Forty thousand pages created over many years do not follow one clean format.

We assumed Markdown should be the destination

Sometimes preserving HTML was safer.

We treated redirects as configuration

They were actually migration data.

We assumed static generation solved every performance problem

Runtime performance and build performance are different concerns.

We manually checked pages instead of validating the system

Automation gave us much more confidence than random visual QA.

What Worked

The strongest decisions were:

inventory every public URL,
normalize WordPress data before Astro sees it,
migrate page types instead of individual pages,
preserve URLs whenever possible,
track assets separately,
validate automatically,
fail on unknown legacy content,
plan the final content sync early.

The final architecture became:

WordPress
    ↓
Extract
    ↓
Normalize
    ↓
 ┌─────────────┐
 │   Content   │
 │   Assets    │
 │  Redirects  │
 └─────────────┘
       ↓
      Astro
       ↓
   Validation
       ↓
    Production

The Biggest Lesson

The hardest part of a WordPress to Astro migration was not Astro.

It was WordPress history.

The old system represented much more than PHP and a database.

It contained:

40,000 URLs
years of content
SEO history
redirects
media
plugins
shortcodes
custom markup
undocumented assumptions

Astro did not need to reproduce WordPress.

It needed to preserve the parts users and search engines depended on.

Final Takeaway

For a large WordPress to Astro migration, do not start with:

npm create astro@latest

Start with three questions:

What exists today?

What must survive?

How will we prove it survived?

Then build the migration around:

Inventory
   ↓
Extract
   ↓
Normalize
   ↓
Transform
   ↓
Generate
   ↓
Validate
   ↓
Deploy

The framework was the new delivery layer.

The real engineering work was migrating 40,000 content contracts without breaking them.

We Cut LCP From 6s to 1.2s Every Change, Measured

Samcorp — Tue, 01 Sep 2026 05:24:43 +0000

The page did not look broken.

It loaded.

The hero appeared.

The buttons worked.

Nothing obvious screamed “performance problem.”

But under mobile conditions, Largest Contentful Paint was taking roughly:

6.0 seconds

That was not a small regression.

It was more than twice the 2.5-second threshold generally considered good for LCP.

So instead of throwing every common performance trick at the page, we treated the problem like a debugging exercise.

One change.

One measurement.

Then the next change.

The final result:

Before: 6.0s LCP
After:  1.2s LCP
Change: -80%

The interesting part was not the final number.

It was discovering that the largest image was not initially the biggest problem.

Here is how the LCP optimization unfolded.

Note: This is a representative performance case study. The measurements are designed to illustrate the debugging and optimization process rather than represent a specific customer project.

First, We Made the Test Repeatable

Performance work becomes unreliable when every change is measured under different conditions.

Before touching the application, we created a repeatable baseline.

The same principle applies to performance and load testing: if the environment and conditions keep changing, the results are difficult to compare or trust.

For each version, we:

Tested the same URL
Used the same mobile profile
Started with a cold cache
Used the same network conditions
Ran multiple tests
Ignored obvious outliers
Compared the median result
Recorded the LCP element and timing breakdown

Lab tools helped us iterate quickly.

Real-user monitoring was still important for validating the final result because actual users have different devices, networks, locations, and cache states.

The lab was our debugging environment.

Field data was our reality check.

The Baseline: Where Were the Six Seconds Going?

Looking at a single LCP number was not enough.

We broke it into four parts:

LCP Component	Baseline
Time to First Byte	1.5s
Resource load delay	2.1s
Resource load duration	1.0s
Element render delay	1.4s
Total LCP	6.0s

That table immediately changed our approach.

The hero image took about one second to download.

But the browser spent 2.1 seconds waiting before it even started downloading it.

Compressing the image first would have attacked the wrong bottleneck.

Change #1: Fix the 1.5s TTFB

LCP: 6.0s → 5.0s

The document itself was arriving too slowly.

Our initial request involved application work that did not need to happen for every page view.

Conceptually, the request looked like:

Browser
   ↓
Application
   ↓
Database/API work
   ↓
Template rendering
   ↓
HTML response

Until that HTML starts arriving, the browser cannot discover most of the resources needed to render the page.

We reduced that delay by:

Caching cacheable HTML responses
Removing unnecessary server-side work
Avoiding repeated data fetches
Moving non-critical work out of the request path
Serving cached responses closer to users where appropriate

The TTFB moved from:

1.5s → 0.5s

Our new breakdown looked approximately like this:

Component	After Change #1
TTFB	0.5s
Resource load delay	2.1s
Resource load duration	1.0s
Render delay	1.4s
LCP	5.0s

A one-second improvement was useful.

But five seconds was still terrible.

More importantly, the trace now made the next problem impossible to ignore.

Change #2: Let the Browser Discover the Hero Immediately

LCP: 5.0s → 3.1s

This was the biggest single improvement.

Our LCP element was the hero image.

But the browser could not discover it early.

The implementation effectively depended on other resources being processed before the hero request became visible.

Patterns like this commonly cause trouble:

.hero {
    background-image: url("/images/hero.webp");
}

If that CSS is in an external stylesheet, the browser needs to:

Receive HTML
   ↓
Discover CSS
   ↓
Download CSS
   ↓
Parse CSS
   ↓
Discover hero image
   ↓
Request hero image

The image may be highly compressed.

That does not matter if its request starts two seconds late.

We moved the important visual into the document as an actual image element.

Conceptually:

<picture>
  <source
    srcset="/images/hero-1280.avif"
    type="image/avif"
  >

  <img
    src="/images/hero-1280.webp"
    width="1280"
    height="720"
    fetchpriority="high"
    alt="Application dashboard"
  >
</picture>

We also verified something important:

loading="lazy"

was not applied to the LCP image.

Lazy loading below-the-fold images is useful.

Lazy loading the element that is supposed to become your largest above-the-fold paint is usually counterproductive.

After the change:

Resource load delay:

2.1s → 0.2s

And overall LCP dropped:

5.0s → 3.1s

That single change saved almost two seconds.

This was the most important lesson from the entire investigation:

Do not only ask how fast the LCP resource downloads. Ask how quickly the browser discovers it.

Change #3: Now We Optimized the Image

LCP: 3.1s → 2.4s

Only now did image optimization become the largest obvious opportunity.

The original hero asset was much larger than necessary.

We addressed three things.

Format

We generated modern image variants instead of sending the same large source everywhere.

Dimensions

A mobile device did not need an image sized for a large desktop display.

Responsive image candidates allowed the browser to choose an appropriate resource.

For example:

<img
  src="/images/hero-1280.webp"
  srcset="
    /images/hero-640.webp 640w,
    /images/hero-960.webp 960w,
    /images/hero-1280.webp 1280w
  "
  sizes="100vw"
  width="1280"
  height="720"
  fetchpriority="high"
  alt="Application dashboard"
>

Compression

We reduced bytes without making the hero visibly degraded.

The result was a substantial reduction in resource transfer time:

Resource load duration:

1.0s → 0.3s

Overall:

3.1s → 2.4s

We had finally crossed the 2.5-second line in our controlled test.

But stopping here would have meant leaving almost another second on the table.

Change #4: Remove Work Blocking the Paint

LCP: 2.4s → 1.5s

At this point the hero was:

Discovered early
Prioritized
Relatively small
Downloaded quickly

Yet there was still a noticeable gap between:

Image downloaded

and:

Image painted

That is element render delay.

The culprit was not the image anymore.

The browser's critical rendering path contained resources that delayed the initial render.

We found:

CSS that was not required above the fold
Scripts executing before they were needed
Third-party code competing for the main thread
Application code doing work before initial content could settle

We started removing things from the critical path.

Non-critical JavaScript was deferred where possible:

<script src="/analytics.js" defer></script>

Non-essential functionality stopped competing with the first render.

We also reviewed the CSS loaded before the hero appeared.

This kind of web performance optimization is easier when frontend architecture, asset loading, and rendering behavior are treated as part of the same performance problem.

The rule was simple:

If the resource is not necessary to display the first screen, why is the first screen waiting for it?

After that cleanup:

Element render delay:

1.4s → 0.5s

LCP:

2.4s → 1.5s

The page now felt dramatically different even though visually almost nothing had changed.

Change #5: Reduce Main-Thread Work Before LCP

LCP: 1.5s → 1.2s

The final improvement was smaller, but it exposed another useful performance pattern.

The main thread was still doing too much work early.

Large JavaScript bundles were being:

downloaded
    ↓
parsed
    ↓
compiled
    ↓
executed

before some of that functionality was actually needed.

We inspected long tasks occurring before LCP and asked:

Does this code need to execute immediately?
Can this component initialize after the initial render?
Can this bundle be split?
Is a third-party script blocking useful work?
Are we hydrating components that are below the fold?
Are we shipping JavaScript for features that are not used on this route?

We delayed non-critical initialization and reduced early JavaScript execution.

The remaining render delay dropped:

0.5s → 0.2s

That brought the final measurement to:

1.2s LCP

Every Change, Side by Side

Here is the complete progression:

Step	LCP	Improvement
Baseline	6.0s	—
Faster TTFB	5.0s	-1.0s
Earlier LCP discovery	3.1s	-1.9s
Responsive optimized image	2.4s	-0.7s
Remove render-blocking work	1.5s	-0.9s
Reduce early main-thread work	1.2s	-0.3s

Overall:

6.0 seconds
     ↓
1.2 seconds

An 80% reduction.

But the table also shows why random optimization is inefficient.

Our largest improvement did not come from compressing the image.

It came from starting its request earlier.

The Final LCP Breakdown

The baseline:

TTFB                  1.5s
Resource load delay   2.1s
Resource load time    1.0s
Render delay          1.4s
                     -----
LCP                    6.0s

After optimization:

TTFB                  0.5s
Resource load delay   0.2s
Resource load time    0.3s
Render delay          0.2s
                     -----
LCP                    1.2s

That decomposition became much more useful than the Lighthouse score itself.

A score tells you something is wrong.

A timing breakdown tells you where to look.

What We Did Not Do

Performance work also requires knowing which optimizations not to make.

We Did Not Lazy-Load the Hero

Lazy loading is valuable for resources outside the viewport.

The LCP image was immediately visible.

Delaying it made no sense.

We Did Not Preload Everything

Preload is not a general-purpose “make the site faster” switch.

Making every resource high priority means nothing is actually high priority.

We reserved resource priority for things that genuinely affected the initial experience.

We Did Not Optimize Only for Lighthouse

Lab measurements helped us compare changes quickly.

But real users do not all have the same CPU, connection, viewport, cache state, or geographic location.

Field performance still mattered.

We Did Not Start by Removing Random JavaScript

“JavaScript is slow” was not a useful diagnosis.

We identified which main-thread work overlapped the LCP window and targeted that work specifically.

Our LCP Optimization Checklist Changed After This

Before this investigation, it was tempting to start with:

Compress the hero image.

Now the order is different.

1. Identify the Actual LCP Element

Do not assume it is the hero image.

Measure it.

2. Break LCP Into Its Components

Look separately at:

TTFB
Resource load delay
Resource load duration
Element render delay

3. Fix the Largest Delay First

Do not optimize what is easiest.

Optimize what is expensive.

4. Make the LCP Resource Discoverable

If it is an image, the browser should ideally discover it directly from the HTML.

Avoid unnecessary dependency chains.

5. Give the Resource Appropriate Priority

For an important above-the-fold image, consider:

fetchpriority="high"

Do not lazily load it.

6. Send Fewer Bytes

Use:

Appropriately sized images
Responsive sources
Modern formats
Sensible compression
Efficient delivery

7. Look at What Happens After Download

A fully downloaded LCP resource can still sit around waiting to render.

Inspect:

Render-blocking CSS
Synchronous scripts
Long tasks
Client-side rendering
Hydration
Third-party JavaScript

8. Validate With Real Users

Your development laptop is not your audience.

Measure the distribution of actual experiences.

Lab Data and Field Data Answer Different Questions

This distinction matters.

During optimization, we wanted to know:

Did this specific change improve the page?

Controlled lab testing is excellent for that.

But after deployment, the question changes:

Are real users actually getting a good experience?

That requires field data.

A single 1.2-second Lighthouse result does not prove that 75% of production users have a 1.2-second LCP.

Real users bring:

different phones
different networks
different locations
different cache states
different viewport sizes
different browser conditions

So we used lab measurements for iteration and field measurements for validation.

That prevented us from confusing a fast benchmark with a fast website.

The Biggest Lesson Was Not “Compress Your Images”

Image optimization mattered.

Caching mattered.

JavaScript mattered.

CSS mattered.

But none of those were the real lesson.

The lesson was:

LCP optimization works better when you debug time instead of applying performance tips.

We started with six seconds.

Then asked where those six seconds were going.

6.0s
├── 1.5s waiting for HTML
├── 2.1s waiting to request the hero
├── 1.0s downloading the hero
└── 1.4s waiting to render it

That immediately gave us a priority order.

By the end:

1.2s
├── 0.5s TTFB
├── 0.2s resource load delay
├── 0.3s resource load duration
└── 0.2s render delay

We did not make the browser magically faster.

We simply stopped making it wait.

Final Takeaway

If your LCP is six seconds, resist the urge to immediately install another optimization plugin, compress every asset, or rewrite the frontend.

Measure first.

Find the LCP element.

Break the metric into its timing components.

Then attack the largest delay.

For us, the sequence was:

6.0s
 ↓
5.0s   Faster server response
 ↓
3.1s   Earlier LCP discovery
 ↓
2.4s   Smaller responsive image
 ↓
1.5s   Less render blocking
 ↓
1.2s   Less main-thread work

Every change had a reason.

Every change had a measurement.

And that is the part of LCP optimization that matters most:

Do not optimize based on assumptions. Optimize based on evidence.

Debugging Governor Limits in a 4M-Record Org

Samcorp — Mon, 31 Aug 2026 12:22:20 +0000

Apex governor limits are easy to understand when your Salesforce org has 20,000 records.

They become much more interesting when one object has more than 4 million.

At that scale, code that looked perfectly reasonable for years can suddenly start failing with errors such as:

System.LimitException: Too many SOQL queries: 101

or:

System.LimitException: Too many query rows: 50001

or the less obvious:

System.LimitException: Apex CPU time limit exceeded

The instinctive response is usually:

“We need to bulkify the trigger.”

Sometimes that is correct.

But in a large-data-volume org, bulkification is often only the beginning.

This write-up walks through a representative debugging scenario involving Apex governor limits, roughly four million records, a trigger that had worked for years, and a fix that required changing more than a for loop.

Note: The scenario and object names below are representative examples designed to demonstrate the debugging process. They are not a customer post-mortem.

The Problem

Imagine an organization with a custom object called:

Usage_Event__c

It stores usage events associated with customer accounts.

Over several years, the table has grown to approximately:

4,120,000 records

Whenever an Account changes, an Apex trigger recalculates some usage information.

The original implementation looked roughly like this:

for (Account account : Trigger.new) {

    List<Usage_Event__c> events = [
        SELECT Id, Duration__c, Processed__c
        FROM Usage_Event__c
        WHERE Account__c = :account.Id
        AND Processed__c = false
    ];

    Decimal totalDuration = 0;

    for (Usage_Event__c eventRecord : events) {
        totalDuration += eventRecord.Duration__c;
    }

    account.Current_Usage__c = totalDuration;
}

If one Account was updated manually, nothing looked wrong.

One query.

A few records.

Fast execution.

Then a bulk integration started updating Accounts in groups of 200.

This is especially important in Salesforce integration architecture, where external systems can turn what looks like a single-record workflow into sustained bulk transactions.

That changed everything.

Failure #1: Too Many SOQL Queries

Salesforce can execute triggers for collections of records.

So if 200 Accounts enter this trigger, the query inside the loop may execute 200 times.

That quickly reaches the synchronous Apex SOQL limit.

The first failure was predictable:

System.LimitException: Too many SOQL queries: 101

The obvious fix was to bulkify the query.

Fix Attempt #1: Bulkify Everything

We collected the Account IDs first:

Set<Id> accountIds = new Set<Id>();

for (Account account : Trigger.new) {
    accountIds.add(account.Id);
}

Then queried everything in one operation:

List<Usage_Event__c> events = [
    SELECT Id, Account__c, Duration__c, Processed__c
    FROM Usage_Event__c
    WHERE Account__c IN :accountIds
    AND Processed__c = false
];

Much better.

We had gone from potentially 200 queries to one.

Problem solved?

Not quite.

The next test failed differently.

System.LimitException: Too many query rows: 50001

That was the moment the problem changed.

We were no longer debugging a simple trigger anti-pattern.

We were debugging large data volume.

Bulkified Does Not Mean Scalable

This is an important distinction.

This:

for (Account account : Trigger.new) {
    // SOQL
}

is bad because the number of queries grows with the number of trigger records.

But replacing it with one giant query does not guarantee scalability.

We had removed one problem:

Too many SOQL queries

and exposed another:

Too many queried rows

Our transaction was now logically bulkified but still trying to bring too much data into one Apex execution context.

That is where debugging Apex governor limits becomes more architectural.

Start With the Transaction, Not the Exception

When a governor limit fails, the last line in the stack trace is not always the real problem.

Instead, I want to know:

What entered the transaction?
        ↓
What automation executed?
        ↓
What queried data?
        ↓
How much data came back?
        ↓
What happened to that data?
        ↓
Which limit grew fastest?

That produces a much more useful investigation.

Salesforce transactions may include more than the Apex class you are currently reading.

They can involve:

triggers,
record-triggered flows,
validation logic,
managed-package automation,
workflow-related actions,
additional DML,
additional trigger executions.

So “my handler only runs three queries” does not necessarily mean the transaction runs only three queries.

The governor belongs to the transaction.

Not your class.

Add Limit Instrumentation

Debug logs are useful, but I also like adding temporary instrumentation around suspicious sections.

For example:

System.debug(
    'SOQL: ' +
    Limits.getQueries() +
    '/' +
    Limits.getLimitQueries()
);

System.debug(
    'Rows: ' +
    Limits.getQueryRows() +
    '/' +
    Limits.getLimitQueryRows()
);

System.debug(
    'CPU: ' +
    Limits.getCpuTime() +
    '/' +
    Limits.getLimitCpuTime()
);

System.debug(
    'DML: ' +
    Limits.getDmlStatements() +
    '/' +
    Limits.getLimitDmlStatements()
);

Then repeat the measurement around expensive operations.

For example:

System.debug('Before usage query CPU: ' + Limits.getCpuTime());

List<Usage_Event__c> events = [
    SELECT Id, Account__c, Duration__c
    FROM Usage_Event__c
    WHERE Account__c IN :accountIds
    AND Processed__c = false
];

System.debug('After usage query CPU: ' + Limits.getCpuTime());
System.debug('Query rows: ' + Limits.getQueryRows());

Do not leave noisy debugging permanently in production code.

But during diagnosis, these checkpoints make it much easier to see where the transaction begins consuming its budget.

The Limits That Mattered

For synchronous Apex, several limits deserve immediate attention during this type of investigation.

Resource	Synchronous Limit
SOQL queries	100
SOQL rows retrieved	50,000
DML statements	150
DML rows	10,000
Apex CPU time	10,000 ms

Asynchronous Apex provides more room for certain limits, including a higher SOQL-query allowance and significantly more CPU time.

But moving bad code asynchronous is not automatically a fix.

You can still write an inefficient asynchronous transaction.

Failure #2: The Query Was Too Broad

The next question was simple:

Why were we retrieving so many Usage_Event__c records?

The query looked innocent:

SELECT Id, Account__c, Duration__c
FROM Usage_Event__c
WHERE Account__c IN :accountIds
AND Processed__c = false

But the data distribution mattered.

Suppose:

Total Usage_Event__c records:     4,120,000
Processed__c = false:             1,480,000

Processed__c = false is not narrowing the table very much.

And several large customers might each own tens of thousands of usage events.

A filter can be logically correct while still being operationally expensive.

Check the Query Plan

This is where I would stop guessing.

Use Salesforce's Query Plan tooling and inspect how the optimizer expects to execute the SOQL.

Things I want to know include:

Is the query using an index?
Is Salesforce considering a table scan?
How many rows does the optimizer expect?
What is the relative cost?
Is one filter far less selective than expected?

This matters because:

Indexed does not automatically mean selective.

A field may have an index and still match so much of the table that using the index provides little benefit.

With four million records, data distribution becomes part of application design.

The Real Question: Why Are We Reading History?

Then we found the more important design problem.

Every Account update caused the application to ask:

Give me every unprocessed usage event belonging to this Account.

But most Account changes had nothing to do with usage.

Changing:

BillingCity

could trigger the same expensive calculation as changing a field that actually affected usage.

That is wasted work.

The first architectural improvement was therefore not another SOQL optimization.

It was reducing when the calculation ran.

Only Execute When Relevant Fields Change

Instead of recalculating on every Account update:

for (Account account : Trigger.new) {
    calculateUsage(account);
}

we identified whether relevant fields actually changed.

Conceptually:

Set<Id> accountsNeedingRecalculation = new Set<Id>();

for (Account account : Trigger.new) {

    Account oldAccount = Trigger.oldMap.get(account.Id);

    if (
        account.Usage_Mode__c != oldAccount.Usage_Mode__c ||
        account.Subscription__c != oldAccount.Subscription__c
    ) {
        accountsNeedingRecalculation.add(account.Id);
    }
}

Now an update to an unrelated field does not start an expensive calculation.

Sometimes the cheapest query is the query you never run.

Failure #3: CPU Became the Next Bottleneck

After reducing the query volume, another test exposed a different failure:

System.LimitException: Apex CPU time limit exceeded

Why?

Because retrieving data was only half of the problem.

The code was also doing work like:

for (Usage_Event__c eventRecord : events) {

    // calculations
    // string transformations
    // map lookups
    // nested conditions
    // repeated aggregation
}

At thousands of iterations per Account, CPU consumption started climbing quickly.

This is a common pattern when debugging governor limits.

You fix:

SOQL queries

then discover:

query rows

then fix that and discover:

CPU

Governor limits often expose inefficiencies layer by layer.

Stop Bringing Back Records You Only Want to Count

This was another useful question:

Do we actually need the individual Usage_Event__c records?

In our example, we mainly needed totals.

The original pattern effectively did this:

List<Usage_Event__c> events = [
    SELECT Account__c, Duration__c
    FROM Usage_Event__c
    WHERE ...
];

Map<Id, Decimal> totals = new Map<Id, Decimal>();

for (Usage_Event__c eventRecord : events) {

    Decimal current =
        totals.containsKey(eventRecord.Account__c)
        ? totals.get(eventRecord.Account__c)
        : 0;

    totals.put(
        eventRecord.Account__c,
        current + eventRecord.Duration__c
    );
}

But the database is much better positioned to perform aggregation.

Depending on the use case, an aggregate query may look more like:

List<AggregateResult> results = [
    SELECT
        Account__c accountId,
        SUM(Duration__c) totalDuration
    FROM Usage_Event__c
    WHERE Account__c IN :accountIds
    AND Processed__c = false
    GROUP BY Account__c
];

Now Apex receives summary results rather than every matching row.

That means:

fewer objects in memory,
less Apex iteration,
simpler business logic,
lower heap pressure,
potentially lower CPU consumption.

An aggregate query is not magic.

The underlying filters still need to scale.

But it is wasteful to retrieve 30,000 records when the application only needs one number.

Add a Real Data Boundary

The original query also had no meaningful time boundary.

It asked for every matching record from years of history.

But the business requirement only needed recent usage.

So instead of:

WHERE Account__c IN :accountIds
AND Processed__c = false

the query could potentially become something closer to:

WHERE Account__c IN :accountIds
AND Processed__c = false
AND CreatedDate >= :cutoffDate

CreatedDate is one of the fields Salesforce can index.

More importantly, the new predicate represents an actual business boundary.

Good query optimization is not:

“How can I trick the optimizer?”

It is:

“What data does the transaction genuinely need?”

Backfills Do Not Belong in a User Transaction

We still had another requirement.

Historical usage needed recalculation across millions of records.

Trying to do that during an ordinary record update would never be a healthy design.

This was a different workload.

So it deserved a different execution model.

Batch Apex for the Historical Work

Batch Apex is designed for processing large datasets in independent chunks.

A simplified example:

public class UsageBackfillBatch
    implements Database.Batchable<SObject> {

    public Database.QueryLocator start(
        Database.BatchableContext context
    ) {

        return Database.getQueryLocator([
            SELECT Id, Account__c, Duration__c
            FROM Usage_Event__c
            WHERE CreatedDate >= :startDate
            AND CreatedDate < :endDate
        ]);
    }

    public void execute(
        Database.BatchableContext context,
        List<Usage_Event__c> scope
    ) {

        // Process the current chunk.
    }

    public void finish(
        Database.BatchableContext context
    ) {

        // Logging, reconciliation, or next step.
    }
}

The important part is not simply that Batch Apex is asynchronous.

The important part is the transaction boundary.

Each batch execution gets its own governor-limit context.

Salesforce also allows a Batch Apex Database.QueryLocator to work with very large result sets—up to tens of millions of records—making it much more appropriate for a four-million-record backfill than a normal synchronous transaction.

Queueable vs Batch

A useful rule of thumb is:

Use Queueable Apex When

work is reasonably bounded,
you need asynchronous execution,
jobs need chaining,
you are processing a specific known workload.

Use Batch Apex When

the dataset itself is large,
millions of records may need processing,
work needs to be split into independent chunks,
limits need to reset between chunks.

Do not move everything asynchronous simply because synchronous code reaches a limit.

Choose the execution model based on the workload.

The Better Architecture

After working through the failures, the architecture looked very different.

Before

Account Update
      ↓
Trigger
      ↓
Query historical usage
      ↓
Load thousands of records
      ↓
Calculate totals in Apex
      ↓
Update account

Every transaction did far too much work.

After

Account Update
      ↓
Did a relevant field change?
      ↓
     Yes
      ↓
Query only required data
      ↓
Database-side aggregation
      ↓
Update required values

Historical work followed a separate path:

Historical Recalculation
      ↓
Batch Apex
      ↓
Selective QueryLocator
      ↓
Small execution scopes
      ↓
Process
      ↓
Reconcile results

This separation was much more important than any single code optimization.

What Actually Fixed the Governor Limits?

There was no magic line of Apex.

The fix was a combination of decisions.

1. Removed SOQL From Loops

This eliminated the immediate 101 queries failure.

2. Reduced Unnecessary Execution

The expensive logic stopped running for unrelated Account changes.

3. Made Queries More Selective

We stopped treating the four-million-row object like a small lookup table.

4. Used Query Plan Instead of Guessing

We verified how Salesforce expected to execute important SOQL.

5. Retrieved Less Data

Transactions asked only for data required by the business operation.

6. Moved Aggregation Toward the Database

Apex stopped iterating through large collections merely to calculate totals.

7. Separated Transactional Work From Backfills

User-facing transactions remained small.

Historical processing moved to Batch Apex.

8. Tested With Production-Like Data Volume

The original implementation had looked fine because the development environment did not reproduce production cardinality.

That last point deserves more attention.

Your Sandbox Can Lie to You

Suppose a development sandbox contains:

12,000 Usage_Event__c records

and production contains:

4,120,000

The same SOQL statement may behave very differently.

A test can prove functional correctness while telling you almost nothing about large-volume performance.

For important LDV paths, ask:

What is production row count?
How are records distributed?
Are a few parent records heavily skewed?
How many records match common filters?
Does the Query Plan change at production volume?
What happens when 200 records enter the trigger?
Which other automation shares the transaction?

Performance testing requires realistic cardinality, not merely realistic field values.

A Governor-Limit Debugging Checklist

When I see an Apex limit exception, this is the order I would investigate it.

SOQL

Is any query inside a loop?
How many queries does the entire transaction execute?
How many rows does each query return?
Are the WHERE filters selective?
What does Query Plan report?
Are we retrieving fields or records we never use?

DML

Is DML happening inside loops?
Can updates be collected and performed together?
Are we updating records whose values did not actually change?
Is DML recursively triggering additional automation?

CPU

Are there nested loops?
Are large collections repeatedly scanned?
Can Sets or Maps replace repeated searches?
Is Flow or other automation consuming the same transaction?
Are calculations being repeated unnecessarily?

Heap

Are we loading large record collections?
Are we selecting large text fields we do not need?
Can data be processed incrementally?
Can aggregation happen in SOQL?

Architecture

Does this work need to happen synchronously?
Is this a transactional operation or a backfill?
Would Queueable or Batch Apex fit better?
Can the system process only changes instead of recalculating history?

The Biggest Lesson

The hardest part of debugging Apex governor limits in a large Salesforce org is realizing that the limit exception is often not the real bug.

The real bug may be an incorrect assumption.

An assumption such as:

“This table will stay small.”

or:

“An Account only has a few child records.”

or:

“This trigger normally receives one record.”

or:

“Bulkifying one query makes the solution scalable.”

Those assumptions can remain invisible for years.

Then the organization reaches four million records and the architecture finally has enough data to prove them wrong.

Governor limits are not just restrictions developers need to work around.

They force us to design transactions with explicit resource boundaries.

And at scale, that usually leads to better systems.

These problems are why scalable Salesforce development requires more than working Apex code; data volume, transaction boundaries, automation, and long-term platform behavior all need to be considered together.

Final Takeaway

If I had to reduce this debugging exercise to five rules, they would be:

Measure the whole transaction, not just your class.
Bulkification is mandatory, but it is not the same as scalability.
Query selectivity matters more as data volume grows.
Do not retrieve data that the transaction does not need.
Move large historical workloads into execution models designed for them.

Four million records should not automatically break Apex.

But four million records will expose code that was designed as if the database would always remain small.

And that is exactly why large-data-volume debugging is such a useful test of Salesforce architecture.

What a Real Code Review Process Looks Like at Scale

Samcorp — Mon, 31 Aug 2026 11:47:46 +0000

On a five-person engineering team, code review can be simple.

Open a pull request.
Message a teammate.
Get an approval.
Merge.

At fifty, one hundred, or five hundred engineers, that approach starts to break.

Reviewers become bottlenecks. Pull requests sit untouched. Teams argue about style. Critical changes get the same level of attention as harmless configuration updates. Senior engineers spend hours reviewing code that automated tools could have checked in seconds.

A mature code review process is therefore not just about reading code before it reaches production.

It is a system for deciding:

What humans should review
Who should review it
What can be automated
Which issues should block a merge
How quickly reviews should happen
How much scrutiny each change deserves

Here is what that process actually looks like at scale.

Code Review Starts Before the Pull Request

One of the biggest mistakes teams make is treating review as something that begins after development finishes.

By then, many expensive decisions have already been made.

For meaningful changes, reviewers should understand the problem before debating individual lines of code.

A strong pull request should explain:

What problem are we solving?
Why is this change necessary?
What approach was chosen?
What risks does the change introduce?
How was it tested?

The reviewer should not have to reverse-engineer the purpose of a change from a 700-line diff.

For larger architectural decisions, the real review may begin even earlier through design documents, technical proposals, RFCs, or architecture discussions.

Code review should validate implementation decisions.

It should not become the first place where the team discovers what is being built.

The Author Reviews First

The first reviewer should always be the developer who wrote the code.

Before requesting review, the author should inspect the complete diff exactly as another engineer will see it.

That simple habit catches a surprising number of problems:

Debugging statements
Accidental file changes
Unclear names
Duplicate logic
Missing tests
Commented-out code
Unnecessary complexity
Incorrect configuration
Forgotten documentation

The pull request should also contain enough context for someone who was not part of the implementation.

A useful description might look like this:

Problem
Users can submit duplicate payment requests when the client retries.

Solution
Add an idempotency key to payment creation requests and persist processed keys.

Testing
- Added unit tests for duplicate requests
- Added integration test for retry behavior
- Tested timeout and retry scenarios locally

Risk
Database writes now include one additional lookup.

Rollback
Disable idempotency validation using the existing feature flag.

Compare that with:

Fix payment issue.

One gives the reviewer a starting point.

The other gives them detective work.

Automation Reviews the Mechanical Problems

At scale, humans should not spend their review time pointing out formatting errors.

That belongs to automation.

Before a pull request reaches serious human review, teams should automate repetitive checks through CI, static analysis, security scanning, and a strong software testing and QA process.

Formatting
    ↓
Linting
    ↓
Static Analysis
    ↓
Unit Tests
    ↓
Integration Tests
    ↓
Security Scanning
    ↓
Build Verification
    ↓
Human Review

Depending on the system, automated gates may also check:

Dependency vulnerabilities
Secrets accidentally committed
Test coverage changes
Database migration safety
API compatibility
Infrastructure policies
License restrictions
Type checking

This changes the role of the reviewer.

Instead of writing:

Please fix this formatting.

The reviewer can spend time asking:

Should this service own this responsibility?

That is a much better use of engineering attention.

Reviewers Are Selected by Ownership, Not Availability

Another process that fails at scale is:

Whoever is online, please approve this.

Approval is not valuable simply because someone clicked the button.

The reviewer needs enough context and expertise to judge the change.

Different parts of a system may require different reviewers.

For example:

Frontend Change
      ↓
Frontend Reviewer

Database Migration
      ↓
Database Reviewer

Authentication Logic
      ↓
Security-Aware Backend Reviewer

Cloud Infrastructure
      ↓
Platform / DevOps Reviewer

Shared API Contract
      ↓
Service Owner + Consuming Team

Large engineering organizations often formalize this with ownership rules such as CODEOWNERS, service ownership maps, or repository-level approval policies.

The goal is not bureaucracy.

The goal is getting the right eyes on the right risk.

Not Every Pull Request Deserves the Same Process

A two-line documentation correction should not require the same approval path as a change to authentication infrastructure.

Scalable teams use risk-based review.

Low Risk

Examples:

Documentation
Copy changes
Tests
Small internal refactors
Non-production tooling

Possible requirement:

One standard reviewer.

Medium Risk

Examples:

Business logic
Public APIs
Database queries
Significant UI behavior
Service integrations

Possible requirement:

Code owner or experienced reviewer.

High Risk

Examples:

Authentication
Authorization
Payments
Encryption
Database migrations
Infrastructure
Privacy-sensitive data
Production-critical architecture

Possible requirement:

Multiple reviewers or domain specialists.

This prevents two bad extremes:

Review everything heavily — which destroys engineering velocity.

or:

Review everything casually — which increases production risk.

Small Pull Requests Are an Engineering Advantage

Large pull requests are difficult to review properly.

Imagine receiving this:

2,840 lines changed across 47 files.

Most reviewers are already tired before opening it.

Large changes increase cognitive load. They also make it harder to identify the important part of the implementation.

Smaller changes are usually easier to:

Understand
Review
Test
Discuss
Merge
Revert
Debug later

A large feature can often be delivered as a sequence:

PR 1 → Data Model
PR 2 → Internal Service
PR 3 → API
PR 4 → UI
PR 5 → Feature Enablement

The feature may still be large.

The review units do not have to be.

The goal is not to chase an arbitrary line-count limit.

The goal is to create changes that a reviewer can understand without loading an entire project into their head.

A Reviewer Should Read More Than the Diff

A weak review focuses almost entirely on syntax.

A Strong reviews go beyond syntax and challenge broader software architecture and engineering decisions that can affect scalability, maintainability, and future development.

A useful review sequence looks like this.

1. Understand the Intent

Read the issue, description, design document, or requirement.

Ask:

Does this implementation actually solve the stated problem?

2. Review the Design

Before inspecting every line, look at the architecture.

Ask:

Is this responsibility in the correct layer?
Are we introducing unnecessary coupling?
Is there already an abstraction that solves this?
Is the solution more complex than the problem?

3. Review Correctness

Now inspect the implementation.

Consider:

Edge cases
Error handling
Null states
Concurrency
Retries
Duplicate operations
Failure recovery

4. Review Tests

Do not check only whether tests exist.

Ask whether they prove the behavior that matters.

A test suite that always passes is not automatically useful.

5. Review Operational Impact

Production code does not live inside a pull request.

Think about:

Logging
Monitoring
Performance
Database load
Backwards compatibility
Deployment
Rollback

6. Review Maintainability

Finally, ask:

Will another engineer understand this six months from now?

That question catches many problems that syntax-focused reviews miss.

Review Comments Need Severity

One of the most frustrating code review experiences is receiving fifteen comments without knowing which ones actually prevent approval.

Teams can solve this by establishing a simple vocabulary.

Blocking

Must change before merge.

BLOCKING: This endpoint allows the caller to update
resources without verifying ownership.

Question

The reviewer needs clarification.

QUESTION: What happens if the worker processes
this event twice?

Suggestion

A possible improvement, but discussion may be appropriate.

SUGGESTION: Could this validation live in the
domain service instead of the controller?

Nit

Minor and non-blocking.

NIT: This variable name could be slightly clearer.

Now the author knows exactly where attention is required.

More importantly, small preferences stop accidentally blocking useful code.

Review the Code, Not the Developer

Good review culture matters as much as review mechanics.

Compare:

Why did you write this in such a complicated way?

with:

This branch introduces several states that seem difficult to reason about. Could we simplify the flow by returning early here?

The second comment discusses the code.

That distinction matters.

Code review should create shared understanding, not defensiveness.

Comments should explain why something is a problem whenever possible.

Instead of:

Change this.

Prefer:

This cache is shared across requests, so mutating it here could create inconsistent results under concurrent traffic. Could we keep this value request-scoped?

Now the review transfers knowledge instead of simply issuing instructions.

Review Speed Is Part of the Process

A technically excellent review system can still fail if developers regularly wait two days for feedback.

Slow reviews create several problems:

Developers lose context
Branches drift
Merge conflicts increase
Work in progress grows
Releases slow down
Engineers begin avoiding smaller improvements because review is painful

Teams therefore need an expectation around response time.

That does not mean interrupting engineers every ten minutes.

It means code review is treated as team work, rather than optional work performed when someone eventually has spare time.

A healthy process might establish expectations for:

Time to First Review

How long does a pull request wait before someone looks at it?

Time Between Review Rounds

After the author pushes changes, how quickly does the reviewer return?

Total Time to Merge

How long do ordinary changes remain open?

The exact targets depend on the organization.

What matters is that review latency is visible.

Separate Human Judgment From Policy

Suppose every reviewer repeatedly comments:

Please add a test.

Eventually, that should probably become a pull request requirement.

If reviewers repeatedly catch:

This repository must not import package X.

That may belong in static analysis.

If every PR needs:

Database team approval required.

That belongs in ownership rules.

A mature code review process converts repeated human instructions into automated policy whenever possible.

Humans should spend their limited attention on problems that require judgment.

Machines should enforce deterministic rules.

A Practical Scalable Review Flow

Putting everything together, a mature workflow may look like this:

Developer Writes Change
        ↓
Developer Self-Reviews
        ↓
PR Includes Context + Testing + Risk
        ↓
CI Runs Automated Checks
        ↓
Ownership Rules Select Reviewers
        ↓
Reviewer Checks Design + Correctness
        ↓
Domain Specialists Review High-Risk Areas
        ↓
Author Addresses Blocking Feedback
        ↓
Required Approvals Complete
        ↓
CI Passes Again
        ↓
Merge
        ↓
Deployment + Monitoring

Notice something important.

The reviewer is only one part of the system.

That is what makes the process scalable.

What Should Teams Measure?

You can improve a process much faster when you can see where it is failing.

Useful signals include:

Metric	What It Can Reveal
Time to first review	Reviewer availability problems
Time to merge	Overall process friction
Review rounds	Unclear requirements or weak initial submissions
Pull request size	Changes that are difficult to review
Reviewer workload	Bottlenecks around senior engineers
Reopened defects	Problems escaping review
CI failure rate	Changes being submitted before they are ready

Metrics require care.

If developers are rewarded simply for reducing review time, they may approve too quickly.

If teams are judged only by pull request size, developers may artificially split changes that should remain together.

Metrics should reveal problems.

They should not replace engineering judgment.

Signs Your Code Review Process Is Not Scaling

You probably have a process problem when:

The same two senior engineers review everything
Pull requests regularly wait days for feedback
Reviews contain mostly formatting comments
Huge pull requests are normal
Nobody knows who owns important parts of the codebase
Approvals are treated as formalities
Reviewers debate personal style preferences
CI regularly fails after review begins
Authors provide almost no context
Security-sensitive code gets ordinary review
Developers merge just to avoid another review round

These problems rarely disappear by telling people to:

“Review better.”

They require changes to the system.

The Goal Is Not Perfect Code

A review process should raise quality without making engineering impossible.

Perfect code does not exist.

A reviewer can almost always find another variable to rename, another abstraction to reconsider, or another edge case worth discussing.

The better question is:

Does this change leave the system healthier than it was before?

If the answer is yes, important risks are handled, tests provide reasonable confidence, and remaining comments are minor preferences, merging may be better than continuing the discussion indefinitely.

Good engineering organizations protect quality.

Great ones learn how to protect quality without stopping delivery.

Final Thoughts

A real code review process at scale is not a mandatory approval button sitting between development and deployment.

It is an engineering system, Authors provide context and review their own work. Automation removes mechanical problems, Ownership rules find the right reviewers.

Reviewers focus on design, correctness, risk, tests, and maintainability. High-risk changes receive deeper scrutiny, Feedback clearly separates blockers from suggestions.

And the organization pays attention to review speed as well as review quality. When those pieces work together, code review becomes more than a defect-catching exercise.

It becomes one of the mechanisms through which an engineering organization maintains its architecture, spreads knowledge, develops engineers, and keeps a growing codebase understandable.

Five Architecture Decisions We Regret

Samcorp — Sat, 29 Aug 2026 11:55:44 +0000

Architecture decisions rarely look terrible when we make them.

Most begin with reasonable intentions:

“We'll need this when we scale.”
“This will keep everything flexible.”
“Microservices will make teams independent.”
“Let's create one shared layer so we don't duplicate code.”

Six months later, that “flexibility” can become the reason a simple feature touches twelve files and three services.

Looking back at common architecture mistakes, the painful ones usually aren't caused by choosing the wrong programming language or framework.

They're caused by making the system more complicated than the problem required.

Here are five decisions I would challenge much earlier.

1. Splitting Into Microservices Too Early

The original reasoning sounds convincing:

Independent deployments
+ Independent scaling
+ Clear ownership
+ Smaller services
= Better architecture

But there's another side:

Service discovery
Network failures
Retries
Distributed tracing
Message queues
Deployment pipelines
Data consistency
Observability

A function call that once looked like:

const user = getUser(id);

can become:

Service A
   ↓ HTTP
Service B
   ↓ Database
   ↓ Event
Service C

Now every boundary is also a failure boundary.

Microservices can be excellent when there is a real need for independent deployment, scaling, ownership, or isolation.

But adopting them before understanding the domain can create a distributed monolith: many deployable services that are still tightly coupled.

What we'd choose instead

Start with strong module boundaries.

Application
├── Users
├── Orders
├── Payments
├── Inventory
└── Notifications

Keep extraction possible.

Then turn a module into a service when there is a measurable reason to do it.

2. Sharing One Database Across Everything

This one feels incredibly productive at first.

Three services need customer information?

Easy:

User Service ─────┐
Order Service ────┼──→ Shared Database
Billing Service ──┘

No duplicate data.

No APIs.

No synchronization.

Then someone changes:

customers.status

Suddenly three systems can break.

The database schema has quietly become the API.

That's one of the more expensive architecture mistakes because the coupling isn't always visible in application code.

What we'd do differently

Make ownership explicit.

Users Service
     ↓
Users Data

Orders Service
     ↓
Orders Data

Other components should interact through deliberate interfaces rather than silently depending on internal tables.

This doesn't mean every application immediately needs ten databases.

It means:

Know who owns the data before everyone starts owning it.

Architecture planning and data ownership are exactly the kinds of decisions worth examining during software architecture consulting before implementation makes those boundaries expensive to change.

3. Building Abstractions Before We Needed Them

Developers dislike duplication.

Sometimes too much.

Suppose two payment providers initially look similar:

processStripePayment()
processPayPalPayment()

So we immediately design:

AbstractPaymentProvider
        ↓
PaymentStrategyFactory
        ↓
PaymentProviderAdapter
        ↓
PaymentExecutionContext
        ↓
ConcreteProvider

Beautiful diagram.

Until the third provider behaves completely differently.

Now we're fighting our abstraction instead of solving the business problem.

What we'd do instead

Accept small amounts of duplication while the domain is still becoming clear.

Three similar implementations can teach us more than one premature abstraction.

The better sequence is:

Build
  ↓
Observe repetition
  ↓
Understand differences
  ↓
Extract abstraction

Not:

Predict future
  ↓
Build abstraction
  ↓
Hope reality agrees

Good architecture removes unnecessary complexity.

It doesn't hide complexity behind more interfaces.

4. Designing for Imaginary Scale

“We should design this for 10 million users.”

How many users do we currently have?

“About 8,000.”

Planning for growth is responsible.

Designing every subsystem around hypothetical extreme scale is different.

It can lead to:

Kafka
Redis
Elasticsearch
Multiple databases
Kubernetes
Microservices
Complex caching
Event-driven everything

before the product actually needs them.

Now developers spend time operating architecture instead of improving the product.

What we'd do differently

Design for:

Current load
+
Expected near-term growth
+
A reasonable safety margin

Then identify the parts that must remain replaceable.

For example:

Today

App
 ↓
PostgreSQL

Later

App
 ├── Cache
 ├── Read replicas
 ├── Search
 └── Queue

A simple architecture with clean boundaries can evolve surprisingly far.

The goal isn't to predict every future requirement.

The goal is to avoid making future change unnecessarily expensive.

That's also why maintainability and controlled growth matter in custom software development: scalability should be designed around actual product requirements rather than added as infrastructure for its own sake.

5. Treating Architecture Decisions as Permanent

This may be the biggest regret.

We once tended to think:

Choose architecture
       ↓
Build system
       ↓
Architecture finished

Real systems behave more like:

Choose
  ↓
Build
  ↓
Learn
  ↓
Measure
  ↓
Change
  ↓
Repeat

The architecture that works for:

5 developers
10k users
1 product

may be wrong for:

50 developers
2M users
8 product lines

That doesn't necessarily mean the original architecture failed.

The context changed.

The real mistake is refusing to reconsider the decision.

Architecture Decision Records Help

One practice I would adopt earlier is recording important decisions.

An ADR doesn't need to be complicated.

Decision:
Use PostgreSQL as the primary datastore.

Context:
Small team, transactional workload,
no demonstrated need for multiple databases.

Why:
Operational simplicity and team experience.

Trade-offs:
Some future workloads may require
specialized storage.

Revisit when:
Database becomes a demonstrated bottleneck.

Now future developers know why something exists.

Without that context, architecture can become archaeology.

The Pattern Behind These Mistakes

These five decisions look different:

Premature microservices
Shared databases
Premature abstractions
Imaginary scale
Permanent decisions

But they have something in common.

We optimized for a future we didn't understand yet.

A better principle is:

Make today's architecture simple while keeping tomorrow's important changes possible.

That doesn't mean ignoring scalability.

It means earning complexity.

A Better Architecture Checklist

Before introducing a major architectural component, ask:

□ What problem does this solve today?

□ Do we have evidence that the problem exists?

□ What new complexity does it introduce?

□ Who will operate it?

□ What happens when it fails?

□ Does it create additional coupling?

□ Can we postpone this decision?

□ How difficult will it be to reverse?

□ What would trigger us to reconsider it?

If we can't answer those questions, the architecture probably isn't ready.

Final Takeaway

The architecture mistakes I worry about most aren't:

Wrong framework
Wrong database
Wrong cloud

They're decisions that create unnecessary coupling and complexity.

A useful architecture doesn't need to predict the next five years perfectly.

It needs to make today's system understandable and tomorrow's changes possible.

So instead of asking:

“What is the most scalable architecture we can build?”

I'd ask:

“What is the simplest architecture that satisfies our current constraints without trapping us later?”

That question tends to produce much healthier systems.

And usually fewer regrets.

Debugging Production Memory Leaks in a Node Monolith

Samcorp — Sat, 29 Aug 2026 11:26:55 +0000

The graph looked harmless at first.

09:00  → 420 MB
12:00  → 610 MB
15:00  → 790 MB
18:00  → 1.1 GB

Traffic dropped.

Memory didn't.

Eventually:

FATAL ERROR:
JavaScript heap out of memory

When a large Node.js monolith starts behaving like this, the tempting fix is:

node --max-old-space-size=8192 app.js

Sometimes that buys time.

It doesn't tell us why memory keeps growing.

A useful Node memory leak investigation starts by identifying which kind of memory is growing, then finding what continues retaining it.

First: Confirm It's Actually a Leak

Not every growing memory graph means leaked JavaScript objects.

Start with:

const memory = process.memoryUsage();

console.table({
  rss: memory.rss,
  heapTotal: memory.heapTotal,
  heapUsed: memory.heapUsed,
  external: memory.external,
  arrayBuffers: memory.arrayBuffers
});

These numbers describe different things.

heapUsed
→ JavaScript objects currently using V8 heap

heapTotal
→ Heap currently allocated by V8

external
→ Memory associated with native/C++ objects

arrayBuffers
→ ArrayBuffer/SharedArrayBuffer memory,
  including Node Buffers

rss
→ Overall resident process memory

That distinction is important.

If:

heapUsed ↑ continuously

we may have retained JavaScript objects.

If:

RSS ↑
heapUsed ≈ stable

we should also investigate Buffers, native modules, allocator behavior, or other memory outside the normal V8 heap.

Don't diagnose from one metric.

Look at the Shape of Memory

Healthy applications allocate memory constantly.

Garbage collection should recover much of it.

Conceptually:

Memory
  │      /\      /\      /\
  │     /  \    /  \    /  \
  │____/    \__/    \__/    \__
  └─────────────────────────────

The interesting pattern is:

Memory
  │             /\
  │          __/  \
  │       __/
  │    __/
  │___/
  └─────────────────────────────

The baseline keeps climbing after garbage collection.

That's when I start asking:

What is still referencing objects that should already be dead?

In a Monolith, Start With Long-Lived Containers

A large Node monolith usually has plenty of objects designed to live for the entire process lifetime.

That makes them excellent leak candidates.

Look at:

Global Maps
Caches
Module-level arrays
Event emitters
Queues
Session stores
Timers
Request registries
WebSocket connections

Consider this innocent-looking cache:

const userCache = new Map();

async function getUser(id) {
  if (!userCache.has(id)) {
    userCache.set(id, await loadUser(id));
  }

  return userCache.get(id);
}

What's the eviction policy?

There isn't one.

If new IDs continue arriving:

10,000 users
   ↓
50,000 users
   ↓
500,000 users
   ↓
Map keeps everything

Technically the objects are still reachable.

So garbage collection is behaving correctly.

The application is the thing leaking.

Watch Event Listeners

Another common source of retention is listeners that are added repeatedly but never removed.

function handleRequest(req) {
  eventBus.on("user-updated", () => {
    updateSomething(req.user);
  });
}

Now imagine that happens for every request.

The listener retains its closure.

The closure may retain:

Request
User
Metadata
Other objects

Instead, lifecycle-sensitive listeners need lifecycle-sensitive cleanup.

function handler(data) {
  // process event
}

eventBus.on("user-updated", handler);

// Later
eventBus.off("user-updated", handler);

The real question isn't just:

“How many listeners exist?”

It's:

“Why is this listener still alive?”

Timers Can Quietly Retain Large Object Graphs

Consider:

function startJob(customer) {
  setInterval(() => {
    refreshCustomer(customer);
  }, 60_000);
}

That callback retains customer.

If jobs are repeatedly created without clearing old intervals:

Timer
  ↓
Closure
  ↓
Customer
  ↓
Associated objects

remain reachable.

Always know who owns a timer and when it ends.

const timer = setInterval(runJob, 60_000);

// lifecycle ends
clearInterval(timer);

This becomes particularly important in monoliths because dozens of unrelated modules may create their own background jobs.

Take Heap Snapshots — Carefully

Guessing eventually stops being useful.

A better investigation is:

Snapshot A
    ↓
Generate representative traffic
    ↓
Wait / reproduce growth
    ↓
Snapshot B
    ↓
Compare

We're looking for objects whose retained population or retained size keeps increasing.

Typical suspects might include:

Array
Map
Object
String
Closure
Buffer wrappers
Application-specific classes

Then inspect the retaining path.

For example:

Global Cache
   ↓
Map
   ↓
UserSession
   ↓
Orders[]
   ↓
Large response objects

Now we have something actionable.

Heap snapshots can temporarily consume significant CPU and memory, so capturing them blindly on a heavily loaded production instance is risky. Use a safe diagnostic strategy appropriate to your infrastructure.

For long-lived Node applications, this kind of monitoring and troubleshooting belongs alongside ordinary Node.js application development rather than being treated only as emergency work after an out-of-memory crash.

Don't Ignore Buffers

Suppose:

heapUsed = relatively stable
RSS      = climbing

The JavaScript heap may not be the primary problem.

Node applications handling:

File uploads
Image processing
Streams
Compression
Large API payloads
Sockets
Binary protocols

often work heavily with Buffer.

That's why I monitor:

const {
  rss,
  heapUsed,
  external,
  arrayBuffers
} = process.memoryUsage();

If external or arrayBuffers grows alongside RSS, the investigation changes direction.

This is one reason simply increasing the V8 heap limit can completely miss the actual problem.

Reproduce One Workload at a Time

A monolith might handle:

REST API
Background jobs
Uploads
Reports
WebSockets
Emails
Scheduled tasks
Integrations

If everything runs simultaneously, identifying the leak becomes difficult.

Isolate workloads.

Baseline
   ↓
Run API traffic
   ↓
Measure

Baseline
   ↓
Run report generation
   ↓
Measure

Baseline
   ↓
Run background jobs
   ↓
Measure

You might discover:

Normal API traffic
→ stable

WebSocket traffic
→ stable

Report generation
→ memory never returns

Now the search space is dramatically smaller.

Add Memory Observability Before the Next Incident

At minimum, track:

RSS
Heap used
Heap total
External memory
ArrayBuffer memory
Process restarts
OOM events
Request volume
Job volume

Then correlate memory growth with application behavior.

Memory spike
     +
Report jobs increased
     +
Large Buffers increased

is much more useful than:

Server uses 1.8 GB.

Production performance should be treated as part of ongoing web application maintenance and optimization, especially when a monolith has accumulated years of APIs, integrations, jobs, and shared application state.

The Debugging Order I Prefer

When investigating a Node memory leak, I use roughly this sequence:

1. Confirm sustained growth
        ↓
2. Compare heapUsed vs RSS
        ↓
3. Check external / Buffer memory
        ↓
4. Correlate growth with workload
        ↓
5. Reproduce the suspicious path
        ↓
6. Capture safe heap snapshots
        ↓
7. Compare retained objects
        ↓
8. Follow retaining paths
        ↓
9. Fix ownership / cleanup
        ↓
10. Repeat the workload

Then ask one final question:

Does memory return to a stable baseline?

If not, keep investigating.

Common Places I'd Check First

For a production Node monolith:

□ Unbounded Map/Set caches
□ Global arrays
□ Event listeners
□ setInterval/setTimeout
□ Request objects retained by closures
□ WebSocket connection state
□ Unfinished promises
□ Large Buffers
□ File-processing pipelines
□ In-memory sessions
□ Background queues
□ ORM/database result retention
□ Application metrics with unbounded labels

Notice that most leaks aren't mysterious V8 bugs.

They're usually object ownership problems.

Something was created.

Something should have released it.

Something didn't.

Final Takeaway

The worst way to debug a Node memory problem is:

Memory high
   ↓
Increase heap
   ↓
Restart server
   ↓
Wait
   ↓
Memory high again

A better approach is:

MEASURE
   ↓
CLASSIFY
   ↓
REPRODUCE
   ↓
SNAPSHOT
   ↓
FIND RETAINER
   ↓
FIX
   ↓
MEASURE AGAIN

The key lesson is that a Node memory leak isn't simply:

“Node is using too much RAM.”

The useful question is:

“Which memory is growing, and what is keeping it alive?”

Once you can answer that, debugging stops being guesswork.

And in a large monolith, reducing the search space is often half the fix.

DEV Community: Samcorp

Our Accessibility Audit Found 340 Issues Here's the Pattern

What the Audit Found

Pattern #1: Contrast Problems Were Mostly Design-System Problems

Pattern #2: Forms Looked Labeled but Were Not Programmatically Labeled

Pattern #3: alt Text Was Not Just Missing — It Was Often Wrong

Pattern #4: Mouse Testing Hid Keyboard Problems

Pattern #5: Icon Buttons Were Easy to See but Hard to Understand

Pattern #6: Heading Problems Came From Styling Decisions

The Most Important Finding Was Duplication

Automated Testing Was Useful — but It Was Not the Audit

What We Changed After the Audit

The Data Changed Our Priorities

The Biggest Lesson

Final Takeaway

What Broke When We Adopted React Server Components

What We Expected

Break #1: "use client" Spread Further Than Expected

What We Changed

Break #2: Browser Assumptions Started Failing

Lesson

Break #3: Our Shared Objects Stopped Crossing the Boundary

What We Changed

Break #4: Moving Data Fetching to the Server Created Waterfalls

The Fix

Lesson

Break #5: Caching Became Part of Application Correctness

What We Changed

Break #6: Server Functions Looked Safer Than They Actually Were

Lesson

Break #7: Debugging Moved Away From the Browser

Break #8: Our Tests Were Testing the Wrong Architecture

Server Logic

Client Components

Integration

One More Lesson: RSC Added a New Security Patch Surface

What Actually Worked

1. Server by Default

2. Small Client Islands

3. Serializable Boundaries

4. Parallel Data Fetching

5. Suspense Around Slow Work

6. Explicit Cache Decisions

7. Authorization Inside Server Functions

8. Server Observability

The Architecture We Ended Up With

What We Would Do Differently

First: Map Interactive Areas

Second: Map Server-Only Data

Third: Define the Boundary

Fourth: Inspect Data Dependencies

Fifth: Define Cache Rules

Sixth: Rebuild the Test Strategy

Seventh: Add Server Observability

The Biggest Lesson

Final Takeaway

Five AI Features We Built and Then Deleted

The Features Looked Good in Demos

1. AI Search Answers

Lesson

2. The AI Code Review Bot

Lesson

3. Automatic Customer Support Replies

Lesson

4. AI-Generated Dashboard Insights

Lesson

5. The Universal AI Assistant

Lesson

What These Failures Had in Common

We Changed How We Evaluate AI Features

Deleting the Feature Was Sometimes the Best Result

The Biggest Lesson

Final Takeaway

What Broke When We Moved From GPT to Self-Hosted

Why We Wanted to Self-Host

Break #1: API Compatibility Was Not Behavior Compatibility

Break #2: Our Prompts Were More Model-Specific Than We Thought

Then We Discovered Chat Templates

Break #3: Structured Output Became Our Problem

We Added Schema Enforcement

Pattern #3: `alt` Text Was Not Just Missing — It Was Often Wrong

Break #1: `"use client"` Spread Further Than Expected