DEV Community: Stephen Ohimor

Pages of Our Latency

Stephen Ohimor — Tue, 22 Sep 2026 02:12:35 +0000

The Missing Adgroups

There are many categories of scaling failures, and chief among them are those that begin in unnoticeable silence. I've encountered the same pagination bug at three of my last four companies. I've deduced the underlying reason to be nearly identical in all cases. If you're working at scale, the default pagination method should be cursor based. In the age of A.I., this rings more true now than ever. Here's the incident that made the cost concrete, the architectural reason it happens, why it matters more now that agents consume APIs too, and how to catch it before it becomes an outage.

🏗️ What this post covers:

What happened: a Friday-afternoon incident where data quietly went missing for a few of our largest clients

Why it happens: page-based pagination's count query scales linearly with table size

Why it's the default anyway: ORMs optimize for zero-to-one speed, not structural assumptions

Why it matters more now: AI agents consuming APIs want stable, deterministic cursors

How to catch it early: growth-based observability, before it becomes an outage

I originally wrote this finding as a journal entry some 4+ years ago. The Buyer Cloud (Beeswax at the time of the original post) models the concept of Adgroups. An Adgroup contains the necessary domain structure to transact on bid requests downstream. Almost every ad-tech company has some concept of this object, and it typically contains entities such as an Advertiser or Brand, Line Items, and references to Creatives. For live bidding, this data must be synced from the UI/API to serving infrastructure in a reasonable amount of time. The faster the better — seconds are great, minutes are acceptable, hours can result in disaster and costly Make Goods.

Performance really matters. What works for clients with a few hundred line items doesn't work for those with millions. The default pagination strategy in most ORMs is usually page based, because the strategy requires no assumptions about the underlying table structure. Most frameworks are designed to take an app from zero to one quickly. An application can unevenly grow out of this strategy over time. That's where the story begins.

On a Friday afternoon, exactly when things should have been winding down, a DM from customer support landed in my inbox. That late in the day, it could only mean an unspoken problem had been brewing for a day or two. I could taste the hysteria through Slack.

The Problem: Data was missing for some (but not all) clients.

I tend to look at problems from multiple angles. This particular issue was known to recur randomly (a word used when engineering hasn't quite figured out the triggering conditions). Not only was it random, but the frequency was per client and could potentially auto-resolve. It appeared in various forms once or twice a month, and because clients used the platform in behaviorally different ways, the escalation priority could be assigned different values (e.g. P1 to P4 — more on that in a future post).

Even though I'd go on to help resolve hundreds of escalations over the years, that first pupil-widening surge — the sympathetic nervous system snapping into action — was an unavoidable part of the job.

A pattern emerged after a thorough review of endpoint latency in Datadog and a cursory review of the adgroup data. Degradation had crept up over the course of the month, then spiked sharply — but only for a few of our largest clients. After years of experience, you go through the usual suspects: Was there a recent change? No. Did the client behavior change? Did they add a substantial amount of data recently? Nope. The affected clients hadn't grown suddenly; they had simply accumulated enough data for an old architectural assumption to become expensive. Conversing with customer support and one of the only stateside engineers available (our team was split between the U.S. and Europe), I was led by my experience to our pagination strategy.

AWS's RDS dashboard includes a quick panel for long-running queries. These are often sizable queries generated by Django's ORM, ranging from 200 to 1,000+ lines. Re-running them sometimes produced result sets of 15 million records or more, but that wasn't the real problem. The culprit was the count query required to compute the page index. Page-based iteration scales linearly while cursor based is done in constant time.

Taking a step back, the larger context is that page-based strategies are rooted in an older web experience (think Web 2.0): a human user searching for specific records within a relatively small dataset. It predates advanced UI components and libraries that support sophisticated search. Parsing large datasets was typically left to ETL processes — transferring gigabytes or terabytes of data over HTTP wasn't common. However, in the age of A.I. — where agents may be the dominant consumers of APIs — cursor-based pagination will likely become the norm. If you're modernizing your API for agent consumption, this is an area worth prioritizing. One advantage is stable cursors, which help ensure records aren't skipped or duplicated. Deterministic ordering also lets agents resume after being paused, and checkpointing allows a cursor to be saved and reused later. While I haven't seen "Spot Agents" (akin to EC2 spot instances), it's probably only a matter of time before Anthropic or OpenAI pitch the idea to enterprise organizations as a creative way to cut costs.

The default pagination strategy is on the cusp of being a technical atavism.

Another contributor to this kind of failure is observability. When it comes to scaling, this nonfunctional requirement seldom gets the attention it deserves. As far as I know, there are no native, out-of-the-box database solutions for monitoring table size. However, both MySQL and Postgres provide exporters. Pair that with Prometheus to store the data in a time-series database and Grafana to query it, and you get a very useful alert.

delta(mysql_info_schema_table_size{table="users"}[7d])

// or to see the percentage increase over the last week

delta(mysql_info_schema_table_size{table="users"}[7d])
  /
(mysql_info_schema_table_size{table="users"} offset 7d)

I've always said that Datadog pays for itself the first time it helps you avert an outage. If you have access to it, leveraging Database Monitoring (DBM) is a no-brainer. DBM became available in 2021/2022 — but with so many features in the product, it may easily go unnoticed. Once enabled, setting up Change Alerts to monitor the growth rate is the surest path to reducing an outage of this nature.

While observability is key to identifying potential future escalations, several non-functional capabilities also benefit from first assessing whether your platform should adopt a cursor-based strategy. Taking a systems-thinking approach, performance improves through constant-time lookups. As a result, reliability increases because another threat vector has been reduced, and availability improves as well. If your database is performance-throttled, dependent systems will likely start returning 5XX HTTP errors. The user experience will look broken. That Zendesk ticket will get created, and your customer support team will DM you the same way this post begins.

Cursor-based pagination is one of the rare scaling choices that improves performance, reliability, and availability at the same time. Don't wait for missing records — or a weekend escalation — to reveal an unsafe default. Find your fastest-growing tables and move them to cursor-based pagination before scale makes the decision for you.

Originally published at Skyscrapers On Sand.

Battle of Attrition

Stephen Ohimor — Thu, 03 Sep 2026 23:26:31 +0000

Lessons Learned from an Acquisition

An acquisition lives or dies on the unglamorous organizational plumbing-- brand identity, staffing, tooling, shipping cadence, culture-- not the press release. I learned this managing attrition through the acquisition of a small startup called Beeswax, first as an Engineering Manager, later as a Director of Engineering. Three things made the difference, and none of them are retention bonuses: mapping ownership so gaps don't hide, treating 1:1s as the highest-leverage thing you do in year one, and defining a vision honest enough to survive being wrong. I'll touch on hiring and firing-- they matter-- but attrition is where acquisitions are actually won or lost.

This isn't a niche problem. M&A volume cratered under macro pressure in 2022 and 2023, but ad-tech had already bucked the trend by 2024-- 73% YoY growth in ad-tech M&A alone-- and by 2025 overall deal value was back above $5 trillion, ahead of the last two decades' average. If you're the acquirer trying to protect the team you just bought, or the team on the other side of that deal, this is worth reading closely-- more of you are going to be living this than at any point in the last five years.

The story starts in 2021. The founders stayed on for a year, but in truth, most of the acquisition work happened after they left.

The shape of this post. An acquisition lives or dies on the unglamorous plumbing-- not the press release.

Why now: M&A-- especially in ad-tech-- is accelerating, so more of you are living this than at any point in the last five years.

Diagnose first: which of the four team states you're in, and whether the problem is your team or the org around it.

Three things that made the difference: Ownership (map the AORs), 1:1s (the highest-leverage tool in year one), Vision (a north star honest enough to survive being wrong).

Attrition's Long Tail

Every team I've run falls into one of four states: Firefighting, Treading Water, Recovering (paying down technical debt), or Driving Innovation. I've taken teams through all four, and leadership matters most in the first three-- a team that's merely fighting fires or treading water is where an acquisition either stabilizes or quietly unravels. Before you diagnose which state your team is in, though, figure out whether the problem is your team or the org around it. As an engineering manager, there's little you can do to fix the org itself; as a director, your leverage grows, and your strongest tactic becomes managing up without overstepping. People like to say impact can come from anywhere regardless of title-- true in small orgs, generally false in big ones.

Two other processes matter too: hiring (generally to meet new business objectives) and firing (a deduplication of roles and talent-- note that the mothership often chooses its own).

Botching the decision making in the first year of an acquisition will lead to a series of cascading problems with each cycle bringing an escalating reverberation of eventual failure. Regardless of the state of hiring and firing, attrition is a part of every acquisition (exceptions may be acqui-hires). Retention bonuses are only so effective-- if you want to keep your talent, keeping the founders on for as long as possible is a much stronger lever. The question of what the founders will do or how long they might stay was a consistent one. Salary was secondary and only became a larger concern after the founders departed.

Most of my early experience was centered around refueling the fire that was nearly extinguished by the journey to acquisition. In the early stages, my team would be considered small compared to the behemoth of an org I would lead a few years later-- roughly 4 employees and 4 contractors. I still believe 8 engineers is a realistic team size for one leader to run well.

With the advent of A.I., executives will push for larger teams anyway. This movement will largely fail-- not because a team that size can't function, but because there aren't enough leaders skilled enough to handle 15+ engineers moving at the speed of Claude.

For additional context, the baseline for attrition in tech has been historically high. The median job tenure for a startup employee receiving equity compensation is a mere 2.2 years, the average tenure at Big Tech is a little over a year. Oddly enough, employees tend to yield the best ROI between years 2 and 4 of their tenure. If leaders are frustrated with delivery-- retention is one of the best places to look.

While 75% of employees depart in the first 3 years, management tells an even more asymmetrical tale-- 40% of managers are lost in the first 24 months. In a hostile takeover, this rate surges past 50%.

100% of the direct reports on my initial team stayed for 2+ years-- some would go on to stay 5+ years. I managed to keep the majority of our highest achieving contractors as well. Here are a few things that made the difference:

First: ownership.

Map out individual team member's AOR.

Find the gaps, communicate those gaps with your team and your manager. If the gap is high value, assign that space to your highest available talent and determine the risk if this gap goes unfulfilled.

Acquisitions are the end goal for a lot of organizations-- the successful exit. For the people inside, though, it's just as often a beginning: a career at a startup and a career at a large parent company look very different, and this is where engineering leadership can make a difference. Your AOR document doubles as a first resource for that conversation.

Prioritize all areas; keep that model at the forefront of your mind when you talk with Product, Customer Service, Leadership and other teams. If you're asked to dedicate time and energy to something that has no mapping-- this is a red flag or an undiscovered area of responsibility.

Second: 1:1s.

1:1 meetings have the most value in the first year. I've found reports to oscillate between doubt and certainty from sprint to sprint. After expanding my team to include both Frontend and Backend engineers-- following my own advice, I met with all of them. "So many broken windows--" one engineer said, followed by a long sigh, as if he had been holding on to a sacred truth turned burden. He was referring to the constant surveillance necessary to keep the system up; the sheer volume of errors and warnings that may result in yet another late night. Complex systems and broken things weren't foreign to me, but I came to understand how those challenges may be energy depleting. I let those words hang after I acknowledged that we had quite a bit of work to do. I didn't try to convince the engineer in this meeting-- I did make a promise to myself to make things better (eventually the team did succeed as we scaled from ~80 integration tests to over 1000 and improved system stability).

When you hear raw feedback about the work environment or platform-- see if it correlates with other team members' perspectives. One of the advantages of being an engineering leader is the ability to triangulate and filter the signal from the noise. One gap I see in leaders is reaching for the last explanation they were told.

Document your 1:1s. Some leaders keep a shared document; keeping your own notes lets you stay prepared for each meeting, pick up where you left off, and check on prior goals. It gives a dry, methodical process a real, natural cadence.

Third: vision.

The most difficult adjustment is the transition from autonomous innovator to subordinate member of a corporate matrix. Having worked at startups and small companies as well as large Fortune 100 organizations, I found the transition challenging. It was a frequent talking point in 1:1s as well as a source of contention across all parts of the acquired organization. That adjustment doesn't have to happen overnight, though-- experienced companies will generally provide a rather large grace period (often years) for the acquired entity to maintain culture. Leverage what exists, but address the long-term concerns: change is inevitable.

Define a vision. It doesn't have to be perfect. A team needs a north star. If you're uncertain-- make that known but provide enough confidence, data and logical arguments to ensure the team can rally around an achievable goal. When you get new information, update your vision. If enough change occurs, formally address the team. Leaving your team in the dark sows irreparable discord.

Conclusion

Acquisitions are won or lost long before anyone notices-- in the quiet decisions about who owns what, who stays, and who gets a real 1:1 instead of a status update. My team didn't just survive the acquisition because of a retention bonus or a slide deck; it thrived because we mapped ownership gaps early, treated 1:1s as a first-year priority instead of a nice-to-have, and gave people a vision to hold onto while the org chart underneath them was still being redrawn.

None of this is glamorous. None of it shows up in a press release. But the platform that survives an acquisition is usually the one whose leaders did the unglamorous plumbing first-- attrition math, ownership maps, and a north star that gets updated honestly instead of quietly abandoned. If you're heading into an acquisition, resist the urge to wait for clarity from above. The org chart will eventually get sorted out. What won't wait is whether the people who know the system best still believe it's worth staying for.

Sources

Originally published at Skyscrapers On Sand.