How I removed the middleman, one phone call at a time

#softwareengineering #architecture #devops #business

In the mid-2000s I worked on a content distribution platform that served over a hundred storefronts. Books, music, comics — different shops, the same underlying content, each store with its own branding.

Among those hundred-plus storefronts, several dozen were technically resold through a third party — a viewer-side company that operated its own delivery infrastructure on top of ours. Their architecture required us to push every file into per-storefront slots on their servers, before the end user could download anything. Hundreds of slots. Per file.

This article is about how those slots disappeared, and what I learned along the way.

How the layers got there

When the platform launched, we were not the dominant player. The market for digital content on Japanese mobile phones was already shaped by a few established companies, and one of them sat between us and several dozen of our storefronts. They had built a viewer client that the end user installed on their phone. They had built a delivery server that the viewer pulled from. They had a working business.

We came in as the catalog provider. The deal was simple in principle: their viewer, our content. To make their viewer work, we had to put files into the directory structure their delivery server expected — one directory per storefront, the same file copied into each.

For a small catalog, this was tolerable. For a growing catalog, it was wasteful. A single new title meant dozens of identical writes — same bytes, different paths — every time we published.

We were the latecomer. We didn't push back. We did the work their architecture required.

The phone call

One Friday afternoon I got a phone call from the lead engineer on their side.

"Could you reduce the frequency of your pushes? Our servers are having trouble keeping up."

The phrasing was polite, but the request was unusual. We had recently parallelized our delivery batches — moved from sequential pushes to concurrent ones — and the volume of writes had multiplied. We were doing what we should have been doing for our growing catalog. They were absorbing it on infrastructure that hadn't been designed for that rate.

I could have agreed. Reducing batch concurrency was a one-line config change on our side. It would have made their afternoon easier and our publishers' release schedules a little slower.

I didn't agree. I made a counter-proposal.

"What if we stopped pushing per-storefront entirely? Could your viewer fetch directly from us, with the storefront as a parameter, instead of from a pre-placed copy?"

There was a pause on the line. Not a hostile pause. The kind of pause an engineer makes when an idea reorganizes itself in their head.

"Let me think about that and get back to you," they said.

The two steps that followed

It would be neat to say they called back the next day and we shipped it. That's not what happened. The conversation that started on that phone call took roughly two years to finish, in two discrete steps. Neither of the steps was urgent. Each of them was treated, on both sides, as a quiet refactor.

Step one: stop the per-storefront duplication.

The first thing we agreed was that, even if the viewer still fetched through their infrastructure, the file did not need to be duplicated per storefront on disk. They modified their delivery server to look up a single canonical copy, with the storefront determined by a parameter on the URL. We changed our push so that each file was uploaded once, not duplicated per storefront.

Our delivery batch shrank by an order of magnitude. Their disk usage dropped accordingly. Nothing changed for the end user.

Step two: remove the dependency on their server entirely.

The second thing was larger. Since the file was already canonical on our side, there was no architectural reason for the data path to go through their infrastructure at all. They updated their viewer to fetch directly from our delivery infrastructure, using the parameters they had previously injected on their server side. They kept the catalog metadata and the user-account layer; everything else went away.

When the last storefront cut over, the middleman delivery server was no longer in the data path. They turned off the disks they had been running for us. We turned off the placement processing we had been running for them.

What I learned

It's tempting to read this as a story about how a smaller company outmaneuvered a larger one. That's not what happened. What happened was that I made a proposal in the middle of a routine operational complaint, and over two years both sides incrementally chose to remove a layer that had stopped earning its keep.

There are a few things I took from it.

A symptom request is an opportunity to ask about the cause. "Please reduce your push frequency" is a symptom request. The honest underlying question is "why are we doing this many writes at all?" Most operational complaints I've received over the years, in the moment, sounded like requests for symptom relief. Some of them turned out to be requests for an architectural conversation that nobody had explicitly opened yet.

When I get a symptom request now, I always try to surface the cause version of the same request, at least once, to see what happens. Sometimes nothing. Sometimes a two-year refactor.

Conviction has to come with the other party's interest. I was able to propose removing their delivery infrastructure because, by that point, our volume was material to them. A meaningful share of their viewer's traffic came from our catalog. Removing a layer that had stopped paying for itself was, for them, a saving — not a loss. Without that alignment, the same proposal would have been a polite no.

The conviction wasn't only mine. It was conviction plus a real shape of interests. Without the second, no amount of conviction moves a vendor's roadmap.

Latecomers can rearrange the layer cake. When we started, we accepted their architecture because they were the established party and we were not. Two years later, we redefined the architecture together. The thing that changed in between was not their willingness to change. It was our weight in the system. The latecomer can become the party who proposes the new shape, once the latecomer is no longer small.

I think a lot of the operational shapes that look fixed in a given industry are, in fact, shapes that nobody has been in a position to renegotiate. The middleman in our case wasn't there because anyone defended its existence. It was there because nobody yet had reason to ask whether it should be.

Twenty-four years later

I'm telling this story now because I keep finding myself doing variations of it. A vendor surfaces an operational complaint. I look at the complaint, and I find — sometimes — that the structure underneath the complaint is the actual subject of the conversation. The complaint is the symptom. The conversation is the door.

Not every complaint opens a door. Most of them are just complaints. But some of them are an invitation to ask whether the current shape is the right one, and a year or two later you find that the shape is no longer there.

This is not what you should do. This is what twenty-four years has taught one specific person to do.

Built with Claude (Opus).

Earlier in this series: