How we bypassed cloud challenges to increase our downloads by 102% with users from a competitor.
The deadline wasn't ours
TV Time announced it was shutting down its operations. For those who had used the app for years, it was a decade of logged movies and series about to evaporate. Thousands of people faced the same problem with no effective solution.
TV Time offered some help to its soon-to-be orphaned users. It was possible to export all activity tracked in the app into a .zip file before the final date. However, this export only solved part of the problem - the user would have the file but nowhere to put it.
I work at Omelete Company, and we have Chippu, an app that allows its users to track their history in the same way as TV Time. The reasoning was simple. The hard part was the deadline: we got the green light with only five days left before TV Time went offline. After that, users who hadn't exported their history would lose everything because there would be nowhere left to export from.
Spoiler: It worked! Between July 12th and August 3rd, the feature resulted in a 102% increase in app store downloads compared to the previous month. We had records of people installing the APP specifically so they wouldn't lose their TV Time history.
Business result
Chippu downloads during the TV Time migration period.
Here, I will share the hidden limits, the obvious ideas we discarded, and a bug that only blew up in production.
The map: three problems, not one
Before getting to the code, an overview is worthwhile. "Import TV Time history" sounds like a single task. It's actually three, and each has a different bottleneck.
- Receive and validate the file. The user sends a ZIP. We need to receive the file without freezing the app, validate if it is a TV Time export, and reply to the user as fast as possible. Bottleneck: response time.
- Extract the data in the cloud. Tens of megabytes of CSV to read line by line, cross-reference with the content database, and transform into records. This takes time, and the user has already closed the app. Bottleneck: execution limits.
- Save and notify. Save everything without duplicating data, keep the backend up and running for active users, and send an email the second the process finishes. Bottleneck: knowing when it finished.
The rest of the text follows this order.
Full flow diagram
Three chained bottlenecks. Each solved for a different reason.
Part 1 - Receive and validate
The promise the architecture couldn't keep
The initial design was as follows: the app requests a temporary URL from a serverless service, sends the ZIP directly to storage, and this triggers background processing in another serverless service. Simple and Fast - the file doesn't need to pass through our server.
Then came the requirement that complicated everything: the APP needed the serverless service to confirm if the ZIP had the correct files before moving forward.
Seems reasonable. It was impossible.
The reason was due to the sequence of executions, not the technology. The serverless service delivers the temporary URL and responds before the file is saved in storage. The upload happens later and doesn't even pass through our infrastructure. Therefore, it's not possible for a response to wait for a file that hasn't been sent yet - it would be like asking the cashier to confirm your purchase before you even enter the store.
The way out was to stop trying to do both things in a single call. I created a second endpoint that the app calls right after the upload. It confirms delivery, validates the file, and returns the verdict right away.
Two calls instead of one. But each has a clear definition of what it can guarantee - and that is worth more than saving a single request.
The 212-byte trick
The question that arose during development was this: wouldn't downloading the entire file just to look inside it be exactly the slowdown point of this validation?
It would. And the answer didn't come from any more expensive service. It came from two details that were already there:
A ZIP stores its index at the end of the file. The list of what's inside - the Central Directory - is in the last few bytes, not at the beginning.
The HTTP protocol knows how to download a piece of a file. The Range header allows you to request "only bytes from X to Y", and the storage respects that.
Putting these two points together, we reached a conclusion: to know what's inside the ZIP, you just need to read its very end.
We just had to convince Python's ZIP library that this remote file was a local file. To do this, we created code that pretends to be a regular file, but under the hood, it only downloads the pieces we need to read:
Range=f'bytes={self._pos}-{end}'
That's essentially it. The library does what it always does - jumps to the end of the file looking for the index - and receives only that fragment.
Result achieved: a 39,323-byte ZIP was validated by reading 212 bytes.
And the gain isn't just the punctual savings. The cost of validation now depends on how many files are in the index, not the weight of the content. A 500MB ZIP validates in the same time as a 40KB one. The bottleneck wasn't optimized: it ceased to exist.
The 212-byte trick
To know what's inside a ZIP, you don't need to open the ZIP.
What doesn't appear in the diagram
Two small decisions in this phase, the kind nobody draws but everybody pays for if forgotten.
A Pre-signed URL doesn't limit size. Whoever signs the URL doesn't control what the client sends through it. The app warns about the 50MB ceiling before the user spends their bandwidth, but client-side validation is a suggestion, not a guarantee. So the server checks the real size at validation time and, if it exceeds the limit, deletes the file and rejects it.
A rejected ZIP is also deleted. This isn't just storage hygiene - the real reason for this line will appear later, in a bug that almost cost us dearly.
Part 2 - Extract the data in the cloud
The same trick, now inside the CSV
Validating a ZIP is reading an index. Processing the entire history is reading everything, line by line. The bottleneck changes: now the problem is that a serverless service has a 15-minute lifespan, and a cinephile's history doesn't fit in that.
The idea was the same as the validation: never download the whole file. The processor reads 5MB blocks starting from a position, and each block is a byte-range request.
The position is counted in bytes, not in characters. Counting after decoding the text could fail right at the first accent. Since a letter in UTF-8 can take up more than one byte, the math silently breaks.
Every block break falls in the middle of a line. With absolute certainty, always. So the code looks for the last line break in the block, processes only up to there, and returns the exact position where the incomplete line begins.
Serverless service that requeues itself
The piece that dissolves the 15-minute limit is missing, and it is almost obvious once seen.
Before reading each block, the serverless function asks how much lifespan it has left. If less than 30 seconds remain, it sends a message to itself with the exact position where it stopped - and finishes cleanly, without errors.
A few seconds later, a new execution starts exactly from there.
The effect is that there is no such thing as a file that is too big. There is a file that needs more rounds. It is the same logic as the 212-byte trick: a platform limit ceased to be a blocker and became a parameter of the algorithm.
Both times, the way out didn't come from hiring larger infrastructure. It came from understanding the file format and the protocol that transports it.
The function that requeues itself
The function gives up before it dies, and leaves a note for the next one.
Six seconds between each call - and why the reason matters more than the number
Here comes the most interesting restriction of the project, and it's not technical: it's about coexistence.
Chippu's infrastructure didn't allow me direct access to the database, so the recording of the TV Time history had to pass through the exact same API that served the public. The weight of this? The import didn't consume limits of an external tool, but it competed for resources directly with our own users. All of this running 24 hours a day to empty a huge queue against the clock.
A burst of thousands of requests there wouldn't just generate a rate-limiting error. It would make the app slow for users who weren't doing any importing, didn't know a migration was happening, and just wanted to rate a movie before going to sleep.
So the decision was deliberately counterintuitive: we intentionally made our own feature slower. One write every six seconds. Nobody asked for this number; it came from choosing a pace that would cause irrelevant noise to the app's normal traffic.
And here it was necessary to look at the documentation carefully: you cannot configure a queue to deliver one message every six seconds.
To bypass this bottleneck, a pause was implemented inside the function itself, before each write.
Except that, by itself, this pause is useless. Fifty executions waiting six seconds in parallel is the same thing as fifty executions with no pause at all.
It can only truly control the flow when combined with a lock: a maximum of one function execution at a time. Then the pause applies to the whole system.
Two settings complete the architecture, and they exist for recovery reasons, not performance:
One item at a time: If the time runs out, only that item returns to the queue. Using batches would make the system reprocess things that had already succeeded.
Generous invisibility timeout: The time the message remains invisible in the queue needs to be much greater than the function's duration.
When each call costs six seconds, cutting calls becomes a product decision
A six-second spacing changes how the system works: every avoided call is six seconds less waiting to finish the import. "Reducing requests" was no longer an optimization but a requirement!
Two decisions were defined based on this:
A full season becomes a single call. Chippu's backend allowed marking a title, a season, or an episode as watched. When we resolve the title, we also get how many episodes each season has. If the person watched all of them, we issue a season mark. If they watched a part, one per episode.
A fully watched 24-episode season dropped from 24 requests to one. Timed on the clock: from 2 minutes and 24 seconds to 6 seconds.
Where the API allowed, we sent in batches of 20: favorites, 'watch later', and simple marks traveled together. The detailed mark, however, remained individual. Since each item has its own date, season, and platform, grouping would mean losing data.
The rule that stuck: if it's standard, it goes in a batch; if it has details, it goes alone.
Combined, both drop a user with thousands of episodes from the thousands of requests tier to the hundreds. Without touching the infrastructure.
One season, two paths
When every request costs 6 seconds, grouping stops being an elegance and becomes a requirement.
Not duplicating without a database
The serverless function that dispatches the actions has no memory. It doesn't query a control table, it doesn't record what it has already done. The entire guarantee of not duplicating comes from the queue — and this was a choice, not an economy.
Each message had a hash calculated from the fields that define the action — user, type, title, season, episode. Reprocessing the same upload generates exactly the same messages, and the queue discards repetitions on its own.
The intermediate aggregation follows the same philosophy. Instead of accumulating episodes in the function's memory — which evaporates between executions — the accumulation happens inside the database, using sets.
A set is consistent by definition: adding the same episode twice doesn't duplicate anything. Reprocessing a block after a failure is safe by construction, not by control.
The dictionary that already existed in-house
Here is the most annoying problem of this whole migration: TV Time CSVs had no universal content identifier. There was only the title name as it appeared on TV Time, and nothing else.
"Breaking Bad" is not data. It is text. And text alone doesn't match with a database.
Fortunately, the missing piece already existed in-house, built for another purpose. Omelete has a movie and series content database. No one built it thinking about history migration.
That was exactly what solved the history migration.
The resolution took two requests: the raw TV Time name goes to the Omelete content database and returns with a public identifier; this identifier goes to the Chippu backend and returns with the internal ID, plus the map of seasons, episodes, and platforms — which is precisely what feeds the full season rule from the previous section.
The cache per content
Two requests per title is the most expensive operation in the entire pipeline. And the cache key does not include the user.
The consequence is direct: the first one to import Breaking Bad pays for the two requests. Everyone else pays zero — including those who import months later. This is not just an optimization, but the difference between the system draining in hours or days.
Two details that make the cache yield even more:
Failure is also cached. A title with no match is registered as "not found" and is never queried again. This way we avoid one failed search per person.
The key is the normalized title. Before becoming a key, the name loses accents, turns lowercase, and has spaces collapsed. Amélie, Amelie, and AMELIE become 'amelie' to facilitate the lookup in the saved cache.
Cache per content, not per user
In a mass migration, catalogs overlap much more than one imagines.
Part 3 — Save and notify
The asynchronous system doesn't know it finished
With everything decoupled by queues, a problem appeared that wasn't in the plan: no component knows that a user's work has finished.
The processor knows it published the actions, but not that they were executed. The executor knows it performed an action, but not that it was the last one. The queue knows how many messages it has, but not whose they are.
And we needed this instant, because it's what triggers the email to a person who closed the app hours ago.
The answer was a new component, which processes nothing: it just observes the system from the outside and decides when to advance. Three of its decisions are worth noting.
The waiting queue respects the order of arrival. Each upload enters a backlog ordered by the original file's date. With a short deadline and concentrated volume, the defined rule was: whoever sent first is served first.
The completion signal is the silence of the queues — including what is currently executing. Every two minutes, it sums the available messages and the ones being processed right now. Just the available ones isn't enough: a queue might report zero precisely because everything is in execution. This is the classic false positive, and summing both categories is what prevents it. On top of that, there's a 90-second wait before declaring completion, providing a window for an executing message to materialize.
One upload at a time. The next one only starts when the previous one has completely drained. Combined with the lock of one simultaneous execution, this makes the system entirely predictable: at any moment there is one user being processed and a maximum of one request executing against the Chippu backend — which, remember, is the same backend serving the live app.
Popcorn is ready
When that user's backlog drains and the queues fall silent, the email goes out: "Popcorn is ready! Your TV Time history was successfully imported."
The delivery is best-effort and doesn't block the queue. If the email fails, the job is marked as completed anyway, with the delivery result recorded alongside. Blocking the entire backlog of users because the email service went down would be trading a small problem for a big one.
The bug that no local test catches
This is an interesting part, because it has nothing to do with the cloud.
Upon completing an upload, the system began to delete the already extracted CSVs. The motivation was good: they served their purpose, take up space, and their absence acts as a physical lock against accidental reprocessing. The deletion is conservative — it only happens if the original ZIP still exists, guaranteeing that everything can be regenerated.
What I didn't realize is that another function, written days before, relied on the existence of those files. The endpoint the APP queries to know if the person had already imported responded based on "does the extracted file exist?".
The instant the cleanup went live, every already processed user started being reported as if they had never uploaded anything. The APP would request the file again and run the risk of the data being duplicated in Chippu.
The fix was to change the logic: the correct indicator is the original ZIP, which is permanent. That is why the rejected ZIP is deleted, back in Part 1 — the file's presence became the signal, and leaving invalid garbage behind would corrupt that signal.
"If the file was extracted, then the user uploaded it" seemed practical. It was a time-bomb trap.
The wrong signal
An invisible bug: two serverless services, written days apart, disagreeing on what a file's existence means.
Keeping the ZIP allowed us to change our minds
The original ZIP is never deleted. Not after being processed, not after the email. It was kept out of a "don't throw away the source" instinct — and it ended up unlocking something that wasn't in the plan.
As we started to better understand the TV Time format, the list of useful files grew and the already received uploads wouldn't need to be requested again. An endpoint reapplies the current interpretation onto the ZIPs that are already stored.
A detail matters a lot here: the reinterpretation, by default, only re-reads — it does not re-execute the actions. Redoing the writes requires asking explicitly.
Separating 're-read the file' from 'execute the actions' is what guarantees the safety of running all this in production without affecting those using the app. The two things seem like one until the day you need to do the first without the second.
The principle: as long as the raw data is available, any interpretation decision is reversible. Keeping the ZIP costs pennies and allows us to change our minds about the file processing — exactly what this project demanded.
How I knew it worked before the deploy
A short deadline usually becomes an excuse to test in production. We did the opposite, with a hybrid local environment — not entirely fake, not entirely real:
| Layer | In the local test |
|---|---|
| Storage and content database | Real, read-only, with CSVs from a true upload |
| Database and queues | Local versions, in Docker |
| Chippu backend reading | Simulation that forwards to the real backend |
| Writing to the Chippu backend | Simulation that only logs the exact payload |
The value is in the asymmetry. Reading is real, so the test faces the true data with all the dirtiness that third-party data has. Writing is fake, so the environment is physically incapable of touching production. And given that the writing would hit the backend serving real users, this guarantee was a requirement.
Since the simulation logs the literal payload of each action, that was where I validated the exact format expected by the backend before any deploy.
The panic button needs to be faster than a deploy
With a window of days, "reverting" needs to be faster than publishing code. That is why the system has two levels of control.
The structural level changes on deploy, leaving a trail: turning queue consumers on or off, adjusting concurrency, changing the log level. A deliberate detail: the consumers boot up turned off by default. Publishing code and activating consumption are separate acts — you can deploy a fix without it immediately starting to drain the accumulated backlog.
The operational level changes in seconds, without a deploy. Pausing the entire pipeline is one command; resuming it, another. This exists for a very concrete reason: if the APP started showing signs of slowness, I needed to be able to shut down the import as soon as possible.
What remained
In the end, the cloud doesn't solve problems by itself. Building this architecture wasn't about connecting services, but about dealing with rigid platform restrictions using shortcuts that nobody documents. It was understanding that scaling infrastructure isn't always the answer and, above all, accepting that the most logical decision is usually the worst one.
Three ideas yielded more than all the others:
The platform's timeout limit became a parameter of the algorithm twice. We beat the first bottleneck with a partial read of the ZIP, and overcame the serverless 15-minute ceiling with a function that recreates itself in the queue. Both times, the way out came from understanding the protocols — not from paying for more infrastructure.
Cache by content, not by user. The same table that accelerates one import accelerates all the following ones, from anyone. In a mass migration, this single choice defines whether the system drains in hours or days.
The intentionally slower feature was the most mature decision of the project. We could have imported everything much faster. We could also have left the APP crawling for thousands of people who had nothing to do with the migration. Choosing the slow pace wasn't a technical limitation: it was deciding who had priority.
Five days. Five functions, two queues, and a stack of decisions that only seemed obvious afterward.
TV Time went offline on schedule. The users' history did not.
Top comments (0)