DEV Community: Sidharth Sangelia

I replaced a five day spreadsheet with a cron job, then spent two weeks fixing what I got wrong

Sidharth Sangelia — Fri, 28 Aug 2026 11:59:51 +0000

Every twenty or twenty five days someone at the company would sit down and build a rate card.

A rate card is a spreadsheet. Countries down one side, weight slabs across the top, a price in every cell, one sheet per carrier. A customer asks "what does it cost me to send 5kg to Germany", and the rate card is the answer, except the rate card has to answer that question for every country and every weight at once, because you do not want to be on a call saying "let me check and get back to you".

Building one took four to five days. Not four to five days of thinking. Four to five days of opening a vendor portal, typing in a country, typing in a weight, waiting, copying a number into Excel, and doing it again. Then the vendors changed their rates and you did it again from scratch.

I did the maths on it once and stopped enjoying my afternoon. Twenty six countries times thirty weight slabs times four vendors is 3,120 lookups. At ten seconds each, assuming nobody blinks or gets a phone call, that is eight and a half hours of pure typing. Nobody was doing it at ten seconds each, and nobody was doing all 3,120. They were doing a subset, for the lanes that came up most, and everything else was a guess or a phone call.

So the number that actually mattered was not the four to five days. It was that we could not answer questions we did not already know the answer to.

Why you cannot just call the API

One note before I go further. We buy from four vendors, and I am going to call them Vendor A through Vendor D for the rest of this post. Which ones they are is not the interesting part, and our commercial terms with them are not mine to publish. What matters is that they behave differently, and most of what follows is about the ways they differ.

Vendor A resells the widest catalogue and is the only one that prices on real dimensions rather than a weight you hand it.
Vendor B is the slowest by a distance and the only one that publishes a rate limit.
Vendor C sits in the middle on everything.
Vendor D is a direct account with a carrier rather than a reseller, so it returns exactly one service per call.

Now, the obvious fix for the spreadsheet problem is "call the vendor APIs instead of typing into their portal". I thought that too.

The problem is the shape of the API. Every vendor we work with exposes a rate endpoint that answers exactly one question: one origin, one destination, one weight, one box, one shipment. You give it a shipment and it gives you back the services it can fly and what each one costs. That is a perfectly reasonable API if a human is booking a parcel. It is useless if you want to compare anything.

Comparison needs a grid. The API gives you a point. To get the grid you need 3,120 points, and 3,120 sequential HTTP calls against four vendors is not something that happens inside a web request. It is not something that happens inside anything with a timeout, actually, which is most things.

There is also a rate limit problem. Vendor B publishes ratelimit-policy: 10;w=60 on every response. Ten calls a minute. Their share of the matrix alone is 780 calls, which at ten a minute is 78 minutes of continuous polite requesting, and that is assuming nothing fails.

So the answer was never going to be "fetch it when someone asks". The answer had to be that the data already lives with us, and the question becomes a SQL query.

The inversion

Here is the whole idea in one line: stop asking the vendors at read time, ask them on a schedule, and store every answer.

A cron job walks the matrix every five nights, hits every vendor for every country at every weight, and writes each answer to Postgres. Building a rate card stops being a data collection problem and becomes a data formatting problem. The Excel file is generated from a table.

That is the part that sounds easy in a blog post. The interesting bits are all in what "the matrix" actually means and what happens when a night goes wrong.

Defining the matrix was harder than building it

Before writing any job I had to decide what a cell of this grid actually is, and it turns out a cell is a surprisingly opinionated thing.

Destination. A vendor prices to a postcode, not to a country. But a rate card has a country column, because customers think in countries. So each country in the matrix carries one canonical postcode, and I picked the capital city's for every one of them. London SW1A, Berlin 10115, Washington 20001.

This rule is consistently wrong in a known way and I decided that was fine. Canberra is not Sydney. Pretoria is not Johannesburg. Washington DC is not New York. Express carriers price zones, and the capital is not always the commercial centre, so a few rows describe a real lane that is not the busiest lane in that country. What makes it safe is that the rule is consistent and every stored row records which postcode produced it. If a row ever looks wrong against a live quote, it is one line to change.

Then the Gulf broke it. The UAE, Qatar and Hong Kong do not have postal codes. The vendor APIs demand the field anyway, so those rows carry 00000 and a syntheticPostcode: true flag that travels with the data, not just with the config, because "why is Dubai empty" is a question somebody asks six months later when the config file has moved on.

Weight. Thirty slabs, log spaced:

0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2,
2.5, 3, 3.5, 4, 4.5, 5,
6, 7, 8, 9, 10,
12, 14, 16, 18, 20,
25, 30, 40, 50, 75, 100

Dense below 5kg, coarse above 20kg. Two reasons stacked on each other: most parcels sit under 5kg, and the price per kg moves fastest down there, so if anyone interpolates between two samples that is where a straight line is least wrong. Above 20kg the curves flatten and an extra sample buys nothing.

The box. This one I did not see coming.

Three of the four vendors take a chargeable weight directly. You say "5kg" and they price 5kg. Vendor A does not. Vendor A takes real dimensions, computes its own volumetric weight, and charges on whichever is higher. Send it no dimensions and it refuses to quote. Send it a big box and your row labelled 2kg is quietly holding the price of a 6kg shipment.

So every slab needs a box, and the box has to guarantee that volumetric stays under actual, or the column label stops being true. The tempting answer is a 1x1x1 cube, which guarantees it and also describes a parcel made of neutronium. A vendor is entitled to reject that, and the stored dimensions become useless as a record of what was asked.

What I settled on is a cube sized so volumetric lands at about half the actual weight:

volumetric = side³ / 5000, and we want that to equal weight / 2
so side = cbrt(2500 × weight)

Nine centimetres at 250 grams, 63 centimetres at 100kg. Dense but physically ordinary. Rounded up to whole centimetres because Vendor A rounds up anyway, and rounding here means the number I store is exactly the number I sent.

Why all of this is a TypeScript file and not an admin screen. An admin screen for editing countries and postcodes would be friendlier. It would also be the wrong call, and this is the design decision I feel most strongly about in the whole system. A canonical postcode is not a preference. It is the definition of what every stored number means. Change London from SW1A to somewhere in the Highlands and every GB row silently starts answering a different question, with nothing in the data to say so. That belongs in a commit somebody reviewed, next to the reasoning, not in a text input with a Save button.

The file also carries a SWEEP_CONFIG_VERSION, bumped whenever a change makes new rows incomparable with old ones. A moved slab, a changed postcode, a changed box formula. Not for adding a country, because existing lanes stay exactly as comparable as they were. And every run stores a full verbatim copy of the config it used, denormalised on purpose, because in two years the config file will say something different and that JSON blob is the only record of what those numbers ever meant.

Why Inngest, and what a "lane" is

About a year ago I hit Vercel's function timeout for the first time and had no idea what to do with a job that simply could not finish inside a request. That is what pushed me to Inngest, and it has since become one of the load bearing pieces of the whole product. Emails, PDF generation, Excel generation, carrier bookings, and now this.

For the sweep, the thing I actually needed was not "background job". It was durable steps. A three hour job that talks to four flaky third party APIs is going to get interrupted, and I wanted the interruption to cost one call rather than the whole night.

The unit of work I settled on is a lane: one vendor, one country, thirty weights.

That decision is worth explaining because I went back and forth on it. The two obvious alternatives:

One run per cell (3,120 runs). Maximum isolation, every cell independent. But 3,120 events fanned out at once, 3,120 runs of platform overhead, and nowhere natural to hold "I am currently walking a weight ladder for this vendor". Pacing becomes something you have to coordinate externally instead of something the loop just does.

One run per vendor (4 runs). Cheap on events, and a disaster on failure. One run holding 780 sequential calls means the failure blast radius is a quarter of the matrix, and a single wedged run takes out every country for that vendor.

The lane sits in the middle. 104 runs, each holding thirty steps. A cell is a step, so it retries on its own and memoises when it succeeds. The loop over weights is ordinary for loop code, which means pacing is just a sleep between iterations. And a country dying takes one country with it.

The whole system is three Inngest functions:

plan-rate-sweep      cron, fans out
   ↓ 104 events
sweep-rate-lane      one vendor × one country × 30 weights
   ↓ 1 event (from the last lane to finish)
finalise-rate-sweep  count, judge, alert if bad

The Inngest decisions that actually mattered

Fan out in one send, not 104

await step.sendEvent(
  "fan-out-lanes",
  vendorIds.flatMap((vendorId) =>
    SWEEP_COUNTRIES.map((country) => ({
      name: rateSweepLaneRequested.name,
      data: { runId, vendorId, countryCode: country.code },
    })),
  ),
);

One step, one batch. 104 separate sends would be 104 HTTP round trips and 104 chances for a partial fan out where half the matrix never gets asked. As a single durable step it either happens or it retries.

Concurrency keyed on vendor, and what it does not do

concurrency: [{ limit: 1, key: "event.data.vendorId" }]

I wrote a comment above this saying "one lane per vendor at a time", and that comment was wrong for weeks in a way that mattered.

What it actually means is exactly one step per vendor executes at a time. A run that is sleeping is not executing, so it gives up the slot. All 26 of a vendor's lanes start within a few minutes of each other and then interleave through that single slot, advancing roughly in lockstep.

Which means the pacing sleep I put between cells paces one lane, not the vendor. The other 25 lanes are free to use the slot while this one sleeps. So the real per vendor call rate is one call per (vendor latency + platform step overhead), and my configured pacing numbers turned out to be a ceiling that has never been reached.

We got lucky. The rates I measured afterwards happened to sit under every limit. If a vendor ever needs a genuinely enforced ceiling, that needs a throttle, not a sleep, and I know that now because I measured rather than trusted my own comment.

The 429 backoff is structural, not a number

There is no decay factor anywhere in this codebase. When a vendor rate limits us, the cell throws Inngest's RetryAfterError carrying the vendor's own Retry-After value, so the step waits exactly as long as they asked.

The nice part is what falls out of that. Because that run holds the vendor's only concurrency slot while it waits, every other lane for that vendor waits too. One 429 pauses the entire vendor for exactly the interval the vendor named, using machinery that survives a deploy and a crash. A hand rolled pace decay would have had to recompute identically on every replay to stay deterministic, and it would still only have slowed the one lane that happened to see the 429.

Sleeps are steps, not setTimeout

step.sleep suspends the run entirely and resumes it later. A setTimeout would hold a serverless handler open doing nothing and burn billed compute for it. It is also the difference between a job that survives a deploy mid run and one that silently stops halfway through the matrix, which on a five day cadence means nobody notices for five days.

An atomic counter, not polling

Something has to notice when all 104 lanes are done. Polling a count(*) over a table this size, repeatedly, is the version of this that does not scale.

Instead the run row carries lanesCompleted, and every lane atomically increments it as its last act and reads back the post increment value. Exactly one lane can ever see itself as the last one, and that lane fires the finalise event. Firing an event rather than finalising inline keeps the closing work out of a run whose concurrency slot the next lane is already waiting on.

Finalise is its own function with idempotency

Two different things can decide a run is over: the last lane, and the planner's backstop when the counter never gets there. Both are expected. Putting the closing logic in one function that both send an event to means there is one implementation of "was this run any good" instead of two that drift apart.

idempotency: "event.data.runId" makes the duplicate harmless at the platform level, and the notification carries its own dedupe key so it is harmless at the inbox level too. The failure I was guarding against is an ops team getting two alerts for one problem and learning to ignore both.

A healthy run says nothing

This runs every five days forever. A job that reports its own success on a schedule trains everyone to filter it, so that when it finally does have something to say, nobody reads it. Only a degraded or failed run sends anything.

Then I shipped it, and it lied to me

The first version ran. It produced thousands of rows. The dashboard said COMPLETED. I was pleased with myself for about a week.

Then I actually looked at the data, and the United States was empty.

Not "the United States had bad prices". The single busiest lane in the matrix, for the vendor with the best coverage, had no rows at all. Neither did Germany. Neither did the UK. The countries with rows were mostly the small ones.

Bug one: the Prisma transaction timeout

Here is the write path in the first version, roughly. For one cell: open a transaction, upsert the call row, then for each product the vendor returned, insert the snapshot row and then its charge rows.

Prisma defaults an interactive transaction to timeout: 5000 and maxWait: 2000. Those defaults are sized for an application server sitting in the same rack as its database. Mine is not. Serverless functions talking to Neon over the open internet, where a transaction's cost is its number of round trips multiplied by that latency, not the work the database actually does.

Count the round trips. One upsert, then three per product. At eight products, which is what a well served lane like the USA returns, that is 27 sequential round trips inside one transaction. 27 round trips fit inside 5,000ms only while latency stays under about 185ms. Above that the transaction expires, the whole cell rolls back, and the lane dies having recorded nothing.

Read that back and the shape of the bug is obvious: the better a vendor's coverage on a lane, the more products it returns, the more round trips the write costs, the more likely the write fails. The system was systematically dropping exactly the lanes it was built to answer questions about. Brazil, where one vendor offers a single service, wrote fine every time.

This is the bug that taught me the most, and not because transaction timeouts are hard. Because the failure was correlated with value and there was nothing in the output that said so. A missing row and a row that was never asked for looked identical.

The fix was two things. First, the write path got rewritten so almost none of it is inside the transaction. Carrier classification, canonical charge naming and decimal coercion are pure CPU work with no database in them, and the old loop was holding a transaction open across all of it. Now they happen before BEGIN. Snapshot IDs are generated with randomUUID() up front rather than read back from the insert, so the child charge rows already know their parent's ID and everything collapses into createMany. The write is now a constant six round trips whatever the vendor returns, whether that is one product or twelve.

Second, I raised the client wide defaults anyway, to timeout: 20_000 and maxWait: 10_000. The sweep no longer needs a long budget, but when I went looking, every other multi statement transaction in the app was one latency spike away from the same failure, and none of them had noticed either. Raising a timeout cannot break a transaction that was already committing. It only stops one that was working from being cut off part way.

Bug two: one bad cell was killing twenty nine good ones

retries: 3 is per step. What I had not internalised is what happens when a step exhausts them: it throws into the function body. And a body that does not catch it ends the run, with every remaining slab unasked and unrecorded.

So one failed write at 0.25kg, the very first slab, killed the entire lane. That is how a country ends up with no rates at all rather than 29 out of 30.

The rule I wrote across the top of that file and then implemented is: a bad cell costs one cell. Every cell is wrapped, and a cell that gives up is recorded as a gap and stepped over.

The catch is deliberately narrow, and this bit took me a while to get right:

if (!isFailedStep(error)) throw error;

Only a step reporting its own exhausted retries gets swallowed. A step that has not run yet does not reject at all, since the SDK suspends the run on an unsettled promise, so nothing here can interfere with normal control flow. But anything that is not this step reporting its own failure is a bug in my code or in the SDK, and it must be allowed to fail the lane loudly rather than get quietly filed away as "the vendor was flaky".

isFailedStep checks instanceof StepError and then falls back to checking the error's name, because the SDK duck types its own error classes the same way, and a bundler ending up with two copies of the package would make the instance check silently false. Which here would mean failing a whole lane over one bad cell, the exact thing the code exists to prevent.

Bug three: the backstop that ate a third of the matrix

The planner does not exit after fanning out. It sleeps past the point the sweep should have finished, then force closes anything still marked RUNNING.

That backstop exists because a run stuck in RUNNING is not cosmetic. The next cron refuses to start while one is in flight, so a single stranded lane would quietly stop the sweep from ever running again, and you would find out five or ten or twenty days later.

I sized the sleep off the configured pacing. Slabs times countries divided by calls per minute, plus a margin. On the 20th of August it fired at 158 minutes, one minute after the estimate ran out, while Vendor B was still legitimately grinding through its lanes. The run got closed, marked COMPLETED, and the rest of the matrix was never collected.

The mistake was using an idealised number where a measured one was available. The pacing figure was never the binding constraint, as I found out later when I understood what the concurrency key was really doing. So now the estimate comes from measured seconds per cell, taken off real run data:

Vendor D    4.0 s/cell   (1.1 s vendor latency)
Vendor C    4.8 s/cell   (0.6 s)
Vendor A    8.1 s/cell   (0.7 s)
Vendor B   11.3 s/cell   (5.0 s, it is simply a slow API)

Rounded up, with a 90 minute margin on top, and derived from a function so that adding a country or a vendor extends the backstop automatically instead of quietly making it too short again.

The asymmetry is the lesson. A backstop that fires an hour late costs nothing at all. One that fires ten minutes early costs a third of the matrix.

Bug four: the health check that could not see absence

This is my favourite one, because the code was correct and the arithmetic was correct and the answer was still nonsense.

Finalise judges each vendor: attempted, of which ok, of which failed. Ratio over a threshold means degraded, means alert.

Every one of those numbers is derived from rows that exist. A lane that died before writing anything contributes nothing to the numerator and nothing to the denominator. It is arithmetically invisible. So a vendor that recorded 346 of its 780 cells scored a 0% failure rate, because the 434 cells it never wrote were never counted as anything at all.

The run that lost a fifth of the matrix reported itself perfectly healthy. It was not lying, exactly. It was answering a question about the rows it had, and I had asked the wrong question.

The denominator is now what the run set out to record, taken from the run's own stored plan, and cells with no row are counted as missing. Absence is the loudest signal this system has and it finally has somewhere to be counted.

I added a second alarm alongside the ratio, because a ratio on its own is still not enough. Losing every USA cell for one vendor is 30 cells out of 780, under 4%, nowhere near any sane threshold. It is also the single worst thing that can happen to this dataset, a whole country vanishing from a vendor's coverage with nothing in the table to say so. So an empty lane is a degradation on its own terms, whatever the ratio says.

All of that arithmetic lives in a pure module with no Prisma and no network, because the same judgement is needed in two places that must never disagree: the finalise function deciding whether to wake someone up, and the admin screen showing a person the same numbers. Those had already drifted once when they were two implementations.

Absence has to mean exactly one thing

Running through all four of those bugs is one idea, and it is the thing I would keep if I had to throw the rest of this post away.

The tempting version of the write path skips the row when there is nothing to store. No quote, no row. It produces a table where "Vendor B has no row for 40kg to Brazil" could mean Vendor B does not fly that lane, or Vendor B was down that night, or the sweep never got that far. Those need three completely different responses and the data cannot tell them apart.

So every attempt lands as a row with a status on it: OK, NO_SERVICE, AUTH_ERROR, RATE_LIMITED, TIMEOUT, VENDOR_ERROR, SKIPPED. Failure rows are written before the code throws to trigger a retry, because if the retries are exhausted that row is the only evidence left of what happened, and a row that only appears on success is a row that goes missing exactly when it matters.

Cells that never got asked at all get filled in too. When a vendor returns a 401 the whole vendor is abandoned for that run, and the remaining slabs are written as SKIPPED in one batched insert with skipDuplicates, so the gap in the table carries its own explanation. That used to be one function call, one transaction and one durable step per cell, which meant abandoning a vendor on its first slab cost 29 more round trips to say nothing happened.

Absence in that table now means one thing and one thing only: never attempted.

The part where FedEx is three different companies

The sweep stores every service a vendor returns, not the cheapest one. One run produced 9,123 rate rows from 2,310 calls, because Vendor A returns about 6.7 services per call, Vendor B 5.5, Vendor C 4.1, and our direct account exactly 1.

What was missing at first was normalisation. productName is free text and three vendors spell the same carrier three ways:

"FEDEX DEL"                Vendor B
"Fedex"                    Vendor C
"Fedex DL - with Pickup"   Vendor A

Group on the raw label and those are three carriers. Which makes the single question this whole system exists to answer, "whose FedEx is cheapest to the US at 5kg", literally inexpressible in SQL.

So every snapshot row also carries a normalised carrier, derived from the label at write time by an ordered rule list where first match wins. Real carriers are matched before reseller brands, so a label like "Premium UPS Ground" from a reseller files under UPS and not under the reseller. One rule is checked before all the others, because that carrier is also one of our vendors and its name shows up in both roles. Anything that matches nothing goes to OTHER and surfaces on a review list on the admin screen, never to a vendor brand fallback, because a wrong answer that looks confident is worse than an obvious gap.

carrier is a String and not an enum on purpose. A new carrier appearing in a vendor's catalogue should be a code rule plus a backfill, not a schema migration, because otherwise the sweep has to file it under OTHER until someone ships one.

The same pass extracts three more things off the label, and these matter as much as the carrier:

dutyMode, because a duty paid rate and a duty unpaid rate are not two prices for the same thing. The cheaper one leaves a customs bill for the customer on delivery. Rank them together and you produce a quote that is wrong in a way nobody notices until the shipment lands.
contentType, documents versus goods, priced completely differently.
pickupIncluded, nullable, because null means the label did not mention pickup. Recording that as false would be inventing a fact about what the price covers.

Every comparison query filters on isComparable and dutyMode. isComparable is set at write time and is just currency === "INR", because one vendor sometimes quotes in something else and a cheapest-of query putting 400 USD against 4,000 INR would be confidently wrong rather than merely unhelpful. Nothing in this system invents an exchange rate.

One correction I had to make on myself, since I got it wrong out loud first: reading the first run I concluded our direct carrier account was uniformly beaten by the resellers. It is not. It wins 210 of the 556 cells where the same carrier is available both ways. It is catastrophically bad on Kuwait specifically, over 300% dearer than the same carrier bought through a reseller, and I had generalised from the Gulf rows I happened to look at first.

Five days to eight seconds

With the matrix in Postgres, the rate card became a formatting problem.

There is now a form: pick countries, pick weights, pick an audience, pick a markup. It queries the stored matrix, groups by carrier, and builds a formatted xlsx with exceljs. One sheet per carrier, logos, frozen panes, a terms sheet. It takes about eight seconds.

The job that used to take four to five days of typing takes eight seconds, and it covers 3,120 combinations instead of the fifty someone had the patience for.

A few decisions in that generator that were not obvious:

exceljs, not the xlsx package already installed. SheetJS community edition cannot do fills, fonts, borders or images, which is the entire requirement for a document that goes to a customer.

Audience is the highest stakes field in the codebase. CUSTOMER means marked up prices with vendor names masked behind our own branding. INTERNAL means raw carrier cost, vendors named, stamped on every sheet, filename ending in -INTERNAL. An internal file is our buying price and therefore our margin, in a spreadsheet, with a customer's name on the cover if you get one field wrong. The zod schema refuses to produce a customer file at 0% markup for exactly that reason, and only customer files are ever uploaded to storage, guarded inside the storage module rather than at the call site because call sites get copied.

Logos are baked in as base64 rather than read from /public at runtime, because /public is not guaranteed to exist on a serverless filesystem. And each image is registered once per workbook behind a WeakMap, because calling addImage per sheet tripled a file to 380KB.

Column A holds weights and nothing else. Excel clips a floating image at a frozen pane boundary, so a logo anchored in the frozen first column came out cut in half. Everything starts at column B now, and there are two tests enforcing it.

The workbook states no transit time at all. That was a business decision that came later. A day count in writing gets read as a promise, and the customs and uplift delays that break it are nobody's to control. The field is not even selected in the query, so the builder has no number it could accidentally print.

What it actually changed

The honest accounting:

A rate card went from four to five days to under ten seconds, and from a partial grid to a complete one.
Questions we could not previously ask are now a SQL query. Whose FedEx is cheapest to Germany at 2kg. Which vendor's fuel surcharge moved this quarter. Whether our direct carrier contract is worth keeping, which turned out to have a much more interesting answer than I expected.
The data is explicitly indicative. Quotation sheets yes, booking prices never. Bookings still price live against the vendor, because a stored number is a number from up to five nights ago and Vendor A's duty figure is only valid near the declared value we pinned. A rate card that is roughly right is useful. A charged price that is roughly right is a refund.
Rows older than seven days are refused for quotation, because the cadence is five days and anything past seven means a sweep was missed and nobody noticed.
Successful raw vendor responses are pruned after 30 days. Failures keep theirs forever, since a failure is the row somebody opens months later asking what actually came back. Without the pruning the raw column alone was roughly a gigabyte a year.

What I would tell myself before starting

Three things.

A background job's real interface is its failure output. I spent my design effort on the happy path, on pacing and concurrency and the shape of the fan out, and all four of the bugs that hurt were in what the system said about itself when things went wrong. The status enum on the calls table, the expected-versus-attempted denominator, the gap rows: all of that is unglamorous bookkeeping and all of it is what makes the data trustworthy.

Measure before you size anything. The backstop was sized off a number I derived on paper, and the number was wrong by more than an hour. The concurrency comment I wrote confidently was wrong about what the concurrency key does. Both were fixed the same way, by reading run data instead of my own reasoning.

Correlated failure is the dangerous kind. Random failures are visible. The Prisma timeout failed hardest on the highest value lanes, and the health check could not see the failures at all, so two bugs stacked into a system that was quietly wrong in exactly the places I most needed it to be right. The 5% of runs that fail randomly will get noticed. The 5% that fail for a reason that correlates with importance will not.

The sweep has run cleanly for a while now. It goes off at 01:00 IST every fifth day, takes two and a bit hours, says nothing when it works, and I mostly forget it exists until someone asks for a rate card and gets it before they have finished asking.

I'm Building a Unified Freight Platform at My Part-Time Job and the Architecture Is Genuinely Interesting

Sidharth Sangelia — Fri, 22 May 2026 08:50:07 +0000

Let me tell you about what I have been building at my part time job, because the architecture problem at the center of it is one of the more interesting things I have worked on.

I am a Pricing and Tech Associate at an international freight forwarding company. Part time. And in that capacity I am essentially responsible for the entire tech layer of the business.

Quick Context: What Has Already Shipped

Before I get into the big thing, the smaller deliverables first.

I built the founder's personal portfolio website with a full blog system backed by Sanity CMS. He can add posts and categories without touching a single line of code. That is live and actively being used.

Multiple pitch and company profile decks. The XGBoost price predictor I wrote about a few weeks back, meant to speed up freight quoting which currently happens manually. And some LinkedIn content strategy and data analytics work which I will get to.

But the project I am most focused on right now is different in scale.

The Platform

A customer facing dashboard. Clients log in, add a booking, compare rates across multiple vendors, track shipments, and manage a wallet.

The CRUD layer and the auth and the wallet are all standard enough. Solvable problems with clear paths.

The genuinely hard part is the vendor integration layer.

The Actual Problem

We work with multiple vendors. Some specialise in specific trade lanes. Some are channel partners of UPS or FedEx, meaning they can offer better rates on certain routes.

Each vendor has their own API. Their own request structure for rates. Their own response shape for booking confirmations. Their own webhook format for tracking updates. Their own error codes and edge cases.

And I need to build one unified platform on top of all of it.

From the customer side: one clean interface. They enter shipment details, see a comparison, book, track. Clean and simple.

From behind the scenes: potentially five different APIs with five completely different structures being called, normalised, and returned in a consistent shape.

This is not a toy project. Real bookings, real money, real shipments. The stakes are not tutorial level.

Why the Adapter Pattern

I have been thinking through the architecture for a while and I keep landing in the same place.

A typed master interface in TypeScript. One canonical shape for everything.

interface FreightVendorAdapter {
  getRates(input: CanonicalRateRequest): Promise<CanonicalRateResponse[]>
  createBooking(input: CanonicalBookingRequest): Promise<CanonicalBookingConfirmation>
  getTracking(trackingNumber: string): Promise<CanonicalTrackingEvent[]>
}

Each vendor gets its own adapter class that implements this interface. The adapter does two things: translates canonical input into whatever that vendor's API actually expects, and translates whatever comes back into the canonical response shape.

From the platform's perspective, every vendor behaves identically. You call getRates and you get back CanonicalRateResponse[]. What the adapter does internally is fully encapsulated.

The TypeScript interface is not optional decoration here. It is the contract that holds the whole system together. If an adapter does not satisfy the interface, it does not compile. That is a hard guarantee, not a convention.

Adding a new vendor later becomes a well defined task: implement the interface, write the translation logic, plug it in. The rest of the platform does not change.

How I Am Thinking About the Rollout

Not building everything at once. That is how you build nothing.

Phase one: tracking. Bookings are still made manually on vendor platforms for now. But if I can surface tracking through our own interface, clients see our UI every time they check their shipment. Builds trust. Makes the platform feel real before it is fully real. Also the most self contained piece, I can implement the adapter, wire up one or two vendors, and have something useful without needing the full booking system done.

Phase two: rate comparison. Internal only first. The team uses it to quote faster. Once there is real confidence in accuracy and reliability, it opens to clients. You do not show clients something you have not stress tested yourself.

Phase three: booking and wallet. Later.

The Things I Did Not Expect to Be Doing

I have been doing data analytics in Excel.

I say this with full awareness of what that sounds like coming from a developer. But here is what actually happened.

We exported the founder's LinkedIn connection data. Thousands of connections. The goal is to segment them for a targeted email campaign. Different message for a logistics professional versus a startup founder versus a procurement manager.

I learned VLOOKUP on the spot. Built filtering logic to create clean segments. And now I am looking into data enrichment tools because the raw LinkedIn export is shallow and a good email campaign needs more context than just name, company, and title.

I also pushed three LinkedIn posts as part of the founder's personal content strategy. LinkedIn is its own craft. The feed is brutal and if the first line does not stop the scroll nothing else lands. I am learning this by doing it.

Being Inside an Industry vs Building For One

This is the part that is hardest to convey but I think matters most.

There is a real difference between building something because you theorised that a problem might exist, and building something because you watched the problem happen in front of you on a spreadsheet it should not be running on.

I have been going to meetings. Visiting shipment hubs. Sitting in rooms where the operational friction is not hypothetical.

When you are properly calibrated to a real context the engineering instincts feel different. You are not guessing at what matters. You can see it.

Where Things Are

Exams from 26th May through mid June, so there is a pause coming. But the foundation is solid and the thinking is clear.

By the end of June the tracking feature should be live. Rate comparison internals after that.

I will write about the adapter implementation as it comes together. There are some genuinely interesting problems in the data normalisation layer, especially around how different vendors model the same concepts (a trade lane, a weight break, an accessorial charge) in completely different ways.

New Laptop, New Job, and Two Weeks of Not Touching My Invoice App

Sidharth Sangelia — Wed, 13 May 2026 13:05:15 +0000

So it has been roughly two weeks since my last post and a lot happened. Not in a "shipped three features" way. More in a "life moved fast and I was just trying to keep up" kind of way.

Let me just get into it.

I Got a MacBook Air M5

This one still feels a bit surreal to write.

I did not ask for it. My parents saw me at a point where they felt confident enough to say okay, this is a good investment. And they just got it. The MacBook Air M5.

I have been building on a setup that was honestly not great for a while now. So when I opened the box I genuinely did not know how to react. It felt less like buying a laptop and more like someone placing a bet on you. That kind of thing does something to you. Makes you not want to waste it.

I Took a Part Time Job

This is the bigger news.

I joined an international freight forwarding company as a Pricing and Tech Associate. Part time. And before you ask, yes I know that is a strange combination of words for a job title. That is sort of the point.

I want to be clear about why I chose this over an internship. An internship is where you learn. A job, even a part time one, is where you implement. There is a real difference. In an internship someone is usually holding your hand through the process. Here there are live business problems and someone actually expects me to solve them. That felt more valuable to me right now.

I Have Been Going to Shipment Hubs

My boss has been taking me around to understand the industry from the ground up. I have been to the Aramex hub. I am going to the FedEx hub soon.

I know this sounds completely disconnected from everything I usually write about. But hear me out.

I have been learning things like port to port delivery vs door to door. How a shipment actually gets processed. The steps involved. The players. The friction points. And I think this is genuinely important because we as software engineers can only build things that actually help people when we understand the problem. Not the surface version of the problem. The real version. The one you only see when you are inside it.

The best micro SaaS ideas do not come from browsing Product Hunt. They come from being inside an industry and noticing the thing that makes no sense or takes way too long or still runs on a spreadsheet when it should not.

I have only been here a few weeks and I am already noticing things.

My Job Title Means I Do Everything

This is either a feature or a bug depending on how you look at it.

I am building a website with a CMS. I am doing LinkedIn content strategy for the business. I am thinking through growth strategy. And I have been tasked with building a price predictor system.

That last one is the most technically interesting.

Right now the biggest friction point for the business is response time. A client asks for a quote. The team has to manually figure out the best price across different carriers, routes, and conditions. It takes time. And in freight forwarding, slow response time costs you deals.

So my job is to build something that gives the best price as fast as possible.

My first instinct was linear regression. Made sense on paper. I have some data, I want to predict a number, sounds like a regression problem. But after digging into it I quickly realized that was not going to work well here. Freight pricing has too many variables, too many non-linear relationships, too much going on for a simple linear model to capture it properly.

After going down a proper rabbit hole with AI and other resources I landed on XGBoost. It handles mixed data types well, it is strong on tabular data, and it does not need a massive dataset to start giving reasonable outputs. I had never worked with ML before this job. Now I am building an ML model. That is a strange sentence to write but here we are.

I Have Also Been Building N8N Automations

Part of simplifying internal processes means automating the repetitive stuff. I have been setting up workflows in n8n to handle things that were previously done manually. Nothing I can go into detail on right now but it is good practice and it is directly solving real problems for real people inside a real business. That hits differently than building a personal project.

What About Invoicepedia

Honestly? I have not touched it in two weeks.

And I am not going to pretend that is fine or spin it into some lesson about rest. It just happened. Life got full and the invoice app sat there.

But I am not worried about it. The foundation is solid, I know what I want to build next, and I will get back to it.

If anything, this job is making me think harder about what Invoicepedia should eventually become. Because now I am watching a real business deal with real operational friction every day. That is going to make the product thinking sharper when I come back to it.

The Honest Bit

I know the conventional advice is to go deep on one thing. Pick a lane. Build expertise in a vertical.

And I get that. I genuinely do.

But I am also at an age where I think variety is the better bet. Not scattered, unfocused variety. Intentional variety. I am doing case competitions. I am building software. I am inside a logistics company learning how a real industry operates. Each of those things is teaching me something the others cannot.

I have been competing in case competitions for a while now and there is always this gap between the recommendations you put in a slide deck and what actually happens when you try to execute them. This job is closing that gap. There are actual stakes. Actual outcomes. A client either gets their quote fast or they do not.

That is worth something even if it does not fit neatly on a resume.

My boss has worked at Aramex, UPS, FedEx, Jeena, DG Group. The kind of experience that takes decades to build. I am sitting next to that and asking questions. I would be an idiot not to.

If you are a freight or logistics person and you have thoughts on XGBoost for pricing models I would genuinely love to hear them. This is new territory for me.

And if you want the longer, more personal version of what this past two weeks has felt like, I wrote about it on my personal blog.

Will keep posting. Back to building soon.

Auth Flows, Onboarding Traps, and Why AI Still Needs a Human to Think

Sidharth Sangelia — Thu, 30 Apr 2026 16:21:42 +0000

So this week was one of those weeks where you go in thinking you'll fix one small thing and come out three days later having redesigned half the app's architecture.

Worth it though. Here's what actually happened.

The Problem I Was Ignoring

My invoice PDF templates were broken. Not broken like crashing, broken like useless. The generated PDFs had placeholder company info, no real sender details, nothing a freelancer could actually send to a client. Just four templates sitting there looking pretty and being completely non-functional.

The fix was obvious: capture the user's company details somewhere and use them in the PDF.

But that opened a whole other question: where in the flow do you capture this? You can't just shove a form in front of someone before they've even seen the product. That's the fastest way to kill your conversion rate.

So I decided to redesign the auth and onboarding flow entirely.

The New Flow

Old flow: sign up → land on dashboard → stare at an empty page.

New flow: create your invoice first, auth later.

The user hits the app, clicks "Create Invoice," fills in the form, sees a live preview of their PDF, all without signing up. When they hit submit, that's when we ask for auth. And immediately after auth, if they haven't gone through onboarding, we collect their company details before anything else.

It sounds simple but it required rethinking quite a bit of how the middleware works.

The logic being: let people see value before asking them to commit. Once they've built something they actually want to send, they're invested. That's the moment to ask for an account. Not before.

The Onboarding Status Problem (and Why the AI Kept Getting It Wrong)

This is the part that actually frustrated me the most this week, and I want to be honest about it because I think it's important.

Once you have an onboarding flow, you need to check: has this user gone through it or not? Seems simple. It isn't.

Option 1: Hit the database every time. Check for a row. If it's there, they're onboarded.

This is the obvious answer. It's also terrible. You'd be doing an unnecessary DB read on every single protected route, every middleware call, every page load. That's a lot of wasted reads for information that barely changes.

Option 2: Use Clerk's user metadata. After onboarding, write onboarded: true to the user's public metadata. From then on, you read it from the JWT, no DB call needed.

This is the right answer. But it's harder to implement correctly, and the JWT update isn't instant, which creates a brief window where the token still shows the old state.

Here's the frustrating part: I was using AI to help me through the bugs I hit with option 2. And multiple times, when I got stuck and asked "can we try a simpler approach?", the AI would just... give me database middleware code. Every time. Just casually suggest checking the DB on every request like it wasn't a problem.

It doesn't know your scale concerns. It doesn't know you're trying to keep reads low. It optimises for "working code" not "well-architected code." And if you don't push back, you end up with the wrong solution wrapped in a green checkmark.

This is something I keep coming back to: AI is a powerful tool but it needs someone who knows what they're trying to build. If you don't have the mental model of what good looks like, the AI will lead you somewhere technically correct and architecturally wrong.

Bug Fixes That Were Quietly Breaking Everything

While I was in there, I caught a few bugs that had been hiding:

user was undefined in ModernInvoice.tsx because InvoicePDFButton wasn't receiving or passing user down to InvoicePDFDialog. Classic prop drilling miss.
There was a hardcoded "Your Company" string that was a leftover from early dev. It had been sitting there, overriding real data, the whole time.
InvoiceData type in InvoicePDFButton was missing customerId, causing a mismatch with InvoiceForPDF. Fixed by dropping the local type and just reusing InvoiceForPDF directly.

None of these were glamorous. But fixing them made the PDF output actually correct for the first time.

A Shift in How I'm Thinking About This

Somewhere in the middle of all this, I stopped thinking like a developer and started thinking like a product person.

I've been asking myself a lot of "should this even be here?" questions. What does the user actually need to see first? What am I making them do that they shouldn't have to? What am I hiding that they need?

The create invoice flow is a good example. I spent a while thinking about whether to use a stepper (step 1 → step 2 → step 3) or a single split-panel view with the form on the left and a live PDF preview on the right.

Steppers feel clean but they're wrong for this. Invoice fields are deeply interdependent: change the line items and the total changes, change the currency and the format changes, change the due date and the payment terms might need to update. A stepper forces you to pretend those dependencies don't exist. A split panel lets you see everything at once and makes the relationships obvious.

I'm also thinking more carefully about what to reveal and when. The template picker, for example, should be a modal that opens when you click "Create Invoice", not a separate page, not a step in a form. Each template card should have a "Preview sample" link that opens a fullscreen overlay. Once you pick one, you land on the builder with that template already applied.

Small decisions. But they compound into an experience that either feels thoughtful or feels like someone just strung features together.

What's Next

The onboarding flow captures company name, address, and basic details for now. No logo yet, because that would mean setting up S3 or Uploadthing and that's a whole project in itself. It's on the list but not the priority.

Next up is getting the create invoice flow to its final form. The split-panel builder with live preview, a template switcher pill in the header so you can swap templates without losing your form data, and the success animation: button morphs to a checkmark, then confetti, then redirect to the invoice detail page.

Also planning to add auto-save to localStorage every 30 seconds as a safety net. Nothing worse than filling in an invoice, your browser hiccups, and you lose everything.

I'll keep posting updates as things come together. If you've been following along, thank you genuinely, it helps more than you'd think. This the detailed post on my personal blog

And if you've hit the "AI keeps suggesting the wrong architecture" wall before, I'd love to hear how you dealt with it. Drop it in the comments.

I Did Market Research on Invoice Apps, and Now I Have More Questions Than Answers

Sidharth Sangelia — Fri, 24 Apr 2026 19:10:58 +0000

So if you read my last post, you know I'm rebuilding Invoicepedia. My old broken invoice app that had both Drizzle and Prisma installed at the same time and a delete button that sometimes just... crashed everything.

I decided before writing a single line of new code that I should actually find out if there is a real gap here. Not a "I think there is a gap" gap. An actual, people-are-complaining-about-this gap.

So I spent a few days going down Reddit rabbit holes. r/QuickBooks, r/waveapps, r/Bookkeeping, r/smallbusiness. Reading actual user complaints. Not the landing page copy. The raw, frustrated, typing-in-caps kind of posts that people write at 11pm when their invoice software just blocked their account.

Here is what I found.

The Complaints Are All the Same Underneath

QuickBooks: people are drowning in features they never use and paying $30+ a month for the privilege. The word that kept coming up was "overwhelmed." One person called it unnecessary complexity for someone who just wants to send an invoice and get paid.

FreshBooks: the horror stories here were something else. Accounts blocked days after payment. Payment information trapped inside the platform. One person literally wrote "I advise small businesses to steer clear." That is not a product review. That is a warning.

Wave: used to be free. Slowly started charging for things. The free tier now cannot fully automate recurring invoices. One thread from August 2025 had someone saying "Wave is in serious trouble" and switching to Zoho just to get reliable bank feeds.

Invoice Ninja: this one hurt a little because I actually tried it once. The UI looks like a cockpit. I am not exaggerating. I opened it, felt genuinely confused, and closed the tab. The Reddit threads confirm this. Long-term paying customers upgrading to enterprise and not getting the features they paid for. One person writing "AVOID AT ALL TIMES" in capitals.

Zoho: missing basic integrations. Support tickets going a week without resolution.

The summary is this. Every single major player has users who are actively looking for an exit. Not because they want more features. Because the basics are broken or too expensive or too complicated or all three at once.

The Gap Is Real But Is It Big Enough

Here is the thing that kept nagging at me after all this research.

The gap is obvious. Freelancers need something dead simple. No accounting bloat. No surprise fee hikes. No cockpit UI. Just: create invoice, send invoice, track who paid, get paid.

But the market also shows something a little discouraging. Users are exhausted. They cycle from QuickBooks to Wave to Zoho and back. There is no dominant winner. Which either means the gap is real and unsolved or it means the gap is a graveyard where many products have already tried and quietly died.

I genuinely do not know which one it is yet.

What I do know is that the complaints are consistent enough that building something focused on doing the basics really well is not a crazy idea. It might just be a very slow one.

So Here Is What I Decided

I am not going to wait until I have all the answers. I have been learning that waiting for certainty is just procrastination dressed up as planning.

My partner and I are going to run a hackathon-style sprint on this. Intense. Time-crunched. We will ship buggy code probably. But we will have something working end to end. PDF generation, email sending, a dashboard that actually shows you useful numbers when you open it.

The goal is not to be FreshBooks. The goal is not to be QuickBooks. The goal is to be the thing a freelancer opens, uses in ten minutes, and does not feel like they need a tutorial to understand.

Five real users. That is the metric. Not five hundred. Five.

The Bigger Question I Keep Avoiding

I am going to be honest here because that is the only reason I write these posts.

I am an economics student. Not a CS degree holder. I have been building projects for a while now and I am getting better. But the job market is rough and AI is making everyone question everything. I watched Harnoor Singh's video recently where he talks about how even someone with NCR and Microsoft on their resume is having difficulty finding a role right now.

If someone with that kind of experience is struggling, what does that mean for someone like me with just projects and self-taught skills?

I wrote a longer version of what I am thinking on this on my personal blog if you want to read that side of it: link to portfolio post

But the short version is this. I think the era of DSA grind plus projects to land a FAANG job is shifting. The wave now is building. Micro SaaS. Shipping end to end. And maybe AI has compressed this timeline faster than anyone expected.

Rebuilding Invoicepedia is teaching me more about product thinking than any tutorial ever did. Not just code. Thinking about why something should exist. Who it is for. What it should not do.

That feels worth something even if I cannot put it on a resume yet.

Questions for You

If you are a freelancer, what invoicing tool are you using right now and what is the one thing you wish it did differently? Drop it in the comments.

If you are building something similar or have done the freelancer SaaS space, I would love to know what you ran into. What was harder than expected? What surprised you?

And if you are also a self-taught dev or non-CS-background person navigating this market, I genuinely want to hear how you are thinking about it. I do not think I am the only one asking these questions to myself late at night.

Will keep posting updates as we sprint through this build.

I'm Rebuilding My Old Invoice App and This Time I Actually Know What I'm Doing

Sidharth Sangelia — Sun, 19 Apr 2026 19:12:33 +0000

So I have this old project called Invoicepedia. Built it like 6 months ago when I was just starting to learn full stack. Next.js, Clerk, Neon, basic CRUD. The usual "I just learned how to connect a database" kind of project.

I opened it recently and honestly it was rough. Delete button crashed the app sometimes. You could submit an empty invoice form with no validation. The Stripe integration was half working. I had both Drizzle and Prisma installed at the same time for some reason. Classic.

I almost deleted it and moved on to something new.

Instead I decided to actually fix it and turn it into something real.

Why Not Just Start a New Project

Every time I start something new I spend like 3-4 days just on setup. Auth, database, environment variables, middleware, all that stuff before you even write a single line of actual product code. And by the time I'm done with setup I've already lost interest or I hit some weird error and just abandon the whole thing.

Invoicepedia already had all of that. It was broken but the foundation was there. Auth worked. Database was connected. Routes existed.

So I thought okay, why not just fix this one instead of starting from zero again. Worst case I learn something. Best case I actually ship something.

First Thing I Did Was Stop Adding Features

Old me would have immediately started adding new stuff. Charts. Dark mode. AI features. Whatever sounded cool.

This time I made a list of everything that was broken and fixed that first. No new features until the existing stuff actually worked properly.

Removed dead dependencies. Rewrote the schema. Added proper Zod validation to every server action. Changed hard deletes to soft deletes because you should never hard delete financial records. Fixed the middleware. Added actual error handling.

It took a week and nothing visually changed. The app looked the same. But after that week I wasn't scared to touch the codebase anymore and that felt different.

What I'm Actually Trying to Build

Yeah I know. Another invoice app. There's like 50 of them already.

But here's the thing — most of them are either too expensive or too complicated for a normal freelancer. FreshBooks charges you $19-30 a month and limits your clients on the cheaper plans. Wave was free and then slowly started charging for things that used to be free. Invoice Ninja has so many features that the UI looks like a cockpit, a normal person opens it and immediately closes the tab.

The people I want to use this are freelancers. Developers, designers, writers. People who just did some work, need to send an invoice, and want to get paid without learning accounting software.

So the whole idea is just — do the basic things really well. Don't try to be QuickBooks. Don't try to be FreshBooks. Just be the thing a freelancer needs to send professional invoices and track who's paid them and who hasn't.

What I'm Shipping First

I made a list of like 10 features I want to build eventually. AI invoice generation, automated payment reminders, client portal, recurring invoices, expense tracking. The full thing.

Then I looked at that list and realised if I wait until all of that is done I will never ship anything.

So the first version is going to have three things:

PDF generation — because an invoice app that can't produce a PDF is just a form.

Email sending — so you can actually send the invoice to your client from inside the app instead of downloading it and attaching it to Gmail manually.

A proper dashboard — right now it's just a table. I'm adding three metric cards at the top so when you open the app you see immediately how much you got paid this month, how much is still outstanding, and how many invoices are overdue. That's actually useful information.

That's it. Ship those three. Get some users. See what they actually want next.

The Goal is 5 Users

Not 500. Not 5000. Five real people who use it and tell me what's wrong with it.

I'll post updates as I build. What I'm adding, what's breaking, what people say when they actually try it. I don't have it all figured out, I'm figuring it out as I go.

If you're a freelancer who's annoyed at paying too much for invoicing software, keep an eye on this. If you're a dev learning full stack, hopefully this is useful to see what an actual messy in-progress project looks like compared to the clean tutorial versions.

Stack is Next.js 15, Prisma, NeonDB, Clerk, Tailwind, shadcn/ui.

Happy to answer questions in the comments if anything here was interesting or you want to know more about specific decisions I made.

HTTP Status Codes: A Developer's Real-World Guide🚦

Sidharth Sangelia — Wed, 03 Sep 2025 16:46:18 +0000

Hey fellow developers! 👋

Last week, I spent three hours debugging what I thought was a complex authentication issue, only to discover it was a simple 422 error that I completely misunderstood. That frustrating evening made me realize how many of us know the "famous" status codes (hello, 404!) but struggle with the nuanced ones that could save us hours of debugging.

So here's my attempt to break down HTTP status codes in a way that actually sticks. No dry documentation here – just real scenarios we face every day.

Wait, Why Should I Care About Status Codes?

Before we dive in, let me tell you why this matters. Last month, our API was returning 200 OK for failed operations (yeah, I know...), and our frontend team was going crazy trying to figure out why their error handling wasn't working. The fix? Proper status codes. It's not just about following standards – it's about making your life and your team's life easier.

The Categories That Actually Matter

2xx - The "Everything's Fine" Family ✅

200 OK - The Reliable FriendThis is your bread and butter. GET request for user data? 200. Successful login? 200. But here's the thing – don't use 200 for everything just because it works.

// Good use of 200
GET /api/users/123
Response: 200 OK with user data

// Bad use of 200 
POST /api/users (creating a user)
Response: 200 OK // Should be 201!

201 Created - The OverachieverI used to ignore this one until I realized how much cleaner my API responses became. Use 201 when you've successfully created something new. Your frontend developers will thank you because they can differentiate between "I got existing data" and "I just created something new."

// Perfect 201 usage
POST /api/posts
{
  "title": "My awesome post",
  "content": "Some content here"
}
Response: 201 Created
{
  "id": 456,
  "title": "My awesome post",
  "created_at": "2024-01-15T10:30:00Z"
}

204 No Content - The Strong Silent TypeThis one's beautiful for DELETE operations and updates where you don't need to send data back. Clean, efficient, and tells the client "job done, nothing more to say."

// Elegant 204 usage
DELETE /api/users/123
Response: 204 No Content
// No response body needed - the status says it all

3xx - The "Look Elsewhere" Family 🔄

301 vs 302 - The Redirect SiblingsThese two confused me for years. Here's how I remember them:

301: "This moved permanently, update your bookmarks" (like when you change your domain)
302: "This moved temporarily, keep using the old URL" (like maintenance pages)

Real talk: Most of the time you'll use 301 for SEO-friendly redirects and 302 for temporary stuff.

304 Not Modified - The Bandwidth SaverThis little gem enables caching magic. When implemented correctly, it can dramatically speed up your app. The client asks "has this changed since last time?" and you respond "nope, use your cached version."

4xx - The "You Messed Up" Family ❌

400 Bad Request - The Vague ComplainerThis is the generic "something's wrong with your request" response. But here's a pro tip: always include helpful error messages in the response body.

// Bad 400 response
Response: 400 Bad Request
"Invalid request"

// Good 400 response
Response: 400 Bad Request
{
  "error": "Validation failed",
  "details": {
    "email": "Email format is invalid",
    "password": "Password must be at least 8 characters"
  }
}

401 vs 403 - The Permission PoliceThis distinction trips up so many developers:

401 Unauthorized: "Who are you? Please log in."
403 Forbidden: "I know who you are, but you can't do this."

Think of 401 as a bouncer asking for ID, and 403 as the same bouncer saying "sorry, members only."

404 Not Found - The CelebrityEveryone knows this one, but here's something interesting: sometimes returning 404 is a security feature. Instead of returning 403 for a private resource (which tells attackers it exists), return 404 to keep things secret.

422 Unprocessable Entity - The PerfectionistThis one saved my sanity. Use 422 when the request is well-formed but semantically incorrect. Like when someone submits a perfectly formatted JSON with an email that passes regex validation but the domain doesn't exist.

// 422 vs 400 example
POST /api/users
{
  "email": "user@fakemaindomain.com", // Valid format, fake domain
  "age": -5 // Semantically wrong
}
Response: 422 Unprocessable Entity
{
  "error": "Invalid user data",
  "details": {
    "email": "Domain does not exist",
    "age": "Age cannot be negative"
  }
}

5xx - The "I Messed Up" Family 💥

500 Internal Server Error - The NightmareThe universal "something went wrong on our end." Never let your users see these in production without proper error handling and logging.

502 Bad Gateway - The Middleman ProblemThis usually means your server is trying to talk to another server (database, external API, microservice) and that conversation failed. Super common with microservices architecture.

503 Service Unavailable - The Planned OutagePerfect for maintenance windows. Include a Retry-After header to tell clients when to try again.

The Debugging Stories You'll Relate To

The Case of the Mysterious 502

Last month, our checkout process started throwing 502s randomly. Turns out, our payment service was timing out, but our API gateway was returning 502 instead of 504. The fix? Proper timeout configuration and using 504 (Gateway Timeout) for actual timeout scenarios.

The 200 That Lied

We had a "successful" user creation endpoint that always returned 200 OK, even when the email already existed. The frontend team was showing success messages for failed registrations. The lesson? Don't be afraid to use 409 (Conflict) for duplicate resources.

The Authentication Maze

Our mobile app was stuck in redirect loops because we were using 302 for OAuth redirects. Mobile apps don't handle redirects the same way browsers do. Switch to proper API responses with tokens instead of relying on redirects.

Status Codes in the Real World

REST API Best Practices

// User Management API
GET    /api/users     → 200 (list of users)
POST   /api/users     → 201 (user created)
GET    /api/users/123 → 200 (user found) or 404 (not found)
PUT    /api/users/123 → 200 (updated) or 404 (not found)
DELETE /api/users/123 → 204 (deleted) or 404 (not found)

Error Handling Patterns

// Frontend error handling
fetch('/api/users', {
  method: 'POST',
  body: JSON.stringify(userData)
})
.then(response => {
  if (response.status === 201) {
    showSuccess('User created successfully!');
  } else if (response.status === 422) {
    return response.json().then(errors => {
      showValidationErrors(errors.details);
    });
  } else if (response.status === 409) {
    showError('User already exists');
  }
})
.catch(error => {
  showError('Something went wrong. Please try again.');
});

The Tools That'll Save Your Sanity

Browser DevTools

Your best friend for debugging. The Network tab shows status codes for every request. Pro tip: right-click and "Copy as cURL" to test APIs directly.

Postman/Insomnia

Perfect for API testing. Set up collections with different scenarios and expected status codes.

curl Commands

# Check just the status code
curl -I https://api.example.com/users

# Follow redirects
curl -L https://example.com/old-url

# Show response headers and status
curl -v https://api.example.com/users

The Quirky Ones Worth Knowing

418 I'm a Teapot ☕Started as an April Fools' joke but is now used by some APIs to detect automated requests. Some developers use it for rate limiting with a sense of humor.

451 Unavailable for Legal ReasonsNamed after the book "Fahrenheit 451," used when content is blocked due to legal restrictions. More relevant than ever in our current internet landscape.

Common Pitfalls I've Seen (And Made)

The "Everything is 200" Syndrome

Don't do this. If something failed, don't return 200 with an error message in the body. Your HTTP client libraries are built to handle different status codes – use them!

Ignoring Response Bodies for Errors

A 400 status code without explanation is useless. Always include helpful error messages.

Mixing Authentication and Authorization

401 for authentication failures, 403 for authorization failures. Keep them separate in your head and your code.

Not Monitoring Status Codes

Set up alerts for unusual patterns in your status codes. A sudden spike in 5xx errors might indicate a problem before users start complaining.

Testing Your Status Codes

// Best example for API testing
describe('User API', () => {
  test('should return 201 when creating user', async () => {
    const response = await request(app)
      .post('/api/users')
      .send({ email: 'test@example.com', name: 'Test User' });

    expect(response.status).toBe(201);
    expect(response.body).toHaveProperty('id');
  });

  test('should return 422 for invalid email', async () => {
    const response = await request(app)
      .post('/api/users')
      .send({ email: 'invalid-email', name: 'Test User' });

    expect(response.status).toBe(422);
    expect(response.body.details).toHaveProperty('email');
  });
});

Wrapping Up

Status codes aren't just numbers – they're a communication protocol between your server and clients. Getting them right makes debugging easier, improves user experience, and makes your APIs more predictable.

The next time you're building an API endpoint, take a moment to think: "What's the most appropriate status code for this scenario?" Your future self (and your teammates) will thank you.

What's your most encountered status code? Mine's probably 422 – I'm apparently really good at sending semantically incorrect requests to my own APIs! 😅

Drop a comment below with your status code war stories. We've all been there, and sharing these experiences helps us all become better developers.

Useful Resources:

HTTP Status Codes Registry - The official source
HTTP Status Dogs - Because learning should be fun
MDN HTTP Status Codes - Comprehensive documentation

From Web to Mobile: Completing My React Native Journey with Maximilian Schwarzmüller

Sidharth Sangelia — Sat, 30 Aug 2025 12:04:59 +0000

After weeks of dedication and hands-on coding, I’ve successfully completed Maximilian Schwarzmüller’s comprehensive React Native course on Udemy. As someone with a solid background in web development, this transition to mobile app development has been both challenging and incredibly rewarding.

A Little Backstory

I’ve always wanted to learn app development. Back in Class 12, I ambitiously tried setting up Android Studio, running the simulator, and building a simple app. What followed was an entire month of battling with environment variables, Java configurations, and errors I barely understood.

By the end of that month, I honestly questioned whether Android development was really for me. The complexity of the setup alone felt like a huge barrier to entry.

Looking back now, I can say this: React Native is a fantastic entry point into the mobile ecosystem. It lowers the barrier, lets you leverage existing React skills, and still leaves room to grow into full native development later as the need arises.

The Course Experience

Maximilian’s teaching approach struck the perfect balance between practical implementation and conceptual understanding. Rather than overwhelming students with theory, he focused on building real applications while explaining the “why” behind each decision. This hands-on methodology made complex mobile development concepts much more digestible.

Key Learning Milestones

Understanding the Component Ecosystem

One of the first major shifts was adapting to React Native’s component structure. Gone are the familiar HTML elements like div, p, and a. Instead, I learned to work with React Native’s specific components:

View instead of div
Text for all text content
TouchableOpacity for interactive elements
Image for media display

This wasn’t just a syntax change — it represented a fundamental shift in how mobile interfaces are constructed.

Navigation Patterns

Web development relies heavily on URLs and browser history, but mobile apps follow different patterns. The course covered:

Stack Navigator: For hierarchical navigation where users drill down into content
Tab Navigator: For parallel sections of an app
Drawer Navigator: For side menu functionality

Each navigation pattern serves specific user experience goals that don’t directly translate from web development.

Native Device Integration

Perhaps the most exciting aspect of React Native is its ability to access native device features. The course introduced me to:

Camera functionality
Location services
Device storage
Push notifications (overview)

Even in their simplest form, these integrations showed me the true power of mobile development — creating apps that live closer to users’ daily lives.

Reinforced React.js Skills

An unexpected benefit was how React Native deepened my React.js knowledge. Concepts like:

Component lifecycle management
State management patterns
Props and data flow
Hooks usage

… all became clearer when applied in a mobile context. It felt like practicing the same music scales on a different instrument — the repetition in a new setting gave me fresh perspective.

Challenges and Realizations

The Learning Curve

While my React background helped, mobile development brought unique challenges:

Understanding platform-specific behaviors (iOS vs Android)
Handling different screen sizes and orientations
Adapting to mobile-first interaction patterns
Debugging on physical devices vs simulators

Beyond the Basics

The course laid a strong foundation, but I know this is just the start. Areas I plan to explore further include:

Native module configuration
Advanced state management in mobile contexts
Performance optimization for mobile devices

Looking Ahead

This course gave me the confidence to start building mobile apps, while also showing me how much more there is to learn.

Immediate Next Steps:

Deep dive into push notifications
Exploring advanced native modules
Building more complex apps independently
Understanding app deployment and store submissions

Long-term Goals:

Mastering mobile-specific UX patterns
Platform-specific optimizations
Cross-platform best practices
Mobile analytics and performance monitoring

Advice for Fellow Web Developers

If you’re considering moving from web to mobile development:

Embrace the differences — don’t force web patterns into mobile contexts
Start with the fundamentals — get comfortable with React Native’s components
Test on real devices — simulators are great, but real-world usage is different
Be patient with native features — setup and testing can be tricky, but rewarding

Final Thoughts

Completing this React Native course feels like opening a door into an entirely new dimension of development. Web will always be my foundation, but mobile offers unique opportunities to create apps that integrate seamlessly into people’s daily routines.

The journey from web to mobile isn’t just about syntax or frameworks — it’s about understanding how people interact with technology in a more personal way.

Huge thanks to Maximilian Schwarzmüller for putting together such a structured and practical course. For anyone on the fence about learning React Native, his teaching makes the transition much less intimidating.

This is just the beginning of my mobile dev journey — and I can’t wait to see where it leads. 🚀 I will be sharing more of my learnings on notebook.thesidharth.com soon.

What about you? Have you ever tried transitioning between development platforms? Share your story — I’d love to hear it.

Building My First AI-Powered App: From Whisper to Vercel Limits

Sidharth Sangelia — Thu, 28 Aug 2025 19:17:50 +0000

I thought building AI apps would be about training models and complex machine learning stuff, but as I dug deeper, I found out - why reinvent the wheel when you can solve 70-80% of tasks with existing models?

Turns out, the real challenge wasn't the AI part. It was everything else.

The Idea: What I Wanted to Build

I'd been consuming tons of great video content on YouTube (both long and short form) and Instagram, and kept thinking: "This would make such a good blog post for better readability and SEO reach."

So I came up with what seemed like a simple solution - a NextJS 15 web app where users could:

Upload a video
Get AI-generated transcript using OpenAI's Whisper-1
Transform it into SEO-optimized content using ChatGPT/Gemini
Download a ready-to-publish blog post

How hard could it be, right? OpenAI already has Whisper for transcription, other LLM models are readily available through APIs. Just connect the dots and boom - instant article creation.

Plus, the blogging and content creation industry is booming. I thought if I could build this, maybe I could create some recurring revenue like the folks on Twitter always talk about.

🛠️ The Tech Stack (What I Thought I Needed)

Frontend & Backend: Next.js seemed like the obvious choice - I could handle both frontend and backend with API routes and server actions.

AI Processing: OpenAI's Whisper-1 for transcription, GPT-4/Gemini for content optimization.

Hosting: Vercel, because... well, it's the easiest deployment ever.

File handling: This became a whole saga (more on this below).

Here's what I thought the flow would look like:


// What I thought the flow would look like
const processVideo = async (file) => {
  const transcription = await openai.audio.transcriptions.create({
    file: file,
    model: "whisper-1"
  });

  const blogPost = await openai.chat.completions.create({
    model: "gpt-4",
    messages: [{
      role: "user",
      content: `Turn this into an SEO blog: ${transcription.text}`
    }]
  });

  return blogPost;  
};

Spoiler alert: It wasn't that easy.

⚡ First Reality Check: The 25MB File Upload Nightmare

My initial plan was to use UploadThing (Theo's creation) since it had a generous free tier. But then I discovered Whisper-1 has a 25MB file limit per upload.

Problem: Most decent-quality videos are way larger than 25MB.

In development, I could easily test with small videos under 25MB, but for a production app that people would actually pay for? This was a major roadblock.

Solution attempt #1: Extract audio from video (audio files are much smaller than video files).

I found ImageKit.io, which could handle uploads, compression, and audio extraction. Perfect! I ditched UploadThing and dove into ImageKit's documentation.

Spent hours implementing the audio extraction feature, writing code, testing locally. Everything looked good. Then came the moment of truth - testing the actual audio file extraction.

Nothing worked.

I added console.logs everywhere, thinking it was a code issue. After hours of debugging, I realized the problem: I'd exhausted ImageKit's free plan just from testing.

The kicker: ImageKit's paid plan costs $80/month and provides way less processing power than I'd need for a robust app.

Solution attempt #2: Use FFmpeg on my own server.

New problem: I'm a student without money for servers.

So I decided to postpone the audio extraction feature and just put a strict 25MB limit for now. "I'll figure this out later," I told myself.

💻 The API Integration That Worked... Until It Didn't

I implemented all the API integrations and functions. Everything worked beautifully on localhost. The transcription was accurate, the content optimization was decent, and I was feeling pretty good about myself.

Time to deploy and show my friends!

I deployed to Vercel, shared the link with excitement, and... nothing worked.

The app would start processing, show a loading state, and then just timeout with a 504 error.

After a lot of head-scratching and debugging, I found the culprit: Vercel's function timeout limits.

My processing pipeline was taking 60+ seconds:

File upload: ~10 seconds
Audio processing: ~20 seconds
Whisper transcription: ~30 seconds
GPT optimization: ~15 seconds

Google search time: "vercel function timeout limit"

The harsh reality:

Hobby plan: 10 seconds
Pro plan: 60 seconds
Enterprise: 15 minutes

Quick math: My processing takes 60+ seconds. Even the Pro plan wouldn't save me.

Bigger realization: This isn't just a Vercel problem. AWS Lambda has 15-minute limits, Netlify has similar constraints. Serverless functions aren't meant for long-running tasks.

❌ What I Built:  User Upload → Vercel 
   Function → Process (60s) → Timeout.

✅ What I Actually Needed:  User Upload → Queue 
   Job → Background Worker → Notify User

🤔 Understanding the Real Problem

Here's what I learned about serverless architecture the hard way:

Serverless functions are great for:

Fast responses (under 30 seconds)
Auto-scaling
Quick API endpoints
Simple data processing

Serverless functions are terrible for:

Long-running AI processing
File manipulation tasks
Complex workflows
Anything that takes time

The fundamental issue: I was trying to fit a long-running AI workflow into a request-response architecture. That's like trying to fit a truck through a car door.

🔧 Solution Discovery: Enter Background Jobs

Back to Google: "background jobs nodejs", "async processing for AI apps"

I discovered several options:

Inngest - Developer-friendly, good free tier
BullMQ - Redis-based, more complex setup
AWS SQS - Powerful but overkill for my needs
Redis Queue - DIY approach

I chose Inngest because:

It handles the infrastructure for me
Great developer experience
Built-in retries and error handling
Free tier was sufficient for testing

Here's the new architecture:

import { inngest } from "./inngest/client";

// API endpoint just queues the job (fast!)
app.post('/api/process-video', async (req, res) => {
  const jobId = await inngest.send({
    name: "video/process",
    data: {
      videoUrl: req.body.videoUrl,
      userId: req.body.userId
    }
  });

  res.json({
    message: "Processing started! We'll notify you when it's ready.",
    jobId,
    checkStatusAt: `/api/status/${jobId}`
  });
});

// Actual processing happens in background (no time limits!)
export const processVideo = inngest.createFunction(
  { id: "process-video" },
  { event: "video/process" },
  async ({ event }) => {
    // Now I can take all the time I need
    const transcription = await processWithWhisper(event.data.videoUrl);
    const blog = await optimizeWithGPT(transcription);

    // Notify user when done
    await notifyUser(event.data.userId, blog);
  }
);

The difference: Instead of making users wait for 60+ seconds (and timing out), I immediately return a "we're processing it" response and handle the heavy lifting in the background.

📚 What I Actually Learned

Architecture Matters More Than I Thought

You can't just throw AI processing into a standard web app architecture and expect it to work. AI apps have fundamentally different requirements:

Long processing times
Unpredictable resource usage
Need for progress tracking
Error handling for expensive operations

Research Infrastructure Constraints First

I should have researched deployment limitations before writing a single line of code. Now I know to ask:

What are the timeout limits?
How much memory/CPU can I use?
What happens if processing fails halfway through?
How do I handle user notifications?

The Hardest Part Isn't the AI

I thought the challenging parts would be:

Getting good transcriptions
Optimizing content for SEO
Fine-tuning prompts

Actually challenging parts:

File upload and processing
Background job orchestration
User experience for async operations
Error handling and retries
Infrastructure costs and scaling

Local Development Can Be Misleading

Everything worked perfectly on my MacBook Pro. But production environments have:

Stricter resource limits
Network latency
Timeout constraints
Different error conditions

Lesson: Test in production-like environments early and often.

🔮 What I'm Building Next

I'm currently rebuilding the entire app with background jobs as a first-class citizen:

New tech stack:

Frontend: Still Next.js, but with real-time progress indicators
Background jobs: Inngest for orchestration
Database: Adding Supabase for job status tracking
File storage: Moving to Cloudinary for better video handling
Notifications: WebSocket connections for real-time updates

Timeline reality check: What I thought would be a 2-week project is now a 3-month learning journey. And honestly? I'm more excited about it now.

Other technologies I discovered:

Langchain & Langraph: For more complex AI workflows
Redis: For caching and session management
WebSocket/Server-sent events: For real-time progress updates
Queue monitoring tools: For debugging background jobs

💭 Advice for Other Developers

If you're building AI-powered apps, here's what I wish someone had told me:

Plan for Async from Day One

Don't build a synchronous AI app and try to make it async later. Design your user experience around the fact that AI processing takes time:

Show progress indicators
Send email/push notifications when jobs complete
Let users check status later
Handle failures gracefully

Research Your Platform's Limits

Before you write any code, understand:

Function timeout limits
Memory constraints
File size restrictions
Pricing for overages

Start with Background Jobs

Even if your AI processing is currently fast, it will get slower as you add features. Background jobs give you:

Better user experience
Easier scaling
Retry mechanisms
Progress tracking

Infrastructure Is Harder Than AI

Getting good results from OpenAI's APIs is pretty straightforward. Getting those results reliably delivered to users in production? That's the real challenge.

It's Totally Normal

If you're struggling with infrastructure for AI apps, you're not alone. Every AI developer goes through this learning curve. The AI part is often the easy part.

What about you?

Have you hit similar serverless limitations while building AI apps? What solutions did you find?

Are you currently building something with AI? What infrastructure challenges are you facing?

Drop a comment below - I'd love to hear about your experiences and maybe we can help each other avoid these pitfalls!

Currently rebuilding this app the right way. Follow my journey as I document everything I learn about building production-ready AI applications.