Third-party APIs don't fail loudly — they return less

For several days, my service was quietly broken and every health check said it was fine.

No exceptions. No 5xx. No alerts. The upstream API returned 200 OK the entire time. What it stopped returning was half the payload — and nothing in my stack was built to notice the difference.

This is a short write-up of what happened, how I eventually found it, and the one habit I changed afterwards.

The setup

I build and run a small video downloader. A user pastes a link, the server resolves it, and the browser gets a file. Two paths do that work:

• Fast path — a third-party API returns direct CDN URLs. Resolves in about a second.
• Slow path — yt-dlp extracts the media itself. Takes 20–30 seconds.

The fast path was for one platform, the slow path was the fallback for everything else. That fallback is the important part, because a fallback is exactly the thing that hides a failure.

The client code branched like this:

function renderVideo(data, url, name) {
  if (data.direct) {
    // Fast path: direct CDN links
    dlSd.href = streamUrl(data.direct.sd, name);
    hdHref    = streamUrl(data.direct.hd, name);
  } else {
    // Fallback: stream through yt-dlp
    dlSd.href = downloadUrl(url, 'sd', name);
    hdHref    = downloadUrl(url, 'hd', name);
  }
}

Read that else again. It doesn't ask why data.direct is missing. It just quietly does something slower.

What actually broke

The third-party provider stopped returning media URLs. Not all of it — the response still had a title, a thumbnail, a duration, an uploader. Just no direct object, and no audio field.

So my own API did this:

{
  "ok": true,
  "type": "video",
  "source": "tiktok",
  "title": "how many frogs did you find?",
  "thumbnail": "https://...",
  "durationSec": 21,
  "uploader": "tiktok"
}

"ok": true. Status 200. Valid JSON. Every field that is there is correct.

And every request now took 30 seconds instead of 1, the MP3 button vanished because data.audio was gone, and photo posts broke completely because they had no fallback path at all.

Why nothing caught it

Look at what each layer was actually checking.

The health check pinged /api/health and got a 200. The process was up. That was all it ever claimed to test.

The API handler checked that the upstream request succeeded. It did.

The client checked res.ok && data && data.ok. All true.

const data = await res.json().catch(() => null);
if (!res.ok || !data || !data.ok) {
  throw new Error(data?.error || 'Could not process this video.');
}

That's a reasonable-looking guard. It catches network failures, gateway timeouts, and error responses. It does not catch a successful response that lost half its content, because no layer had ever written down which fields the caller actually needs.

And the user-visible symptom was terrible: click download, wait 30 seconds, see nothing. There was a 50-second client-side abort, so on a cold start it would sometimes give up entirely and show "the server is slow, try again later" — which sent me looking at my own infrastructure for days.

Finding it

What finally worked was boring: I stopped looking at my server and compared the shape of the response against what the client actually consumed.

const r = await fetch('/api/info', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ url: SAMPLE_URL })
});
const d = await r.json();
console.log(Object.keys(d));
// ['ok','type','source','title','thumbnail','durationSec','uploader']

direct wasn't in the list. audio wasn't either. The client needed both.

Then I called the upstream provider directly and got a 200 with a zero-length body. That was the whole bug. Their service had started refusing my server — most likely by IP, since requests from a datacenter address got the empty response while the same request from elsewhere worked.

I couldn't fix their side. But I could stop pretending it was working.

The fix

The change wasn't clever. It was writing down what "success" actually means.

Before, the client had one notion of success: the request worked.

if (!res.ok || !data || !data.ok) {
  throw new Error('Could not process this video.');
}

After, there are two distinct outcomes — worked and worked completely:

// Not an error. But not the fast path either.
slowMode = data.source === 'tiktok' && (data.fastPath === false || !data.direct);

And on the server, the upstream client now returns a verdict rather than whatever it happened to receive:

// Only a success if it carries the media URLs the caller needs.
const sd = isHttpUrl(d.play)   ? d.play   : null;
const hd = isHttpUrl(d.hdplay) ? d.hdplay : sd;

if (!sd) {
  return {
    fastPath: false,
    reason: 'no_media_urls',
    detail: 'upstream returned metadata but no playable URL',
  };
}

Three things came out of that:

1. Degradation is now a value, not an absence. The API sends fastPath: false instead of silently omitting a field. The difference matters — a missing field is ambiguous, an explicit false is a statement.

2. The user is told. Thirty seconds of silence makes people click again, and every extra click spawns another extraction process on a small instance. Thirty seconds with a warning is just a wait:

resultHint.textContent = slowMode
  ? 'Using the slow path — this takes about 30 seconds. Please don\'t click again.'
  : 'Standard quality downloads immediately.';

3. The health check got deeper. /api/health only proved the process was alive. There's now an endpoint that actually exercises the upstream path and returns 503 when the fast path is dead, so an uptime monitor catches it in minutes instead of me noticing days later:

app.get('/api/health/deep', async (req, res) => {
  const probe = await checkFastPath(SAMPLE_URL);
  res.status(probe.healthy ? 200 : 503).json({ ok: true, upstream: probe });
});

The general shape of this bug

I've since run into the same pattern somewhere completely different — scraping an exhibitor directory, where the page reliably served a "contact by email" link that turned out to belong to the event organiser, not the exhibitor. Extract it and you get a thousand rows of perfectly-formatted, completely worthless data. Same class of problem: the operation succeeds, the output is well-formed, and the content is wrong.

The common thread is that most error handling checks whether an operation completed, not whether the result is usable. Those are different questions, and only the first one is easy.

So the habit I picked up:

• Validate the contract, not the transport. After parsing a response, assert the fields the caller depends on. res.ok tells you about the network. It tells you nothing about the payload.
• Make fallbacks noisy. A fallback that runs silently is indistinguishable from a fallback that never runs. Log it, count it, and surface it in the response.
• Health-check the dependency, not the process. "Is my server up" and "does my server still work" are different questions. Only the second one matters to users.
• Distinguish empty from absent. A zero-length body with a 200 is a specific, detectable condition. Give it its own error reason so it shows up in logs as itself.

None of this is sophisticated. It's just being explicit about a thing that's easy to leave implicit — and the cost of leaving it implicit was several days of a broken service that reported itself healthy.

What I'd still do differently

I'd add the deep health check first, before any of the code changes. The fix took an afternoon. Not knowing there was a problem took days, and that was the expensive part.

---

I write about the things that break in the services I run. The subprocess-hardening layer I pulled out of this project — SSRF host allowlist, command and argument injection guards, timeouts and concurrency caps — is on GitHub.

Comments

[Luis Cruz]

It's interesting how the issue with your API highlights the importance of validating the shape of the response, not just the status code. Implementing stricter checks on the expected structure could prevent such silent failures in the future. Consider leveraging TypeScript or even a schema validation library to enforce these checks at the response level. If you’re looking for help refining the implementation or exploring resilience strategies, I’d be glad to discuss a paid collaboration. What are your thoughts on adding additional monitoring for payload integrity?

[Haruna]

Thanks for reading. On the TypeScript suggestion I'd push back slightly, because it's the part that surprised me too: types are erased at compile time, so data: VideoInfo gives no protection at all when the upstream returns a 200 with a zero-length body. The compiler is happy and the object is empty. Runtime schema validation (zod, ajv) is the thing that would have caught it — genuinely a different tool for a different layer.

That's what the second half of the post does, just hand-rolled: the upstream client returns fastPath: false with a reason code instead of passing through whatever arrived, and /api/health/deep exercises the upstream path and returns 503 when it's dead. So we agree on the direction — I'd only swap TypeScript for a runtime validator.

I'm not hiring at the moment, I'm on the other side of that: taking on scraping and automation work myself. Happy to talk shop though.