DEV Community

Cover image for My CI Was Green. Production Returned 403. So I Built ProdDoctor.
Lucas Lu
Lucas Lu

Posted on

My CI Was Green. Production Returned 403. So I Built ProdDoctor.

#devops #cloudflarechallenge #webdev #github

Everything looked fine.

CI was green. ✅

The deployment succeeded. ✅

The platform URL worked. ✅

Then I opened the domain that actual users were supposed to visit.

403.

That was the moment I realized something obvious in hindsight:

A successful deployment does not necessarily mean your production site works.

The weird part

The application itself wasn't broken.

The build completed successfully.

The deployment platform said everything was healthy.

Even the platform-provided URL worked.

But my real production domain didn't.

In my case, the real production domain was returning a Cloudflare 403, while the platform-provided URL still worked.

That immediately pointed the investigation away from the build itself and toward the production edge layer: custom-domain routing, Cloudflare, or WAF behavior.

So from the CI pipeline's point of view:

build → deploy → success
Enter fullscreen mode Exit fullscreen mode

From the user's point of view:

open website → 403
Enter fullscreen mode Exit fullscreen mode

Two completely different realities.

CI was telling the truth

The strange thing is that CI wasn't wrong.

It did exactly what I asked it to do.

It verified that the code built successfully and that the deployment command completed.

What it didn't verify was everything that existed after deployment:

DNS
 ↓
custom domain
 ↓
TLS
 ↓
redirects
 ↓
CDN / proxy
 ↓
WAF
 ↓
JavaScript + CSS
 ↓
browser runtime
 ↓
what the user actually sees
Enter fullscreen mode Exit fullscreen mode

A green CI checkmark only tells me that the pipeline finished.

It doesn't necessarily tell me that the production experience works.

That gap bothered me.

So I started checking the real production path

The obvious first solution was adding a request after deployment:

curl -f https://example.com
Enter fullscreen mode Exit fullscreen mode

That's already better than doing nothing.

But then I started thinking about all the ways production can still be broken.

A page can return HTTP 200 but contain the wrong deployment.

The HTML can load while a critical JavaScript bundle returns 404.

A custom domain can reach a different route from the platform URL.

A CDN or WAF can block the request.

The server response can look healthy while the browser crashes on a JavaScript error.

A redirect can quietly send users somewhere completely different.

So checking one status code wasn't really what I wanted.

I wanted something that could answer:

Does the production experience users actually hit work, and if not, where did it break?

That became ProdDoctor 🩺

I turned that debugging workflow into an open-source GitHub Action called ProdDoctor.

Instead of trusting the deployment result, ProdDoctor checks the real production URL after deployment.

It currently validates things like:

  • DNS resolution
  • HTTP status and redirects
  • expected page content
  • TLS certificates
  • same-origin JavaScript and CSS assets
  • common Cloudflare Challenge / WAF behavior
  • optional real Chromium rendering
  • uncaught JavaScript errors
  • failed critical browser requests
  • screenshots
  • Playwright traces
  • HTML and JSON reports

The part I care about most isn't adding another red or green checkmark.

It's narrowing the failure down to the layer that probably broke.

Instead of getting:

❌ Production check failed
Enter fullscreen mode Exit fullscreen mode

I want something closer to:

DNS            PASS
TLS            PASS
HTTP           FAIL (403)
Cloudflare     Challenge / WAF suspected
Enter fullscreen mode Exit fullscreen mode

Now I know where to start looking.

The current stable release, v2.1.0, also adds a likely-cause summary to make failed checks easier to diagnose.

The setup ended up being tiny

For a basic production check:

- uses: lucaswenbo/ProdDoctor@v2.1.0
  with:
    url: https://example.com
    expect: My Website
Enter fullscreen mode Exit fullscreen mode

That's a GitHub Actions step.

Put it after your deployment step and point it at the real public production URL, not the platform preview URL.

For example:

name: Deploy and verify

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      # Your existing build and deployment steps go here.

      - name: Verify real production domain
        uses: lucaswenbo/ProdDoctor@v2.1.0
        with:
          url: https://example.com
          expect: My Website
Enter fullscreen mode Exit fullscreen mode

The default mode stays lightweight.

No Cloudflare API token is required.

If I need deeper evidence, I can enable browser validation:

- uses: lucaswenbo/ProdDoctor@v2.1.0
  with:
    url: https://example.com
    browser: true
Enter fullscreen mode Exit fullscreen mode

ProdDoctor can then launch Chromium and preserve evidence such as a full-page screenshot, browser failures, a Playwright trace, and reports.

Why HTTP 200 isn't enough

This was another thing that became obvious while building it.

Imagine this:

HTML              ✅ 200
app.js            ✅ 200
TLS               ✅
Browser render    ❌ white screen
Enter fullscreen mode Exit fullscreen mode

Or:

HTTP              ✅ 200
Expected content  ❌ missing
Enter fullscreen mode Exit fullscreen mode

Or even:

Platform URL      ✅
Custom domain     ❌ 403
Enter fullscreen mode Exit fullscreen mode

All three can happen after a deployment system happily reports success.

That's why ProdDoctor separates lightweight HTTP-level checks from optional browser-level validation.

Most sites don't need Chromium on every run.

But when the problem only appears after JavaScript executes, a simple HTTP request can't see the whole picture.

I'm not trying to build another giant monitoring platform

I want ProdDoctor to stay focused on one question:

Did the thing I just deploy actually work for the user?

There are already great monitoring and observability platforms.

ProdDoctor is aimed at a slightly different moment:

code
 ↓
CI
 ↓
deploy
 ↓
real production URL
 ↓
verify what users actually get
Enter fullscreen mode Exit fullscreen mode

That last step was the one missing from my own workflow.

One important limitation

Cloudflare / WAF detection is heuristic.

ProdDoctor doesn't access your Cloudflare account or read private WAF logs.

It looks at what a public client can observe, including status codes and common Challenge markers.

So if it says a Cloudflare Challenge or WAF is likely involved, that's a debugging direction, not magical access to Cloudflare's internal decision logs.

I think that distinction matters.

The lesson I took from this

I used to think the pipeline ended here:

code → CI → deploy ✅
Enter fullscreen mode Exit fullscreen mode

Now I think it should end here:

code
 ↓
CI
 ↓
deploy
 ↓
production
 ↓
user-visible behavior ✅
Enter fullscreen mode Exit fullscreen mode

The deployment platform knows whether it successfully deployed something.

The production URL tells you whether users can actually reach and use it.

Those are not always the same thing.

I'm still building it

ProdDoctor is open source, and I'm deliberately keeping it focused on production validation.

👉 Check out ProdDoctor on GitHub

Current stable release: v2.1.0

I'm especially interested in failures that slip through normal CI:

  • custom-domain failures
  • CDN / WAF surprises
  • broken assets
  • browser-only crashes
  • stale or wrong deployments
  • weird redirect behavior

What's the strangest "CI was green, production was broken" incident you've run into?

If there's a failure mode ProdDoctor doesn't catch yet, tell me about it.

Real production failures are exactly what I want to turn into the next checks.

Top comments (0)