Gotcha: chasing a bug that was never in my code

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

The build was done. Themis Lex worked on my machine, and not in the "works if you squint" way. A court clerk enters their role, describes their workflow, picks a data sensitivity level, and gets back a PDF with two sections: where AI can safely support the work, and where it must never touch it. Claude via Bedrock generates the assessment. Server-side PDF render. No accounts, no storage, session ends when the download does.

Three weeks solo, for the Women in AI Accelerator Spring 2026 Build Challenge.

Initial commit went in at 7:06pm on May 9. I pushed to AWS Amplify. Build went green. I opened the live site, filled out the form, hit submit.

Nothing.

Twenty eight seconds later, "Request timed out."

I told myself the bug was not in my code. Everything ran locally. This had to be a platform problem. That belief carried me all night. It mostly held up. The exception was the first thing I should have checked.

Here is the commit log, because it tells the story better than I can:

19:06  Initial commit: Themis Lex MVP
20:31  refactor: migrate Bedrock auth to IAM compute role
22:29  diag: log credential env vars at runtime (booleans only, remove after fix)
22:40  fix: forward BEDROCK_MODEL_ID to SSR runtime via next.config.js env
22:50  fix: switch to InvokeModelWithResponseStreamCommand to beat 28s Lambda timeout
  ...
06:28  fix: remove unused type export that broke isolatedModules build
06:37  fix: end-to-end response streaming to beat Amplify 28s gateway timeout
06:48  fix: reduce max_tokens to 3000 to fit Amplify 30s timeout
06:52  fix: reduce max_tokens to 2000, 3000 still exceeded 30s timeout
07:00  fix: switch to Claude Haiku 4.5 to fit Amplify 30s timeout

Ten and a half hours from first deploy to the fix that shipped it. That gap between 22:50 and 06:28 is me sleeping on it, which turned out to be the second most productive thing I did.

The error message was the absence of an error message

My first instinct was that my code threw and I swallowed it. I had a try/catch on the Bedrock call and it logged nothing. No stack trace. No AccessDenied. No throttle. My catch block was never entered, because my code never reached the part that could fail.

That is a specific kind of awful. A loud error points at a line. Silence points at everything.

The AWS SDK was stuck upstream of my logic, walking the credential provider chain looking for something to sign with, finding nothing, and waiting until the gateway killed it. From the outside that looks like a slow API. From the inside it is a permissions problem wearing a timeout costume.

So here is the correction to my own title, four paragraphs in. One of the bugs was absolutely in my code.

// c136357, initial commit. This disables the provider chain.
function createClient(): BedrockRuntimeClient {
  return new BedrockRuntimeClient({
    region: process.env.AWS_REGION || 'us-east-1',
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID || '',
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY || '',
    },
  });
}

// 792b94f, "migrate Bedrock auth to IAM compute role"
function createClient(): BedrockRuntimeClient {
  return new BedrockRuntimeClient({
    region: process.env.AWS_REGION || 'us-east-1',
  });
}

Passing credentials at all tells the SDK you have this handled, so it stops looking. Empty strings are still a value, so it tried to sign with nothing. Omit the object entirely and the default chain covers both local dev reading .env.local and production reading the Lambda's assumed role. Two lines deleted. That was mine.

At 22:29 I committed a diagnostic that logged credential env vars as booleans only, never values, because I could not tell from the outside whether the runtime had credentials at all. If you are stuck in the same silence, that is the move. Do not log the values. Log whether they exist.

One thing, since this challenge is powered by Sentry: I had no error tracking on this app at all. It also would not have mattered in the usual way, because there was no exception to capture. Nothing threw. The request stopped existing when the gateway hung up, and application level error handling has nothing to report about a request that never failed so much as ended. What I needed was not a stack trace. It was something watching from outside the Lambda that could tell me requests were dying at a suspiciously round twenty eight seconds, which is the pattern I only saw hours later by reading CloudWatch REPORT lines by hand.

I changed two things at once, so I still cannot tell you which one worked

I also changed the compute role's trust policy to include both amplify.amazonaws.com and lambda.amazonaws.com, and attached the role at the branch level instead of relying on the app level default. Redeployed. The credential hang was gone.

Classic mistake. Two changes, one test, no isolated variable.

When I went back to check my work for this post, the official docs contradict both of my theories. The AWS compute role page shows a single trust principal, amplify.amazonaws.com, and says an app level role applies to all branches by default. Another engineer tested the two principal theory and rejected it, finding instead that roles created by CLI behaved differently from roles created in the console.

I made both IAM changes at the same time as deleting the credentials object, the hang stopped, and I cannot prove which one did it. It may well have been the two deleted lines all along. Adding the second principal is harmless. Branch level attachment is recommended anyway. But if you came here for a confirmed root cause on the IAM half, I do not have one, and I would rather say that than hand you a confident answer I cannot back up.

What I can hand you is the diagnostic, which beats my guess:

aws amplify get-app --app-id YOUR_APP_ID
aws amplify get-branch --app-id YOUR_APP_ID --branch-name main

Compare computeRoleArn on both. And treat the IAM Console's "Last activity" on the role with suspicion, because the build service assuming the role lights it up whether or not your runtime ever gets credentials. It told me the role was in use. The role was in use. Just not by the thing that needed it.

The docs told me half of it. I was reading the wrong half.

Credentials working. Next failure, eleven minutes later: BEDROCK_MODEL_ID came back undefined in production. Set in Amplify Console. Visible during build. Gone at runtime.

This one is documented, and it is my miss, not AWS hiding the ball. The page is Making environment variables accessible to server-side runtimes, and it says the behavior is intentional. Amplify env vars flow into the build environment. They are not injected into the SSR Lambda. Vercel, Railway, and Render do both, which is why the assumption never got questioned.

AWS's sanctioned fix is a build spec line writing values into .env.production before the build. My amplify.yml does no such thing. It runs npm ci and npm run build and that is it. What I did instead, at 22:40, was inline the value through next.config.js:

const nextConfig = {
  env: {
    BEDROCK_MODEL_ID: process.env.BEDROCK_MODEL_ID,
  },
};

Mine works. Theirs is the documented path. Use theirs if you want to point at a doc when a teammate asks.

The nastier version of this bug is the silent one, and I shipped it. lib/bedrock.ts looked like this:

const modelId = process.env.BEDROCK_MODEL_ID || 'us.anthropic.claude-haiku-4-5-20251001-v1:0';

If that env var ever failed to reach the runtime again, nothing throws. The app quietly runs the hardcoded fallback, and I find out when someone tells me the output changed. I wrote a warning about this trap and then shipped the trap.

I fixed it while writing this post. The build now fails if BEDROCK_MODEL_ID is missing, the runtime throws a named config error instead of falling back, and the API route returns a message the browser can show instead of a blank screen. Writing it down is what made me go look at it.

Then Bedrock threw AccessDeniedException complaining about AWS Marketplace actions I did not think I was using. Since the October 2025 model access change, Bedrock auto enables models on first call, the enablement runs through Marketplace, and the calling principal needs permission for it. The old Console "Model access" page is retired.

{
  "Effect": "Allow",
  "Action": [
    "aws-marketplace:ViewSubscriptions",
    "aws-marketplace:Subscribe"
  ],
  "Resource": "*"
}

Resource: "*" because Marketplace subscriptions are not scoped to model ARNs. Documented, in a Bedrock security blog post, with no link to it from anywhere in the Amplify docs. To find it you have to already suspect Marketplace, which I did not, because I was configuring Bedrock.

The wall that is not written down anywhere

Credentials working. Permissions working. Env vars working. My API route still died at twenty eight seconds.

Amplify Hosting SSR has a hard gateway timeout in the twenty eight to thirty second range, and there is no setting AWS exposes to raise it.

I looked for this. It is not in Amplify quotas, which covers app counts and artifact sizes. It is not in troubleshooting SSR, which discusses 504s only in terms of a response size cap. As far as I can find, the number appears in no AWS documentation page at all.

Where it does appear is aws-amplify/amplify-hosting issue 3223, still open, where an AWS engineer states that increasing the timeout on SSR compute is not supported at this time. A companion issue at 3475 gets redirected back to it.

This is the part of the night I could not have prevented by reading. It is a platform constraint that lives in a GitHub thread.

On the numbers, I want to be precise about what I know. My CloudWatch REPORT lines showed Duration: 28006ms, which is the Lambda being killed, not the generation finishing. My planning notes from block 4 testing put Sonnet 4.6 at roughly forty two seconds at max_tokens=6000. That is an observation from testing, not instrumented measurement, because the call never completed inside the Lambda's life. I never got a clean number. I got a wall.

Streaming was the right idea, twice, and the wrong fix both times

Streaming looked like the answer. Keep bytes flowing, keep the connection alive. I tried it in two separate shapes, seven hours apart.

At 22:50 I switched to InvokeModelWithResponseStreamCommand inside lib/bedrock.ts, collecting chunks server-side. The route handler did not change. It still returned one JSON payload at the end. That keeps the Lambda demonstrably alive during generation, which is something, but it does nothing for a gateway measuring wall clock time on the response.

Then I slept.

At 06:37, after a build break at 06:28 over an unused type export, I went end to end. streamBedrock() became an async generator. pages/api/assess.ts set Content-Type: text/plain and Transfer-Encoding: chunked and called res.write() per chunk. The client used response.body.getReader().

Still dead at twenty eight seconds.

CloudWatch showed the Lambda receiving Bedrock chunks the whole time. The client saw the platform timeout. Same symptom as the credential bug, completely different cause. I spent the first twenty minutes of it re-checking IAM.

Next.js Pages API routes buffer. My writes did not reach the gateway as they were written. They queued until res.end(), and by then the connection was gone. I reverted the whole end to end attempt eleven minutes later.

AWS does document this, sort of. Amplify support for Next.js lists "Next.js streaming" flatly under unsupported features. One line, no detail. I read that page. I read it as a caveat rather than a wall, which is a reading comprehension problem I have been thinking about since.

The server-side collection version survived, and it is what runs today. The client still gets one JSON response.

Going smaller did not work. Going faster did.

After the revert, I did the predictable thing and made the response smaller.

max_tokens 6000 to 3000 at 06:48. Still over.
3000 to 2000 at 06:52. Still over.

Four minutes apart, watching the same wall. At that point the output was small enough to hurt the product and still too slow to ship.

So at 07:00 I stopped shrinking the response and changed the model. Claude Sonnet 4.6 to Claude Haiku 4.5.

Before: nothing rendered. Twenty eight second hang, platform error page, max_tokens cut to a third of the design target and still failing.

After: PDF generated and downloaded. The test call that finally returned HTTP 200 showed 14.3 seconds in my terminal. That number is not in any log file or benchmark in the repo, so take it as what it is, one observed run during a live test at seven in the morning. What I can point to is the outcome: it fit, and it fit with enough room that I raised max_tokens back up to 4000, above where I had cut it and closer to what the assessment needs.

Themis Lex went live at themislex.org running Haiku.

There is a trade here and I am not going to pretend otherwise. Haiku's assessments are less nuanced than Sonnet's were. For a tool telling court staff where AI must never touch their work, the nuance is the thing they are paying attention to. I logged the quality delta as a v2 item rather than calling it a win.

The other paths out, if you cannot shrink or speed up enough:

• Lambda Function URL with response streaming enabled. Fifteen minute timeout, server sent streaming that works.
• A separate Lambda behind API Gateway. Sixty second timeout.
• Move hosting. Vercel Pro gives sixty seconds and first class Next.js streaming. None of those are an overnight fix. They are architecture decisions, which means they belong in the plan before you build, not in the panic after you deploy.

What I would do differently

About half that night was avoidable and half was not, and the split is the lesson.

Avoidable: the env var behavior and the streaming limitation are both written down. I did not read the right pages, and once I did, both took minutes. If you are deploying Next.js SSR to Amplify, read ssr-environment-variables.html first. It states the runtime gap outright and links onward to compute roles, which is the on ramp to the whole IAM half of my night.

Also avoidable: the credentials object. That was two lines of my own code buying me hours of platform suspicion.

Not avoidable: the twenty eight second wall. No doc, no quota page, no warning. A GitHub issue and an AWS engineer saying no.

The question I did not ask before choosing a platform, and will ask every time now: will any route in this app take longer than twenty five seconds end to end? LLM calls, PDF renders, large file work, slow third party APIs. If yes, Amplify SSR is the wrong default and no amount of configuration changes that.

If no, Amplify is fine. It is a good product for fast routes and content sites and normal CRUD, and the developer experience holds up. I do not mind that the constraint exists. I mind that I found it at 6:52 in the morning, cutting max_tokens for the second time, instead of on a docs page in week one.

I still use Amplify. I stopped reaching for it first, and I now run a fifteen minute timing spike on the slowest route before committing to any host. Pass or fail, before I build on it.

Themis Lex shipped. It runs a smaller model than I designed it around, and I would rather tell you that than let you think the night went clean.

---

Quick context if you are new here: I came to code from the courtroom. Jury services to AI builder in about a years, self-taught, learning in public. I direct, the agents generate, I validate and decide. I build the Clew Suite and a handful of civic tech tools. That is the lens I am writing from.

Repo: github.com/earlgreyhot1701D/themis-lex
Live: themislex.org

AI Assisted. Human Approved. Powered by NLP.