Surfacing swallowed provider errors in the Sentry MCP test client

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

Sentry MCP is a Model Context Protocol server that exposes Sentry's error and performance data to AI assistants. It ships a small CLI test client (packages/mcp-test-client) that drives the server through a real LLM so you can exercise the tools end to end. This fix lands in runAgent inside that test client. It closes issue #500, which was filed by Sentry's founder.

Bug Fix or Performance Improvement

When the client hit a provider error (an invalid API key, a rate limit or a 5xx), it printed (No response generated) and moved on. The real reason was lost. The cause is a detail of the Vercel AI SDK the repo pins (ai@6): streamText does not throw when the request fails. It ends textStream with zero chunks and reports the failure only through an onError callback the code never set. So the for await loop finished empty, runAgent never entered its catch, the Sentry span kept its OK status and nothing was captured. The client then fell through to the misleading empty-response message. The failure was invisible to the user and to Sentry at the same time. I confirmed this against the installed SDK before writing anything: a probe showed textStream yields zero chunks without throwing, while onError fires with an APICallError carrying statusCode: 401. The issue guessed the error might sit on result.errorPromise; on this SDK version that property does not exist, so the fix does not use it.

Code

The client now captures what the SDK reports through onError, then surfaces it once the stream drains so it flows through the existing catch:

let streamError: unknown;

const result = await streamText({
  model: languageModel,
  system: SYSTEM_PROMPT,
  messages: [{ role: "user", content: userPrompt }],
  tools,
  stopWhen: stepCountIs(maxSteps),
  experimental_telemetry: { isEnabled: true },
  onError: ({ error }) => {
    // Provider failures arrive here rather than as a thrown error.
    streamError = error;
  },
  onStepFinish: (/* ... */) => { /* ... */ },
});

for await (const chunk of result.textStream) {
  /* ... stream chunks to the terminal ... */
}

// Surface the provider failure instead of the silent fallback. Throwing
// routes it through the catch, which marks the span errored and rethrows.
if (streamError !== undefined) {
  if (isStreaming) {
    logStreamEnd();
    isStreaming = false;
  }
  throw new Error(describeProviderError(streamError, provider), {
    cause: streamError,
  });
}

The mapper turns the raw error into the message the issue asked for. It also unwraps a RetryError, because retryable statuses (429 and 5xx) are retried by the SDK and then delivered wrapped, with the HTTP status on lastError:

export function describeProviderError(error, provider) {
  const label = provider === "openrouter" ? "OpenRouter API" : "OpenAI API";
  const envVar =
    provider === "openrouter" ? "OPENROUTER_API_KEY" : "OPENAI_API_KEY";

  const apiError = findApiCallError(error); // unwraps RetryError / cause
  if (apiError) {
    const status = apiError.statusCode;
    if (status === 401 || status === 403)
      return `${label} authentication failed. Please check your ${envVar} environment variable.`;
    if (status === 429)
      return `${label} rate limit exceeded. Please wait and try again.`;
    if (status !== undefined && status >= 500)
      return `${label} service error. The service may be temporarily unavailable.`;
    return `${label} request failed${status ? ` (HTTP ${status})` : ""}: ${apiError.message}`;
  }
  const message = error instanceof Error ? error.message : String(error);
  return `${label} request failed: ${message}`;
}

My Improvements

The behavior change is small on purpose. A provider error now produces a clear, provider-aware message that names the environment variable to check. The legitimate empty-response path is untouched, so a genuine empty reply still reads (No response generated). I added nine tests. Integration tests through runAgent prove a 401 stream surfaces the auth message, rejects and does not print the fallback. They also prove a real 429 (driven through the SDK's actual retry path, so a real RetryError) surfaces the rate limit message. Unit tests cover the full mapping: 401, 403, 429, 5xx, the RetryError unwrap, a generic 400 and a non-API error. To keep the fix honest I reverted only the source change and watched the new tests fail, then restored it and watched them pass. The package suite goes from 70 to 79 passing. Type-check, Biome lint and format plus ast-grep are all green on the changed package.

Best Use of Sentry

The whole point of the fix is that Sentry can finally see the failure. runAgent wraps its work in a Sentry span; before the fix that span stayed OK on a provider error and no event was captured. Now the thrown error runs through the catch, which calls span.setStatus({ code: 2 }) and rethrows, so the top-level Sentry.captureException in the CLI records it and the cause carries the original APICallError for full context. I proved this with a real send to a Sentry DSN: the harness stubs a 401 (no real model call is placed), runs the real runAgent and logs through beforeSend and beforeSendTransaction exactly what leaves the process. Before the fix, the root span status transmitted is ok and no exception event is sent. After the fix, a real exception event is captured and the root span status transmitted is internal_error. The same before and after is also enforced offline by span-status assertions in the test suite, so the guarantee does not depend on the network.

The same failure, now a real Sentry issue (environment bugsmash-issue-500-demo, level error). The stack trace shows the fix's own code: the throw new Error(describeProviderError(...)) that routes the error through the catch so Sentry records it. Before the fix this printed (No response generated) and no event reached Sentry at all.

AI assistance (Claude, Anthropic) was used in developing this change. The design, review and verification were done by the author. Verified locally before submitting: the @sentry/mcp-test-client vitest suite (79 passing, up from 70), tsc --noEmit for the package, Biome lint and format, ast-grep, plus a real Sentry send showing the error now reaches Sentry with the span marked errored.