Build a Fail-Open Shim for a Free Model Endpoint That Goes Quiet

Why this is worth reading

A free model endpoint is the most useful dependency you should never trust. It can throttle, hang, or return structurally invalid JSON, and if that call sits inside a CLI or batch script, one 429 can stop a run that was otherwise fine. The useful fix is not a bigger prompt or a paid fallback; it is a small adapter that places the model behind the same failure discipline you would apply to any external API.

In this walkthrough, you will build a TypeScript shim with a timeout, schema check, usage log, and deterministic fallback. The result is copyable into any Node CLI and gives you a clean exit when the free endpoint stops answering.

MonkeyCode's operator describes the project as open source and lists a free model tier with a 30 million token allowance and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not run a long-term audit of the quota or server lifetime, so I treat both as claims to verify rather than constants. The shim below does not depend on them; it only assumes that the endpoint will eventually fail for reasons other than token count.

The failure shape you are isolating

When you call a model endpoint from a CLI, three problems tend to arrive at the same time:

1. Timeout — the provider is overloaded or the network path is slow, so your script hangs.
2. Structural failure — the response is HTTP 200 but the content is not the JSON shape your next step expects.
3. Quota or availability failure — the endpoint returns 429 or 500 and your script has no retry or fallback path.

Your adapter should not need to know which one happened to keep running. It should make one attempt, enforce a deadline, validate what came back, record what it used, and return a stable local result when the model path fails.

Define the contract

Start with a small TypeScript file that defines the model provider and the fallback decision shape. You do not need a framework; fetch, AbortController, and a few type guards are enough.

type ChatMessage = {
  role: 'system' | 'user' | 'assistant';
  content: string;
};

type ModelProvider = {
  name: string;
  endpoint: string;
  key?: string;
  timeoutMs: number;
  maxTokens: number;
  schemaCheck?: (value: unknown) => boolean;
};

type CallResult<T> = {
  result: T;
  source: 'model' | 'fallback';
  usage?: Record<string, number>;
  error?: string;
  latencyMs: number;
};

The interface is deliberately small. You can swap the provider later without touching the rest of your CLI, which matters when a free endpoint changes its base URL or authentication style.

Parse the model response carefully

Chat-style endpoints often wrap the assistant answer inside choices[0].message.content. That content may be a JSON string or plain text. Your parser should not throw on the first mismatch; it should return a failed parse that the fallback path can handle.

function parseModelContent(raw: unknown): { ok: true; value: unknown } | { ok: false; error: string } {
  if (!raw || typeof raw !== 'object') {
    return { ok: false, error: 'non-object response' };
  }

  const obj = raw as Record<string, unknown>;
  const content = obj.choices?.[0]?.message?.content;

  if (typeof content !== 'string') {
    return { ok: false, error: 'missing choices[0].message.content' };
  }

  try {
    return { ok: true, value: JSON.parse(content) };
  } catch {
    return { ok: true, value: content };
  }
}

function extractUsage(raw: unknown): Record<string, number> | undefined {
  if (!raw || typeof raw !== 'object') return undefined;
  const obj = raw as Record<string, unknown>;
  const usage = obj.usage as Record<string, unknown> | undefined;
  if (!usage) return undefined;

  return Object.fromEntries(
    Object.entries(usage).filter((entry): entry is [string, number] => typeof entry[1] === 'number')
  );
}

Add the fail-open call

The core function makes one attempt, aborts it after the configured timeout, and always returns a result. The fallback is what keeps your batch job alive.

async function callWithFallback<T>(
  provider: ModelProvider,
  messages: ChatMessage[],
  fallback: (messages: ChatMessage[]) => T
): Promise<CallResult<T>> {
  const started = Date.now();
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), provider.timeoutMs);

  try {
    const response = await fetch(provider.endpoint, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        ...(provider.key ? { authorization: `Bearer ${provider.key}` } : {})
      },
      body: JSON.stringify({
        messages,
        max_tokens: provider.maxTokens
      }),
      signal: controller.signal
    });

    clearTimeout(timer);

    if (!response.ok) {
      return {
        result: fallback(messages),
        source: 'fallback',
        error: `HTTP ${response.status}`,
        latencyMs: Date.now() - started
      };
    }

    const raw = await response.json();
    const parsed = parseModelContent(raw);

    if (!parsed.ok || (provider.schemaCheck && !provider.schemaCheck(parsed.value))) {
      return {
        result: fallback(messages),
        source: 'fallback',
        error: parsed.ok ? 'schema mismatch' : parsed.error,
        latencyMs: Date.now() - started
      };
    }

    return {
      result: parsed.value as T,
      source: 'model',
      usage: extractUsage(raw),
      latencyMs: Date.now() - started
    };
  } catch (err) {
    clearTimeout(timer);
    return {
      result: fallback(messages),
      source: 'fallback',
      error: err instanceof Error ? err.name : String(err),
      latencyMs: Date.now() - started
    };
  }
}

Notice that an abort error and an HTTP 500 both end up in the fallback branch. That is deliberate. At 2 a.m., you usually care that the run continued, not which exact transport error occurred.

Make the fallback honest

The fallback should not pretend to be a model. It should return a stable shape that your CLI can detect and act on. For a canary task, a local echo with fallback: true is enough.

function fallbackReply(messages: ChatMessage[]) {
  const lastUserMessage = messages
    .slice()
    .reverse()
    .find((message) => message.role === 'user');

  return {
    ok: false,
    fallback: true,
    reason: 'model_unavailable',
    echo: lastUserMessage?.content.slice(0, 140) ?? ''
  };
}

If your next step requires a real model decision, this fallback will not satisfy it. In that case, your CLI should stop instead of continuing silently. A fail-open shim is only correct when a deterministic local result is acceptable for the task.

Wire up a canary command

Add a simple command-line path so you can test the adapter without building the rest of the application.

const isTaskResult = (value: unknown): boolean => {
  if (!value || typeof value !== 'object') return false;
  return 'ok' in (value as Record<string, unknown>);
};

const provider: ModelProvider = {
  name: 'free-model',
  endpoint: process.env.MODEL_ENDPOINT ?? '',
  key: process.env.MODEL_KEY,
  timeoutMs: Number(process.env.MODEL_TIMEOUT_MS ?? 8000),
  maxTokens: Number(process.env.MODEL_MAX_TOKENS ?? 256),
  schemaCheck: isTaskResult
};

if (process.argv.includes('--canary')) {
  const result = await callWithFallback(
    provider,
    [
      { role: 'system', content: 'Return JSON with an ok boolean.' },
      { role: 'user', content: 'canary' }
    ],
    fallbackReply
  );

  console.log(JSON.stringify(result, null, 2));
}

Run it with a short timeout and an empty endpoint to confirm the fallback path fires.

npm init -y
npm install typescript tsx @types/node
MODEL_ENDPOINT=http://127.0.0.1:9 MODEL_TIMEOUT_MS=1000 npx tsx free-model-shim.ts --canary

The expected output has source: 'fallback' and an error such as AbortError or fetch failed. That is the clean exit you are designing for.

Test against a local mock server

A real free endpoint can change under you, so a reproducible local mock is more useful than a live URL for the first test pass. This mock reads a failure mode from the environment and does nothing else.

// mock-model-server.mjs
import http from 'node:http';

const mode = process.env.FAIL_MODE ?? 'ok';

const server = http.createServer((req, res) => {
  res.setHeader('content-type', 'application/json');

  if (mode === 'timeout') {
    setTimeout(() => res.end(), 30000);
    return;
  }

  if (mode === 'bad-json') {
    res.end(JSON.stringify({ choices: [{ message: { content: '{"ok":' } }] }));
    return;
  }

  if (mode === 'http-500') {
    res.statusCode = 500;
    res.end(JSON.stringify({ error: 'upstream exploded' }));
    return;
  }

  res.end(
    JSON.stringify({
      choices: [{ message: { content: '{"ok":true,"text":"canary"}' } }],
      usage: { prompt_tokens: 12, completion_tokens: 4, total_tokens: 16 }
    })
  );
});

server.listen(Number(process.env.PORT ?? 8787));

Start the mock in one terminal, then run the canary from another.

node mock-model-server.mjs &
MODEL_ENDPOINT=http://127.0.0.1:8787 MODEL_TIMEOUT_MS=1000 FAIL_MODE=timeout npx tsx free-model-shim.ts --canary

The test matrix you should record looks like this:

FAIL_MODE	Expected source	Expected error
timeout	fallback	AbortError
bad-json	fallback	schema mismatch or JSON parse
http-500	fallback	HTTP 500
ok	model	none

If any row returns source: 'model' when it should not, your parser or schema check is too permissive.

What this shim will not catch

This is not a production model gateway. It will not catch semantic drift, prompt-injection attempts against tool calls, streaming responses, or changes to the provider's data-retention policy. It also does not tell you whether the 30 million token allowance is still accurate; for that you need a separate token ledger and periodic quota audit. The shim only keeps a single request from taking your CLI down.

Do not use fail-open behavior when the model result is required for a decision. If you are labeling user data, approving a destructive command, or generating paid output, a silent local fallback can be worse than a crash. In those paths, make the fallback fail loudly or require a --strict flag.

When to use it

Use this pattern for side-project automation where one model call should not stop the rest of the run: release-note drafts, changelog categorization, small batch summaries, internal triage notes, or a canary that checks whether an endpoint is awake before a longer job starts.

The shim is intentionally provider-neutral. If you want to try it against a real free endpoint instead of the mock later, the open-source MonkeyCode project's free model access and free server option are a reasonable canary target. Just verify the current token limit and server constraints directly before you wire it into anything long-running.