DEV Community: Micheal Heypico

Prompt versioning: treat prompts like code (with a 3-model test harness)

Micheal Heypico — Thu, 01 Oct 2026 12:00:13 +0000

Prompt versioning: treat prompts like code (with a 3-model test harness)

Prompts are code. They have regressions, environments, and review requirements - they just don't have a compiler to warn you.

What we do at HeyPico

Every prompt lives in git with a version tag. Every change runs against a fixed eval set before shipping, tested on three models minimum: one frontier (Claude or GPT class), one cheap (DeepSeek or GLM class), one wild card. Why three? Because providers fail differently, and a prompt tuned to one model's quirks silently breaks when you switch.

The failures are humbling:

A prompt that worked on Claude lost tone constraints on a smaller model.
A JSON-output prompt returned wrapped markdown on one provider, clean JSON on another.
An instruction that everyone follows ("answer in <=2 sentences") gets ignored by exactly the cheap models you'd use for bulk work.

The setup, concretely

prompts/ directory, one file per prompt, versioned.
evals/ directory, one JSONL per prompt with input + expected properties.
A runner that hits each model through one API endpoint and diffs behavior.
Merge blocked until eval diff is clean.

The multi-model routing part is one API key with 32 models - the eval runner is ~200 lines of Python. Ping us if you want to replicate it.

Structured output gotchas across 32 LLM providers (and how to test for them)

Micheal Heypico — Mon, 28 Sep 2026 03:02:40 +0000

Structured output gotchas across 32 LLM providers (and how to test for them)

We route 32 models behind one OpenAI-compatible key at heypico.ai, which means we see every provider's idea of "JSON mode" up close. They are not the same.

The gotchas that actually bit us

Trailing commas and prose wrappers. Some models emit clean JSON in JSON mode; others still wrap objects in markdown fences even when told not to. We now strip fences before parsing.
Enum drift. You ask for status: "ok" | "error"; some providers return SUCCESS. A schema check with explicit enum coercion fixed a production bug.
Function-calling shape differences. Arguments arrive as a JSON string on some providers, an object on others. Normalize at the boundary.
Refusals disguised as output. A model that refuses a task sometimes still emits valid-but-wrong JSON. Test refusal paths, not just happy paths.

The harness

Every prompt ships with a schema + a 15-case test set. CI runs it against 3 models minimum (one frontier, one cheap, one open-weights) before merge. It is boring and it catches real bugs.

If you want the multi-model part without the plumbing: HeyPico gives you 32 models behind one OpenAI-compatible key. Singapore-based, CASA Tier 2.

Model failover for AI apps: stop writing retry loops, start routing

Micheal Heypico — Thu, 24 Sep 2026 12:00:04 +0000

Model failover for AI apps: stop writing retry loops, start routing.

Like a lot of teams, our AI stack was four subscriptions deep: ChatGPT Plus, Claude Pro, Gemini, and a transcription tool. US$80 a month, four logins, and comparing models on the same task meant copy-paste between tabs.

We consolidated into HeyPico — 32 frontier models behind one OpenAI-compatible key — and the architecture decisions that mattered most:

One endpoint, every provider

Every model behind our key speaks the OpenAI chat-completions format:

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_HEYPICO_KEY",
    base_url="https://api.heypico.ai/v1"
)

response = client.chat.completions.create(
    model="gpt-5.6",  # or claude, gemini, deepseek, glm, grok...
    messages=[{"role": "user", "content": "Summarize this transcript"}]
)

Change one string, get a different provider. Your retry logic, logging, and eval harness stay untouched.

Failover is a config change, not a retry loop

We got throttled by a provider mid-demo once. With routing in place, failover becomes:

Coding tasks: primary Claude, fallback GPT
Long-context summarization: primary Gemini, fallback DeepSeek
Cheap bulk classification: GLM or Qwen

Your product stops dying when one provider has a bad day.

Prompt versioning from day one

Prompts are code. We keep ours in git and test every change against 3+ models before shipping. Structured output is where providers differ most — test that path specifically.

Try it

HeyPico is Singapore-based, CASA Tier 2 certified, and your data never trains models. First 100 developers get a free trial: heypico.ai/api-integration

We run a Telegram community where builders tell us what breaks: t.me/heypico

Prompt versioning: treat prompts like code (with a 3-model test harness)

Micheal Heypico — Mon, 21 Sep 2026 03:03:13 +0000

Prompt versioning: treat prompts like code (with a 3-model test harness)

Prompts are code. They have regressions, environments, and review requirements - they just don't have a compiler to warn you.

What we do at HeyPico

The failures are humbling:

A prompt that worked on Claude lost tone constraints on a smaller model.
A JSON-output prompt returned wrapped markdown on one provider, clean JSON on another.
An instruction that everyone follows ("answer in <=2 sentences") gets ignored by exactly the cheap models you'd use for bulk work.

The setup, concretely

prompts/ directory, one file per prompt, versioned.
evals/ directory, one JSONL per prompt with input + expected properties.
A runner that hits each model through one API endpoint and diffs behavior.
Merge blocked until eval diff is clean.

The multi-model routing part is one API key with 32 models - the eval runner is ~200 lines of Python. Free trial if you want to replicate it.

Structured output gotchas across 32 LLM providers (and how to test for them)

Micheal Heypico — Thu, 17 Sep 2026 17:31:52 +0000

Structured output gotchas across 32 LLM providers (and how to test for them)

We route 32 models behind one OpenAI-compatible key at heypico.ai, which means we see every provider's idea of "JSON mode" up close. They are not the same.

The gotchas that actually bit us

Trailing commas and prose wrappers. Some models emit clean JSON in JSON mode; others still wrap objects in markdown fences even when told not to. We now strip fences before parsing.
Enum drift. You ask for status: "ok" | "error"; some providers return SUCCESS. A schema check with explicit enum coercion fixed a production bug.
Function-calling shape differences. Arguments arrive as a JSON string on some providers, an object on others. Normalize at the boundary.
Refusals disguised as output. A model that refuses a task sometimes still emits valid-but-wrong JSON. Test refusal paths, not just happy paths.

The harness

Every prompt ships with a schema + a 15-case test set. CI runs it against 3 models minimum (one frontier, one cheap, one open-weights) before merge. It is boring and it catches real bugs.

If you want the multi-model part without the plumbing: HeyPico gives you 32 models behind one OpenAI-compatible key. Singapore-based, CASA Tier 2, free trial for the first 100 devs.

How we replaced 4 AI subscriptions with one API key (and added model failover)

Micheal Heypico — Mon, 14 Sep 2026 04:34:54 +0000

Like a lot of teams, our AI stack had grown into a subscription pile:

ChatGPT Plus — $20/mo
Claude Pro — $20/mo
Gemini — bundled but limited
A transcription tool — $17/mo

About US$80 a month, four logins, and every time we wanted to compare models on the same task we had to copy-paste between tabs.

So we built HeyPico — and in this post I'll share the architecture decisions that actually mattered, in case you're building something similar.

1. One OpenAI-compatible endpoint for everything

The single best decision we made: every model behind our key speaks the OpenAI chat-completions format. No SDK rewrites when you switch models.

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_HEYPICO_KEY",
    base_url="https://api.heypico.ai/v1"
)

response = client.chat.completions.create(
    model="gpt-5.6",  # or claude, gemini, deepseek, glm, grok...
    messages=[{"role": "user", "content": "Summarize this meeting transcript"}]
)

Change one string, get a different provider. Your eval harness, your retry logic, your logging — all of it stays untouched.

2. Model failover instead of retry loops

We got throttled by a provider mid-demo once. Never again. With 32 models behind one key, failover becomes a config change:

Coding tasks: primary Claude, fallback GPT
Long-context summarization: primary Gemini, fallback DeepSeek
Cheap bulk classification: GLM or Qwen

The point isn't that any single model is better. It's that your product stops dying when one provider has a bad day.

3. Prompt versioning from day one

Prompts are code. We keep ours in git, and every prompt change gets tested against 3 models before shipping. The failures are humbling — a prompt that works on Claude sometimes breaks subtle constraints on smaller models. Testing caught several of those before users did.

4. Structured output is where providers differ most

JSON mode, function calling, response formats — the details vary more between providers than raw quality does. If your pipeline depends on structured output, test that path specifically. We built our schema-checking layer after a silent formatting bug made it to production.

Try it

HeyPico is a Singapore-based, CASA Tier 2 certified platform (your data never trains models). First 100 developers get a free trial:

heypico.ai/api-integration

We also run a Telegram community where builders tell us what breaks: t.me/heypico. I read every message.

What's your current AI stack costing you? The math surprised us when we actually added it up.

Cover photo by DeepMind on Unsplash.