DEV Community

Krish Verma
Krish Verma

Posted on

How I built a settings file that survives the next build: 2 failing tests, 1 Custom route, 0 silent switches

How I built a settings file that survives the next build: 2 failing tests, 1 Custom route, 0 silent switches

Ankita AI is a desktop AI assistant by Krish Verma — a little companion that lives on your desktop and gets things done while you're away. Last month it refused to open at all, because of one line in its own settings file. The fix: 40 tests, zero rewrites of the saved profile, one honest notice in Settings. This is the story of that fix, in Ankita — the site is at ankita-sable.vercel.app.

The morning it wouldn't open

A newer build had saved a provider name this build didn't know yet, and the next launch died at DesktopSettingsStore.load with SAVED_PROFILE_LOAD_FAILED: Choose a supported provider. The app wouldn't reach Settings. The user couldn't fix the setting that was breaking the app, because the app wouldn't open far enough to show Settings.

That is the worst kind of settings bug: the recovery path sits behind the thing that's broken. Deleting the profile would have thrown away the endpoint, key, and model the user had already configured.

What the loader does now

The repair lives in desktop/electron/settings.mjs, in the load() method, and it follows one rule: never let an unknown provider name lock the user out; never silently switch them to a different service.

When the saved file loads, the store checks whether saved.provider is a string this build doesn't recognise:

const compatible = typeof saved.provider === 'string'
  && !providers.has(saved.provider);
this.data = validatePatch(
  compatible ? { ...saved, provider: COMPATIBLE_PROVIDER } : saved
);
Enter fullscreen mode Exit fullscreen mode

COMPATIBLE_PROVIDER is just 'custom' — the existing OpenAI-compatible route. If the saved profile already carries a compatible endpoint and key (customApiBase, customApiKey), the app reuses them through Custom and shows a notice in Settings: "Your saved provider is newer than this build. Using its saved compatible connection through Custom."

With no endpoint saved, the app does the other honest thing: it opens Settings and shows "Open Settings / Providers to connect it; no replacement service was selected", and picks nothing. In fact applyDesktopSettings clears the inherited env endpoint, key, and model the moment a saved provider exists, so an absent endpoint can never silently borrow another service's credentials.

The fields this build doesn't know yet

The second half of the fix is about the fields, not the provider name. Every settings save used to hard-fail if the file contained a key the build didn't recognise — a newer window talking to an older main process would die with "Unknown desktop setting" on every save. Now validatePatch drops unknown keys from the validated data, and the store keeps them aside:

this.preserved = Object.fromEntries(
  Object.entries(saved).filter(([key]) => !keys.has(key))
);
Enter fullscreen mode Exit fullscreen mode

Later saves write { ...this.preserved, ...next } back to disk, so unknown fields survive round-trips privately — never displayed, never validated, never lost. Saves are temp-file plus rename at mode 0o600, so a half-written file can't strand the next launch. The boundary stays strict where it matters: an explicit update to an unsupported provider still throws Choose a supported provider. The loader is forgiving about what the past wrote; the writer is strict about what the user chooses now.

The numbers that prove it

The bug was reproduced first, with the exact saved profile: SAVED_PROFILE_LOAD_FAILED, two tests failing against 16 passing in the settings file. After the fix, the focused run across five files came back 40 tests, 40 pass, 0 fail, 0 skipped, in 14.583 seconds. Verification went beyond unit tests: native dev and built-renderer launches against an exact private copy of the failing profile, with successful model discovery and the notice visible in Settings.

Two smaller guards ride along in the same file. A saved context window or max-token value is capped at 10,000,000, so a typo can't ask for a billion-token window — and an explicit saved value always wins over the default, which matters because a compatible endpoint often advertises no context window at all. And testCustomProvider probes {base}/models with an 8-second timeout before the user commits to a custom endpoint, with a specific hint on HTTP 401 about the combined wk-<id>.ws-<secret> token format.

What this doesn't do

Loading never writes the file, so the profile on disk still names the newer provider — the normalisation to Custom happens in memory only. If that newer provider isn't actually OpenAI-compatible, the Custom route fails later at the /models probe or the first model call, not at load; the fix buys you a launch, not a guarantee. Preserved unknown fields are carried, not migrated — uninterpretable until a build that knows them arrives.

Tell me I'm wrong

The debatable choice is the reuse itself: when a saved provider name is unknown, I reuse its endpoint and key through Custom instead of stopping at setup and asking. That keeps the app opening, but the assistant is then talking to an endpoint under a label the user never chose — "Custom" — with a notice they might never read. Is silent reuse the right default, or should an unknown provider always stop at setup and make the user confirm? Tell me I'm wrong: where would you draw that line, and what would you do with the saved key while you wait?

How do you handle forward compatibility in your own app's settings — what do you preserve, what do you refuse, and what has bitten you?

— Krish Verma, creator of Ankita AI

Top comments (2)

Collapse
 
mateo_ruiz_6992b1fce47843 profile image
Mateo Ruiz •

The point about later steps breaking tests that previously passed is especially important. I'd treat every foundation step as a checkpoint with an independent acceptance test, then keep those checks in a regression suite as the chain progresses.

At IT Path Solutions, this is a key consideration when taking AI-generated applications toward production. The agent's tests are useful for fast feedback, but the acceptance criteria should live outside the agent's control. Otherwise, it can unintentionally redefine success around the implementation it just produced.

I'd also make the feedback loop evidence-based: return the exact failing test, error, and expected behavior to the agent rather than a generic instruction to fix it. That makes each repair round more targeted and gives the next step a verified foundation.

The real measure of a coding agent isn't how many consecutive tasks it can claim to complete, but how much verified functionality survives across the entire chain.

Collapse
 
krish_verma_77e28d3fd63ca profile image
Krish Verma •

You're right, and the checkpoint framing is the part I keep coming back to. For this fix the checkpoint was literal: the exact saved profile that broke the launch, reproduced first, and then the tests in test/desktop/desktop-settings.test.mjs pin each behaviour — 'a newer saved provider opens through its existing compatible endpoint without rewriting the profile', and 'an unknown settings key is ignored so a newer window can still save'. Those tests are committed to the repo and run in CI (release.yml runs npm test), so the acceptance bar lives outside whatever agent wrote the fix — it can't quietly redefine success, it has to make the committed tests pass.

The evidence-based loop is also how I actually work a repair round: the failing test name plus the exact error is the whole prompt, not 'fix it'. The honest caveat is that my suite pins behaviour, not intent — a test can pass while the notice in Settings goes unread, which is exactly the silent-reuse worry I ended the article on. The regression suite catches the launch breaking again; it doesn't catch the user never seeing why they're on Custom.``