DEV Community: Bromena

Retraining on a schedule is easy. Promoting on a schedule is how models quietly get worse

Bromena — Sat, 14 Nov 2026 18:47:59 +0000

A lot of teams get to the same place with their first production model: a cron job retrains it every week, and every week the new version replaces the old one. It feels like automation. It's actually a coin flip with a deploy button.

Retraining is the safe part. Promotion is where a model gets worse without anyone noticing. Here's the small set of checks we'd put between "a new model exists" and "users are getting its predictions", with code you can lift.

1. Promote against a frozen eval set, not against last week

The new model should earn its place by beating the current one on the same evaluation set. If the eval set changed, the comparison means nothing, and the gate should say so instead of guessing.

def promote(champion: dict, challenger: dict, min_gain: float = 0.005, max_latency_ms: float = 120) -> tuple[bool, str]:
    """Promote only if the challenger beats the champion on the same frozen eval set and stays inside the latency budget."""
    if challenger["eval_set"] != champion["eval_set"]:
        return False, "different eval sets: not comparable"
    gain = challenger["auc"] - champion["auc"]
    if gain < min_gain:
        return False, f"gain {gain:+.4f} below {min_gain}"
    if challenger["p95_latency_ms"] > max_latency_ms:
        return False, f"p95 latency {challenger['p95_latency_ms']} ms over budget"
    return True, f"promote: gain {gain:+.4f}"

champ = {"eval_set": "eval-2026-10", "auc": 0.861, "p95_latency_ms": 85}
print(promote(champ, {"eval_set": "eval-2026-10", "auc": 0.864, "p95_latency_ms": 90}))
print(promote(champ, {"eval_set": "eval-2026-10", "auc": 0.874, "p95_latency_ms": 95}))
print(promote(champ, {"eval_set": "eval-2026-11", "auc": 0.902, "p95_latency_ms": 80}))

(False, 'gain +0.0030 below 0.005')
(True, 'promote: gain +0.0130')
(False, 'different eval sets: not comparable')

The third case is the one that bites. A challenger scored on a newer, easier eval set looks like a big win. It isn't evidence of anything.

Two choices matter more than the code:

min_gain is a product decision. A gain smaller than your metric's run-to-run noise is noise. Measure that noise once (retrain the same model with different seeds) and set the threshold above it.
Latency is part of "better". A model that's 1% more accurate and blows the p95 budget is a regression for the people waiting on it.

2. Watch the inputs, and agree on thresholds before anything pages anyone

Drift alerts are only useful if someone agreed in advance what counts as drift and who gets woken up. A common, simple input check is the Population Stability Index (PSI) per feature:

import numpy as np

def psi(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float:
    """Population Stability Index between a reference sample and a live sample of one feature."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

rng = np.random.default_rng(7)
ref = rng.normal(100, 15, 20_000)
print(round(psi(ref, rng.normal(100, 15, 5_000)), 3))   # same distribution
print(round(psi(ref, rng.normal(112, 18, 5_000)), 3))   # shifted after a market change

0.002
0.542

A widely used rule of thumb reads PSI under 0.1 as stable and over 0.25 as a real shift. Treat those as a starting point, not a law: tune them per feature, write them down, and name the owner of each model who receives the alert. An alert that goes to a shared channel goes to nobody.

3. Ship with traffic you can take back

Even a model that passed the gate can fail on live traffic. Two patterns keep that cheap:

Shadow: the new model scores real requests, but its predictions aren't used. You compare offline.
Canary: a small slice of traffic gets the new model, with an automatic stop if error rates jump.

Both need one release path for every model, usually through a model registry, so "which version is live?" always has a single answer.

4. Rehearse the rollback before you need it

Every release path needs a way back to the last good version, and that way back needs to have been run at least once before a real model depends on it. A rollback nobody has tried is a hope: the old artifact got garbage-collected, the feature schema moved on, or the one person who knew the command is on holiday.

A practical version: in the same week you set up promotion, deploy a deliberately worse model to staging, let the canary stop fire, and roll back. Time it. Write the steps down.

5. Size the platform to the models you have

You don't need a full Kubeflow install to do any of this. With a handful of models, MLflow (or any registry) plus a scheduled pipeline usually covers it. Start by automating the riskiest manual step in how your models ship today, and add pieces only when a release actually needs them.

A checklist to steal

[ ] Retraining can run on a schedule; promotion runs through a gate.
[ ] The gate compares champion and challenger on the same frozen eval set, with a minimum gain above measured noise and a latency budget.
[ ] Drift thresholds are written down per feature, with one named owner per model.
[ ] New versions go out as shadow or canary, with an automatic stop.
[ ] The rollback has been rehearsed and timed.

How we scope this kind of platform work, what our MLOps engineers are screened on, and the stack that comes up most often are at bromena.com/ai/mlops.

— Bromena team

Seven design scenarios for interviewing AI and data engineers, and what a pass looks like

Bromena — Fri, 13 Nov 2026 13:30:09 +0000

A generic coding test tells you almost nothing about an AI engineer. A strong backend developer can still get evaluation design wrong. A strong data scientist can still struggle to serve a model under load. The mistakes differ by role, so the interview has to differ too.

These are the design scenarios we use, one per role, with the evidence a candidate has to show to pass. Steal them for your own loop.

Machine learning engineer

Scenario: serve a ranking or forecasting model behind an API with a fixed latency budget, then plan how it gets retrained.

A pass shows:

an offline metric tied to the business outcome, not just accuracy;
a test set with no leakage from training;
a serving plan with its latency and cost stated.

Generative AI engineer

Scenario: question answering over a large set of internal documents, with a ceiling on cost per request.

A pass shows:

an evaluation set built before any prompt tuning;
retrieval measured on its own, separate from generation;
a cost per request they can estimate and defend.

AI agent developer

Scenario: an agent that opens and updates tickets through a ticketing API, where one tool times out halfway through a task.

A pass shows:

timeouts and retries for each tool;
actions that are safe to run twice;
a log of every step;
human approval where a mistake costs money.

The timeout is the whole point of this one. Most demo agents have never met a tool that fails.

Computer vision engineer

Scenario: a defect detector that has to keep up with a production line camera on an edge device.

A pass shows: a labeling guide with a check on label disagreement, metrics for each class, and latency measured on the device itself, not on a laptop GPU.

NLP engineer

Scenario: pull fields from scanned contracts with inconsistent layouts and OCR errors.

A pass shows: error analysis field by field, a simple baseline before any large model, and a plan for documents the model can't read.

MLOps engineer

Scenario: release a new model version through a canary, with an automatic rollback trigger.

A pass shows: versioned models and data, a rollout that stops itself on a bad metric, drift alerts with an owner, and a rollback they have actually run.

Data engineer

Scenario: design the tables a training pipeline reads from, with late-arriving events and an upstream schema change.

A pass shows: a data model that keeps history, quality checks that stop a bad batch before training, and backfills that can rerun without duplicates.

Two rules that make the scenarios work

Change a constraint halfway through. Halve the budget or tighten the latency target and watch how the design holds up. Candidates who memorised an architecture stall; candidates who understand it adjust.
Score each stage on its own. Someone who aces the coding screen and struggles in the design review shouldn't average out into a pass. Write down the evidence behind every score, so the hiring manager can read it later.

We run these as part of a four-stage evaluation at Bromena, and the client still runs its own interviews afterwards. If you use a different scenario for one of these roles, we'd like to hear what it catches that ours misses.

— Bromena team

How to compare a nearshore engineer to a US hire using only public BLS data

Bromena — Thu, 12 Nov 2026 19:17:45 +0000

US hiring teams often compare a nearshore rate with a US salary and stop there. That comparison is incomplete: a salaried employee also costs benefits, paid leave and the employer's payroll taxes. The US government publishes those pieces. You can rebuild a fully loaded US cost from them, then compare it with an all-in hourly rate, without relying on a vendor's private spreadsheet.

This walk-through uses only official Bureau of Labor Statistics (BLS) releases, plus a worked example with the all-in ranges Bromena publishes for AI and data roles. Every arithmetic step is below, so you can swap in your own salary or rate and rerun it.

1. Start with a US wage that actually has a BLS category

BLS does not publish a national wage series named "machine learning engineer" or "data engineer". The closest occupation with national data is software developers.

From the Occupational Outlook Handbook, pay for May 2025:

Median annual wage: $135,980
Highest 10 percent (the 90th percentile): more than $214,670

Source: BLS, Occupational Outlook Handbook, Software Developers.

If your team pays above those figures, the gap you calculate later will be wider. If you have a wage for the exact role you are filling, use that instead of the national median.

2. Turn the wage into a fully loaded employer cost

Wages are not what a US employer spends. The BLS Employer Costs for Employee Compensation (ECEC) series splits private-industry compensation into wages and benefits.

In June 2026, for private industry workers:

Total compensation averaged $46.89 per hour worked
Wages and salaries averaged $32.82, or 70.0 percent of that total
Benefit costs averaged $14.07, or the remaining 30.0 percent

The release date is 9 September 2026 (USDL-26-1494). Source: BLS, Employer Costs for Employee Compensation, June 2026.

Dividing a wage by 0.70 adds the benefits layer:

Median: $135,980 / 0.70 = $194,257
90th percentile: $214,670 / 0.70 = $306,671

Two caveats, both of which shrink the US figure relative to a real hire:

ECEC's 70.0 percent is an average across private industry, not a software-only ratio. It is the official number you can cite without mixing surveys.
This still leaves out recruiting fees and the months a role stays empty. Those costs are real, but they have no single public series that belongs in the same table, so they stay out of the comparison.

3. Price the same year with an all-in hourly rate

An all-in nearshore rate already includes the engineer's pay and the vendor's fee. To put it on an annual basis, multiply by 2,080 hours, the full-time year BLS uses for annual wages.

Worked example, using ranges Bromena reviewed in October 2026 for AI and data roles (the low end of each range is mid-level; the high end is lead):

Comparison	US hire, fully loaded	Nearshore, 2,080 hours	Difference
Roles other than data engineer, low end ($65/hour)	$194,257	$135,200	30% less
Same roles, high end ($105/hour)	$306,671	$218,400	29% less
Data engineer, low end ($58/hour)	$194,257	$120,640	38% less
Data engineer, high end ($88/hour)	$306,671	$183,040	40% less

The arithmetic, so you can check it:

$65 × 2,080 = $135,200; ($194,257 − $135,200) / $194,257 ≈ 30%
$105 × 2,080 = $218,400; ($306,671 − $218,400) / $306,671 ≈ 29%
$58 × 2,080 = $120,640; ($194,257 − $120,640) / $194,257 ≈ 38%
$88 × 2,080 = $183,040; ($306,671 − $183,040) / $306,671 ≈ 40%

Cloud, GPUs and model API usage are not in either column. They should run on the customer's accounts in both models.

4. What this does not claim

It does not say every nearshore engineer costs 29 to 40 percent less. That range is the result of this method, with these inputs, for these published rates.
It does not treat "software developers" as the same job as an ML or data engineer. It treats it as the closest official wage category, and says so.
It does not include output, seniority mix, or the cost of a bad hire. Those belong in a hiring decision, not in this table.

If you want to rerun the same steps with a different US wage or a quoted rate, Bromena publishes a calculator that follows this method and lists the same sources.

Sources

U.S. Bureau of Labor Statistics, Occupational Outlook Handbook, Software Developers, pay for May 2025: https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm
U.S. Bureau of Labor Statistics, Employer Costs for Employee Compensation, June 2026, released 9 September 2026 (USDL-26-1494): https://www.bls.gov/news.release/ecec.nr0.htm

— Bromena team