DEV Community: Kate Steele

Isolation Forest Anomaly Detection on Login Logs in Python: Thresholds, Explanations, Drift

Kate Steele — Tue, 03 Nov 2026 15:28:09 +0000

Stolen passwords just lost first place. Verizon’s 2026 Data Breach Investigations Report says 31% of breaches now start with software vulnerabilities, “beating stolen passwords as the top way attackers get in.” The same report still lists stolen credentials among the most frequent causes of breaches, and those attacks leave a trail in one boring table: the authentication log.

The pitch for machine learning here is familiar. Signature rules only catch what someone already wrote a rule for, and an unsupervised model learns “normal” and flags the rest. I wanted to see what that looks like on login data with real numbers, so I built the whole loop with Isolation Forest anomaly detection in Python and scikit-learn: a synthetic auth log with labeled attacks, per-user features, training, threshold selection, evaluation against a hand-written rule baseline, per-alert explanations, and a drift scenario that forces a retrain.

Short version of the results: the rules won on precision (0.906 vs 0.426), the forest won on recall (0.894 vs 0.780), and the forest was the only one that caught any of the quietest attack. Every number below comes from a real run of the anomaly detection Python code in this post.

What we’ll build:

A log generator with a fixed seed and four labeled attack types
Per-user, per-hour features that only look backward in time
An Isolation Forest and three ways to set its threshold
Precision, recall and PR-AUC, plus a comparison with SIEM-style rules
“Why did this fire?” explanations for single alerts
A drift week that breaks the threshold, and two ways to fix it

Stack: Python 3.13.18, scikit-learn 1.9.1, pandas 3.0.5, numpy 2.5.3, pytest 9.1.1. The full experiment runs in well under a minute on 2 CPUs.

What the attacks look like in an auth log

Before choosing among anomaly detection algorithms, it helps to write down what each attack leaves behind. These are the four I generate, mapped to MITRE ATT&CK where it applies:

Attack	What the log shows	Feature that should notice
Credential stuffing (T1110.004)	A few datacenter IPs try 1-2 passwords on many accounts within an hour	`ip_fanout`, `new_device`, `n_fail`
Brute force, then takeover	10-39 failures on one account, then a success from a new device abroad	`n_fail`, `fail_ratio`, `new_country`
Impossible travel	A clean login from another continent shortly after the real user’s login	`geo_speed_kmh`, `new_device`
Low-and-slow stuffing	One failed try per account from a residential IP in the victim’s own city	almost nothing, by design

OWASP describes credential stuffing as “the automated injection of stolen username and password pairs” and notes it works because people reuse passwords. MITRE’s detection guidance for T1110.004 starts where you’d expect: rapid failures across different users from one IP. Identity providers ship their own versions of the travel signal: Microsoft Entra ID Protection, for example, has both “atypical travel” and “impossible travel” risk detections. The fourth attack is there to keep the experiment honest. A careful attacker spreads attempts across residential proxies and stays under every per-IP and per-account threshold.

Pipeline from raw login events to hourly per-user windows, ten features, Isolation Forest score, alert-budget threshold and an explained alert, with a retraining loop

Step 1. Generate auth logs with labeled attacks

Real labeled login data is hard to get and impossible to share, so the generator does three things on purpose:

Normal users have habits. Each of 400 users has a home city, 1-3 home IPs, 1-2 devices, a typical login hour and a login rate. About 8% of logins come from a mobile IP and 4% start with a mistyped password.
Normal users also do odd things. Some take business trips, buy new phones or forget their password and fail 4-8 times in a row. These are the false positives any detector has to live with.
The training period is not clean. Days 1-21 contain a quarter-scale copy of the attacks, unlabeled, because your historical logs aren’t clean either.

The attack injector is the interesting part:

def attacks(rng, users, day_from, day_to, scale=1.0):
    rows = []
    n = lambda k: max(1, int(round(k * scale)))
    # 1) Credential stuffing: a few datacenter IPs, many users, 1-2 tries each, rare success.
    for _ in range(n(2)):
        t0 = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=float(rng.uniform(0, 24)))
        ips = [_ip(rng, "45.95") for _ in range(3)]
        for u in rng.choice(users, size=n(40), replace=False):
            ts = t0 + pd.Timedelta(minutes=float(rng.uniform(0, 50)))
            for _ in range(int(rng.integers(1, 3))):
                ok = rng.random() < 0.05
                rows.append(_event(ts, u["user"], rng.choice(ips), "Frankfurt", "bot-" + str(rng.integers(1e6)),
                                   16509, ok, "credential_stuffing"))
                if ok:
                    break
    # 2) Brute force -> takeover: many failures on one account, then success from a new device abroad.
    for u in rng.choice(users, size=n(6), replace=False):
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=float(rng.uniform(0, 24)))
        city, ip = rng.choice(ATTACK_CITIES), _ip(rng, "91.200")
        for k in range(int(rng.integers(10, 40))):
            rows.append(_event(ts + pd.Timedelta(seconds=15 * k), u["user"], ip, city, "atk-dev", 9123, False, "brute_force"))
        rows.append(_event(ts + pd.Timedelta(minutes=12), u["user"], ip, city, "atk-dev", 9123, True, "brute_force"))
    # 3) Impossible travel: phished password used from another continent, no failures at all.
    for u in rng.choice(users, size=n(10), replace=False):
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=(u["peak_hour"] + 1.5) % 24)
        rows.append(_event(ts - pd.Timedelta(hours=1), u["user"], u["ips"][0], u["city"], u["devices"][0], u["asn"], True, "normal"))
        rows.append(_event(ts, u["user"], _ip(rng, "102.89"), rng.choice(ATTACK_CITIES), "atk-" + u["user"],
                           36873, True, "impossible_travel"))
    # 4) Low-and-slow stuffing via residential proxies in the victim's own city: one try per user.
    for u in rng.choice(users, size=n(25), replace=False):
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=(u["peak_hour"] + rng.normal(0, 2)) % 24)
        rows.append(_event(ts, u["user"], _ip(rng, "73." + str(rng.integers(50, 99))), u["city"], u["devices"][0],
                           u["asn"], False, "low_and_slow"))
    return rows

«Full generator (users, normal traffic, benign oddities)»:

def make_users(rng, n_users):
    users = []
    for u in range(n_users):
        city = rng.choice(HOME_CITIES, p=[.22, .18, .14, .14, .1, .12, .1])
        users.append({
            "user": f"u{u:04d}",
            "city": city,
            "ips": [_ip(rng, f"73.{rng.integers(0, 50)}") for _ in range(rng.integers(1, 4))],
            "devices": [f"d{u:04d}{k}" for k in range(rng.integers(1, 3))],
            "asn": int(rng.choice([7922, 701, 20115, 5650, 2856, 3320])),
            "peak_hour": float(rng.normal(14, 2.5)),   # UTC
            "rate": float(rng.uniform(0.8, 5.0)),      # logins per weekday
        })
    return users

def normal_traffic(rng, users, days, day0=0):
    rows = []
    for u in users:
        for d in range(day0, day0 + days):
            weekend = (START + pd.Timedelta(days=d)).dayofweek >= 5
            for _ in range(rng.poisson(u["rate"] * (0.3 if weekend else 1.0))):
                hour = (u["peak_hour"] + rng.normal(0, 2.0)) % 24
                ts = START + pd.Timedelta(days=d, hours=hour)
                ip = rng.choice(u["ips"]) if rng.random() > 0.08 else _ip(rng, "172.56")  # mobile
                dev = rng.choice(u["devices"])
                if rng.random() < 0.04:  # typo, then success
                    rows.append(_event(ts, u["user"], ip, u["city"], dev, u["asn"], False, "normal"))
                    ts += pd.Timedelta(seconds=int(rng.integers(5, 60)))
                rows.append(_event(ts, u["user"], ip, u["city"], dev, u["asn"], True, "normal"))
    return rows

def benign_oddities(rng, users, day_from, day_to):
    """Legit behavior that *looks* suspicious: trips, new phones, forgotten passwords."""
    rows = []
    for u in rng.choice(users, size=max(1, len(users) // 25), replace=False):   # business trips
        d = int(rng.integers(day_from, day_to))
        dest = rng.choice([c for c in HOME_CITIES if c != u["city"]] + ["Singapore", "Warsaw"])
        for k in range(3):
            ts = START + pd.Timedelta(days=d + k, hours=float(rng.uniform(7, 20)))
            rows.append(_event(ts, u["user"], _ip(rng, "185.20"), dest, u["devices"][0], 9009, True, "normal", "trip"))
    for u in rng.choice(users, size=max(1, len(users) // 30), replace=False):   # new phone
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=u["peak_hour"] % 24)
        rows.append(_event(ts, u["user"], u["ips"][0], u["city"], f"{u['user']}-new", u["asn"], True, "normal", "new_phone"))
    for u in rng.choice(users, size=max(1, len(users) // 40), replace=False):   # forgot password
        ts = START + pd.Timedelta(days=int(rng.integers(day_from, day_to)), hours=u["peak_hour"] % 24)
        for k in range(int(rng.integers(4, 9))):
            rows.append(_event(ts + pd.Timedelta(seconds=20 * k), u["user"], u["ips"][0], u["city"],
                               u["devices"][0], u["asn"], False, "normal", "forgot_password"))
    return rows

def generate(seed=42, n_users=400, train_days=21, test_days=7):
    """Days [0, train_days) are 'history' (with a little unlabeled attack noise), the rest is test."""
    rng = np.random.default_rng(seed)
    users = make_users(rng, n_users)
    total = train_days + test_days
    rows = normal_traffic(rng, users, total)
    rows += benign_oddities(rng, users, 0, total)
    rows += attacks(rng, users, 3, train_days, scale=0.25)        # contamination nobody labeled
    rows += attacks(rng, users, train_days, total)
    df = pd.DataFrame(rows).sort_values("ts", kind="stable").reset_index(drop=True)
    df["is_attack"] = df["label"] != "normal"
    return df, users
python 3.13.18, scikit-learn 1.9.1, pandas 3.0.5, numpy 2.5.3
events: 27,481  users: 400
train windows (days 1-21): 16,023  unlabeled attack windows hiding inside: 20
test windows  (days 22-28): 5,403
label
normal                 5280
credential_stuffing      80
low_and_slow             25
impossible_travel        10
brute_force               8

The test week has 123 attack windows out of 5,403, a base rate of 2.3%. Keep that number in mind: a detector that flags everything gets precision 0.023.

Step 2. Feature engineering per user and time window

This isn’t classic time series anomaly detection on a single metric. Each row is one user in one clock hour, and every feature compares that hour with what the user did before it. Getting the “before” right matters more than the model choice.

def build_windows(events: pd.DataFrame, history: dict | None = None):
    """Aggregate events into (user, 1h) windows. Returns (windows, history) so the
    per-user state can be carried into the next batch (train -> test -> drift)."""
    ev = events.copy()
    ev["window"] = ev["ts"].dt.floor("1h")
    # Network-side feature: how many distinct users each source IP touched in that hour.
    fanout = ev.groupby(["ip", "window"])["user"].nunique().rename("ip_users")
    ev = ev.join(fanout, on=["ip", "window"])

    ev = ev.sort_values(["window", "user", "ts"], kind="stable")
    hist = history if history is not None else {}
    out = []
    recs = ev.to_dict("records")
    i = 0
    while i < len(recs):
        j = i
        while j < len(recs) and recs[j]["window"] == recs[i]["window"] and recs[j]["user"] == recs[i]["user"]:
            j += 1
        g, first = recs[i:j], recs[i]
        window, user = first["window"], first["user"]
        h = hist.setdefault(user, {"devices": set(), "countries": set(), "asns": set(),
                                   "hours": np.ones(24), "last": None})
        # Geo-velocity for every event against the previous *successful* login, including
        # successes earlier in the same window. My first version checked only the first event
        # of the window and read 0 km/h when the attacker logged in right after the real user.
        speed, prev = 0.0, h["last"]
        for e in g:
            if prev is not None:
                km = haversine_km(prev[1], prev[2], e["lat"], e["lon"])
                hours = max((e["ts"] - prev[0]).total_seconds() / 3600, 0.25)  # 15-min floor for IP-geo noise
                speed = max(speed, km / hours)
            if e["success"]:
                prev = (e["ts"], e["lat"], e["lon"])
        fails = [not e["success"] for e in g]
        attack = [e["label"] for e in g if e["is_attack"]]
        seen = lambda key, known: int(bool(known) and any(e[key] not in known for e in g))
        out.append({
            "window": window, "user": user,
            "n_attempts": len(g),
            "n_fail": sum(fails),
            "fail_ratio": sum(fails) / len(g),
            "n_ips": len({e["ip"] for e in g}),
            "new_device": seen("device", h["devices"]),
            "new_country": seen("country", h["countries"]),
            "new_asn": seen("asn", h["asns"]),
            "geo_speed_kmh": speed,
            "hour_rarity": float(1 - h["hours"][window.hour] / h["hours"].sum()),
            "ip_fanout": max(e["ip_users"] for e in g),
            "is_attack": bool(attack),
            "label": attack[0] if attack else "normal",
            "note": next((e["note"] for e in g if e["note"]), ""),
        })
        # Update state only from *successful* logins: failed attempts must not teach the
        # model that the attacker's device or country is "known".
        ok = [e for e in g if e["success"]]
        for e in ok:
            h["devices"].add(e["device"]); h["countries"].add(e["country"]); h["asns"].add(e["asn"])
        h["hours"][window.hour] += len(ok)
        if ok:
            h["last"] = (ok[-1]["ts"], ok[-1]["lat"], ok[-1]["lon"])
        i = j
    return pd.DataFrame(out), hist

Three decisions in there are worth calling out:

State updates only from successful logins. If failed attempts updated “known devices,” a brute-force attacker would teach the model their own device during the failures and look familiar by the time they succeed. There’s a test for this.
Geo-velocity is checked for every event in the window. My first version only checked the first event of each hour. When the attacker logged in right after the real user in the same hour, it compared the user with themselves and returned 0 km/h. The fixed version reports 21,034 km/h for that window.
ip_fanout is the one network-level feature. It counts how many distinct users a source IP touched in that hour. Network anomaly detection usually starts from exactly this kind of cross-account view, and it’s what MITRE recommends for stuffing.

Heavy-tailed columns go through log1p before training:

def to_matrix(windows: pd.DataFrame) -> np.ndarray:
    """Heavy-tailed columns go through log1p (Isolation Forest draws split values uniformly
    between a node's min and max). In this dataset it barely moved PR-AUC; see gotchas."""
    X = windows[FEATURES].astype(float).copy()
    for c in ["n_attempts", "n_fail", "n_ips", "geo_speed_kmh", "ip_fanout"]:
        X[c] = np.log1p(X[c])
    return X.to_numpy()

Step 3. Train the Isolation Forest

The idea behind Isolation Forest, from Liu, Ting and Zhou’s 2008 paper, is that anomalies are “few and different.” Each tree picks a random feature and a random split value between that feature’s min and max, and keeps splitting. Rare points get isolated after a few splits, so their average path length across the forest is short. The scikit-learn docs put it as “noticeably shorter paths for anomalies.” The paper used 256-point subsamples and 100 trees. I use 300 trees for steadier scores; max_samples=256 matches scikit-learn’s "auto" setting at this data size.

def train(X_train, contamination="auto", seed=7):
    return IsolationForest(n_estimators=300, max_samples=256, contamination=contamination,
                           random_state=seed).fit(X_train)

def anomaly_score(model, X):
    """Higher = more anomalous. score_samples() returns the opposite sign."""
    return -model.score_samples(X)

One sign convention trips people up: score_samples() returns the opposite of the paper’s score, and lower means more abnormal. I flip it once so that “higher = worse” everywhere else.

Step 4. Pick a threshold: contamination vs. score threshold

This is where most Isolation Forest tutorials stop at contamination=0.01 and move on. Here are four options on the same trained model:

contamination='auto'  offset_ = -0.5 | alerts= 1080  precision=0.114  recall=1.000  PR-AUC=0.873
contamination=0.01    offset_ = -0.695 | 1st percentile of train score_samples = -0.695
contamination=0.01    alerts=  151  precision=0.669  recall=0.821  PR-AUC=0.873
budget  5/day -> score>=0.705   alerts=  132  precision=0.750  recall=0.805  PR-AUC=0.873
budget 10/day -> score>=0.691   alerts=  162  precision=0.623  recall=0.821  PR-AUC=0.873
budget 20/day -> score>=0.652   alerts=  258  precision=0.426  recall=0.894  PR-AUC=0.873
budget 40/day -> score>=0.617   alerts=  352  precision=0.321  recall=0.919  PR-AUC=0.873

What the output says:

contamination="auto" is not “let the model decide.” The scikit-learn docs say the offset is fixed at -0.5, as in the original paper. On this data that means 1,080 alerts in a week, a 20% alert rate. The paper itself warns that when every score sits near 0.5, there may be no clear anomalies, and with mostly discrete, low-variance features many normal windows score near that line.
contamination=0.01 is just a quantile of the training scores. The fitted offset_ (-0.695) equals the 1st percentile of score_samples on the training set, to three decimals. It’s not a statement about how many attacks exist. It’s a statement about how many alerts you’ll get on data like your training data.
So set it from your alert budget. If the SOC can triage 20 login alerts a day, compute the cut-off that produced 20/day on the training period and use that score threshold directly:

def threshold_for_budget(scores, alerts_per_day, n_days):
    """Score cut-off that yields a fixed number of alerts on a reference period."""
    k = int(alerts_per_day * n_days)
    return np.sort(scores)[::-1][k - 1]

At 20/day the model produced 258 alerts in the test week: about 21 false positives per day, which matches the budget, plus the attacks on top. PR-AUC doesn’t depend on any threshold, which is why it’s identical in every line.

Step 5. Evaluate with precision, recall and PR-AUC

With 2.3% positives, ROC-AUC can look great while most alerts are noise, so I report precision, recall and PR-AUC. scikit-learn’s average_precision_score computes it as a recall-weighted mean of precisions, which “is not interpolated” and avoids the optimism of the trapezoidal rule. The model scores 0.873 against a no-skill baseline of 0.023.

Precision-recall curve of the Isolation Forest on the test week, with operating points for contamination auto, contamination 0.01, the 20-alerts-per-day budget, and the rule baseline sitting above the curve

Step 6. Compare with a rule baseline

An anomaly detection machine learning model should beat what a security engineer writes in ten minutes. Here’s that ten-minute version:

def rule_baseline(w: pd.DataFrame) -> np.ndarray:
    """What a SIEM correlation rule would typically do."""
    return ((w["n_fail"] >= 5)
            | (w["geo_speed_kmh"] > 1000)
            | ((w["new_country"] == 1) & (w["new_device"] == 1))
            | (w["ip_fanout"] >= 10)).to_numpy()

Five failures in an hour is an alert, not a lockout. For comparison, NIST SP 800-63B only requires the verifier to cap consecutive failed attempts on an account at no more than 100.

rules             alerts=  106  precision=0.906  recall=0.780
iforest@20/day    alerts=  258  precision=0.426  recall=0.894
rules OR iforest  alerts=  267  precision=0.412  recall=0.894

attack windows caught, by type:
                     rules  iforest  total
label                                     
brute_force              8        8      8
credential_stuffing     78       80     80
impossible_travel       10       10     10
low_and_slow             0       12     25

false positives of iforest@20/day by cause:
note
plain normal    144
new_phone         3
trip              1
  of the plain-normal ones: 72% have 3+ logins in the hour, 81% contain a mistyped password
false positives of rules by cause: {'plain normal': 6, 'trip': 4}
  rule false positives where the user's previous successful login was the attacker's: 4

The rules win on precision by a wide margin: 96 of their 106 alerts are real. They catch every brute-force and impossible-travel window and 78 of 80 stuffing windows. The forest catches all three of those types too, but pays for it with 148 false positives. Its one real advantage is the bottom row: it caught 12 of the 25 low-and-slow attempts. The rules caught none.

The false positives are revealing. The forest’s top complaint is “busy hour with a typo”: 72% of its plain-normal false positives have 3+ logins in one hour and 81% include a mistyped password. That’s rare, so the forest flags it, but rare isn’t the same as malicious. Four of the rules’ ten false positives are more interesting: the victim’s own next login after a takeover. Their “last known location” was now the attacker’s, so coming home looked like impossible travel. Under the labels these count as false positives, but an analyst would want to see them.

Two caveats before anyone quotes “rules beat ML.” First, I wrote the rules and the generator, so the rules know exactly what the attacks look like. That’s the best case for rules, and real attackers don’t send you their source code. Second, rules OR iforest adds nothing here, because every attack the rules catch is already in the forest’s top 258. A more useful hybrid is rules for paging and the forest’s top-N as a daily review queue for what the rules can’t express.

Matrix of attack types against detectors: rules and Isolation Forest catch counts per attack type, plus the top false-positive patterns for each

Step 7. Explain an alert

An alert that says “score 0.784” gets ignored. The cheapest model-agnostic explanation I know is ablation: reset one feature at a time to its training median and measure how much the anomaly score drops.

def explain(model, X_train, x, top=3):
    """Ablation: reset one feature at a time to its training median and see how much the
    anomaly score drops. Big drop = this feature is what made the window look odd."""
    med = np.median(X_train, axis=0)
    base = anomaly_score(model, x[None, :])[0]
    probes = np.repeat(x[None, :], len(FEATURES), axis=0)
    probes[np.arange(len(FEATURES)), np.arange(len(FEATURES))] = med
    drops = base - anomaly_score(model, probes)
    order = np.argsort(drops)[::-1][:top]
    return base, [(FEATURES[i], float(drops[i])) for i in order if drops[i] > 0]
u0090 09-23 14:00 label=impossible_travel score=0.784 alert=yes
    new_device     value=      1.0  score drop if typical: 0.041
    geo_speed_kmh  value=  21033.8  score drop if typical: 0.032
    n_ips          value=      3.0  score drop if typical: 0.028
u0336 09-22 05:00 label=credential_stuffing score=0.804 alert=yes
    new_device     value=      1.0  score drop if typical: 0.027
    n_attempts     value=      4.0  score drop if typical: 0.019
    n_fail         value=      3.0  score drop if typical: 0.017
u0214 09-22 12:00 label=normal/trip score=0.766 alert=yes
    n_attempts     value=      4.0  score drop if typical: 0.085
    geo_speed_kmh  value=  25540.6  score drop if typical: 0.048
    n_ips          value=      3.0  score drop if typical: 0.039

The impossible-travel alert reads well: new device plus 21,000 km/h. The stuffing alert points at the new device and the failures. Notice what’s missing, though: ip_fanout never shows up, even though it’s the strongest stuffing signal. The third alert is a business trip, and its explanation shows the same impossible-travel pattern as a real attack, because a hotel IP on another continent is geographically odd.

Single-feature ablation has a known blind spot: when two features are odd together, resetting either one alone may barely move the score. In the test suite, a synthetic 30-failure window got its explanation attributed to n_attempts rather than n_fail, because both were extreme together. Treat explanations as hints for the analyst, not proofs.

Step 8. Drift and retraining

Days 29-35 simulate a normal IT change week. Two things ship at once: a corporate VPN starts sending 40% of users through two gateways in Frankfurt, and a new SSO client silently re-authenticates half the users twice after every login, about 20 minutes apart. Days 29-30 have no attacks, so the model has a chance to re-baseline. Days 31-35 get a full set of attacks.

PSI train -> IT-change week (>0.25 usually = significant shift):
    ip_fanout      2.64
    n_attempts     0.57
    hour_rarity    0.30
    new_asn        0.11

old model, old threshold, days 31-35: alerts=  367  precision=0.313  recall=0.943  PR-AUC=0.868
    alerts/day: 73  (budget was 20)
retrained (days 10-30), new threshold:   alerts=  269  precision=0.416  recall=0.918  PR-AUC=0.898
IPs with >50 distinct users in the IT-change week: ['198.51.100.10', '198.51.100.11']
old model + egress allowlist feature:   alerts=  356  precision=0.323  recall=0.943  PR-AUC=0.870

attack windows caught in days 31-35, by type:
                     old  retrained  allowlist  total
label                                                
brute_force            7          7          7      7
credential_stuffing   80         80         80     80
impossible_travel     10         10         10     10
low_and_slow          18         15         18     25

alerts per day, old model: [77, 33, 26, 28, 8, 51, 35, 87, 66, 74, 82, 30, 36, 145]
attack windows per day:    [52, 5, 3, 7, 4, 44, 8, 0, 0, 6, 6, 13, 24, 73]
false positives/day: test week 21.1 | days 31-35 old 50.4 | retrained 31.4 | allowlist 48.2

Population Stability Index screams about ip_fanout (2.64): two IPs now serve over 50 users each. A common rule of thumb treats PSI above 0.25 as a significant shift, though Yurdakul’s 2018 thesis notes these cut-offs are conventions with no stated error rates. But the feature with the loudest PSI isn’t what broke the alerts. Fixing it with an egress allowlist barely helps (50.4 → 48.2 false positives/day). My reading: each split picks one of ten features at random, so one extreme feature moves the average path length less than you might expect. The real damage comes from n_attempts: background SSO re-auth makes a normal hour look like a busy hour, and “busy hour” was already the forest’s main false-positive pattern.

Retraining on a rolling 21-day window (days 10-30, so it includes two days of the new normal) brings false positives down to 31.4/day and PR-AUC up from 0.868 to 0.898. It also costs something: it catches 15 low-and-slow attempts instead of 18. That’s the trade-off with every unsupervised retrain. The model learns whatever the window contains, including attacks nobody labeled.

Two smaller findings. hour_rarity drifts on its own (PSI 0.30) because per-user hour histograms keep sharpening as history grows: a cold-start effect, not a behavior change. And the 77 alerts on day 22 aren’t a model problem: that day contains a full stuffing campaign (52 attack windows). An alerts-per-day chart without attack counts next to it would have sent me looking for a bug.

Alerts per day over two weeks, with the IT change marked: false positives per day go from 21.1 in the test week to 50.4 after the change, and 31.4 after retraining

Where the model gets it wrong

Collecting the failures in one place:

Low-and-slow stuffing: 12 of 25 caught. One failed login from the right city on the right device during working hours is indistinguishable from a typo. Only cross-account features (many accounts, same password-failure pattern, same hour) can see it, and that’s a different model.
Busy users: most false positives are legitimate heavy users who mistyped a password.
Travelers: a business trip through a foreign hotel IP looks exactly like impossible travel.
The victim’s return: after a takeover, the real user’s next login looks like an anomaly. That’s useful, but it breaks naive labeling.
Silent masking: 20 attack windows sit in the training data. The paper calls this masking: a dense cluster of anomalies is harder to isolate. Real logs will have more of it than my 20.

Gotchas I hit

Window-boundary bugs in features. The geo-velocity bug above raised no error, and PR-AUC only moved from 0.870 to 0.873 after the fix, because new_device and new_country carried those windows anyway. Only the explanation exposed it. Write a unit test for a two-logins-in-one-hour case.
log1p didn’t matter here. Isolation Forest splits uniformly between min and max, so I expected heavy tails to hurt. PR-AUC with log transforms was 0.873; with raw counts, 0.870. Measure before you add preprocessing.
The “obvious” network feature didn’t help the score. Dropping ip_fanout moved PR-AUC from 0.873 to 0.875. The stuffing windows were already isolated by new_device and failures. For a single strong signal, a rule is still a better tool.
Random seeds move PR-AUC by about 0.01. Five random_state values gave 0.866 to 0.878. Don’t celebrate a 0.005 improvement from a feature change without checking several seeds.

PR-AUC with log1p on counts: 0.873   raw counts: 0.870
PR-AUC over 5 random_state values: 0.878 0.869 0.871 0.870 0.870
PR-AUC without the network feature ip_fanout: 0.875

Tests that pin the behavior

The suite checks the claims this post depends on: deterministic generation, causal features, failed attempts not whitelisting the attacker’s device, in-window impossible travel, offset_ equal to a training quantile, and explanations pointing at the right feature family.

def test_features_are_causal(data):
    """Appending future events must not change features of earlier windows."""
    cut = START + pd.Timedelta(days=8)
    w_full, _ = build_windows(data)
    w_past, _ = build_windows(data[data["ts"] < cut])
    early = w_full[w_full["window"] < cut].reset_index(drop=True)
    cols = [f for f in FEATURES if f != "ip_fanout"]  # fan-out is per hour, also causal
    pd.testing.assert_frame_equal(early[cols], w_past[cols])

def test_contamination_is_just_a_train_quantile(data):
    w, _ = build_windows(data)
    X = to_matrix(w)
    m = train(X, contamination=0.02)
    assert m.offset_ == pytest.approx(np.percentile(m.score_samples(X), 2))
    assert abs((m.predict(X) == -1).mean() - 0.02) < 0.002
.......                                                                  [100%]
7 passed in 2.98s

Production checklist

Compute features causally, from state that only successful logins update, and test window boundaries explicitly.
Set the threshold from an alert budget on recent data, not from contamination="auto".
Report precision, recall and PR-AUC per attack type, plus false positives per day. A single F1 hides the low-and-slow row.
Keep rules for known high-signal patterns (fan-out, failure bursts, impossible travel) and use the forest as a ranked review queue for the rest.
Ship an explanation with every alert, and treat it as a hint.
Monitor false positives per day, not only PSI. The loudest drifting feature may not be the one that hurts.
Retrain on a rolling window after known changes, compare recall per attack type before and after, and allowlist infrastructure such as VPN egress IPs in the features.
Treat user feedback (“that was me, I was traveling”) as labels, and eventually move to a supervised model on top of these features.

Wrapping up

Isolation Forest is a reasonable first anomaly detection algorithm for login logs: it needs no labels, trains in seconds and gives a ranked list. On this data it didn’t replace rules. It complemented them, catching half of an attack class the rules missed, at the cost of a noisier queue. Most of the work, and most of the bugs, were in the features and the threshold, not the model.

What does your team use for suspicious-login detection: vendor risk scores, hand-written SIEM rules, or your own models? And how do you get labels for the alerts nobody escalates?

Building a Web MVP That Does Not Collapse After the First 100 Users

Kate Steele — Tue, 20 Oct 2026 14:16:52 +0000

Most MVP architecture advice falls into one of two extremes.
The first says, “Move fast, because everything will be rewritten.” The result is often a fragile application with business logic inside UI components, inconsistent data, missing logs, and production changes that feel dangerous.
The second says, “Design for millions of users from day one.” That path produces infrastructure for traffic the product has not earned and a delivery schedule that postpones the most important test: whether anybody needs the product.
A better web MVP sits between those extremes. It is intentionally small, technically legible, and operationally safe. It does not predict every future requirement, but it gives the team room to learn without rebuilding after each discovery.
Here is a practical way to approach it.

1. Define the product boundary before the system boundary

Architecture starts with the user outcome.
Write down the primary actor, the action they need to complete, and the state change that proves the action succeeded. For an appointment platform, the important state change may be “available slot” to “confirmed booking.” For a B2B reporting tool, it may be “raw data uploaded” to “report shared.”
This definition prevents the technical design from expanding around hypothetical features. It also helps identify the minimum set of capabilities required across the stack.
Map the happy path first, then add the failure paths that would damage trust:

invalid or incomplete input;
expired sessions;
rejected payments;
duplicate requests;
unavailable third-party services;
authorization failures;
interrupted uploads;
notifications that cannot be delivered. An MVP can omit secondary workflows. It cannot pretend that predictable failures do not exist.

2. Prefer a modular monolith until distribution solves a real problem

Microservices are useful when teams need independent deployment, scaling, ownership, or fault isolation. Most early products do not yet have those constraints.
A modular monolith is often a stronger starting point. Keep clear internal boundaries for accounts, billing, inventory, messaging, reporting, or other domains. Modules should expose deliberate interfaces rather than reaching into one another’s tables and private logic.
This structure keeps development and deployment understandable. It also allows later extraction if one domain develops different performance, security, or ownership needs.
The important word is modular. A single codebase is not automatically simple. Naming, folder structure, dependency direction, and ownership of business rules still matter.

3. Keep business rules out of the browser

Modern front-end frameworks make it easy to place substantial logic in client components. Some validation belongs there because immediate feedback improves usability. The source of truth should still live on the server.
Prices, permissions, eligibility, quotas, workflow transitions, and transaction status should be verified by trusted backend code. A user can modify requests, replay calls, or bypass interface restrictions. The application must remain correct when the browser behaves unexpectedly.
A useful test is to imagine replacing the current interface with a mobile client or partner API. If the core rules would need to be rebuilt, too much domain logic probably lives in the front end.

4. Model data around identities and state transitions

Early schemas often mirror screens. That works until the product adds another role, a review process, historical reporting, or an integration with a different data lifecycle.
Define the main entities, ownership rules, and transitions. Ask which records represent durable facts, which changes need a history, who may modify each record, what must remain unique, and what happens when related data is deleted.
Use database constraints for facts the database can enforce. A unique email, one active reservation per slot, or a required relationship should not rely only on application code.
Migrations should be versioned from the beginning. “We will clean up the database after validation” is rarely as easy as it sounds because production data quickly becomes part of the product’s reality.

5. Buy commodity capabilities, own differentiating logic

A first release usually should not include a custom identity system, email delivery service, payment processor, file storage layer, or analytics engine. Mature providers can remove months of non-differentiating work.
Outsourcing a capability does not outsource responsibility. Wrap third-party APIs behind your own service boundary. Keep provider-specific identifiers separate from internal IDs. Verify webhook signatures. Handle retries and duplicated events. Store enough context to investigate failures without logging secrets.
Also define what happens when a dependency is slow or unavailable. Can the user retry safely? Does the system display a pending state? Will a background job reconcile the result? Is an operator alerted?
Integrations are part of the application, not plumbing outside it.

6. Implement authorization as a policy, not a scattered condition

Authentication answers who the user is. Authorization answers what that user may do.
Even a simple MVP can have customers, operators, administrators, owners, and invited members. Rules spread quickly when every controller contains its own role check.
Centralize authorization decisions and check both action and resource. Include negative tests that verify users cannot read, update, or infer resources outside their scope.

7. Add observability before you need to debug a customer’s problem

A console full of unstructured messages is not an operational plan.
At minimum, production should provide searchable application logs, error reporting with stack traces, health checks, basic performance metrics, and alerts for failures that require action. Use correlation or request IDs so one user journey can be followed across application code and external calls.
Logs should answer practical questions:

Which user or organization experienced the problem?
What operation was attempted?
Which application version handled it?
Which external provider was involved?
Was the request retried?
Did the final state become success, failure, or pending? Do not place passwords, tokens, full payment data, or other sensitive fields in logs. Observability should improve control without creating another security problem.

8. Build responsive behavior into the component system

“Mobile support later” often means redesign later.
Responsive web development is not only about adding breakpoints. It affects information hierarchy, table behavior, navigation, touch targets, form length, upload flows, modal usage, and the amount of context visible at once.
Define a small set of layout primitives and test the core journey on real viewport sizes as it is built. Do not wait until the final QA week to discover that a critical dashboard becomes unusable on a narrow screen.
The first release may optimize for desktop if the audience works primarily on large displays. It should still fail gracefully on other devices and communicate unsupported constraints clearly when necessary.

9. Protect the deployment path

An MVP needs a repeatable route from code to production.
Use separate test and production environments. Keep configuration outside the codebase. Automate checks for the most valuable workflows, include migrations in the release procedure, maintain a rollback or forward-fix strategy, and test data recovery.
The pipeline does not need dozens of stages. It needs to make routine changes boring.

10. Optimize what users can feel

Premature optimization wastes time, but ignoring performance distorts product feedback.
Measure page load, API latency, slow queries, and front-end weight. Compress images, paginate large results, cache stable content, and avoid sending data the current view does not need. Focus on the main journey, not an impressive landing-page score that hides a slow signed-in workflow.

Review architecture through the MVP question

A web development MVP should make the product easier to test, operate, and change. Every technical choice can be reviewed against those three goals.
Does the choice help the team release a credible user outcome? Can developers and operators understand failures? Can the system absorb the next likely change without a rewrite? Is the complexity solving a present constraint or only an imagined future one?
The right architecture is not the most advanced architecture. It is the smallest structure that protects learning.
Build the narrow path well. Keep rules explicit, data consistent, integrations observable, permissions enforceable, and releases repeatable. Then let actual usage tell you which part of the system deserves to become more sophisticated.

Building a Web MVP That Does Not Collapse After the First 100 Users

Kate Steele — Fri, 16 Oct 2026 15:03:30 +0000

1. Define the product boundary before the system boundary

invalid or incomplete input;
expired sessions;
rejected payments;
duplicate requests;
unavailable third-party services;
authorization failures;
interrupted uploads;
notifications that cannot be delivered. An MVP can omit secondary workflows. It cannot pretend that predictable failures do not exist.

2. Prefer a modular monolith until distribution solves a real problem

3. Keep business rules out of the browser

4. Model data around identities and state transitions

5. Buy commodity capabilities, own differentiating logic

6. Implement authorization as a policy, not a scattered condition

7. Add observability before you need to debug a customer’s problem

Which user or organization experienced the problem?
What operation was attempted?
Which application version handled it?
Which external provider was involved?
Was the request retried?
Did the final state become success, failure, or pending? Do not place passwords, tokens, full payment data, or other sensitive fields in logs. Observability should improve control without creating another security problem.

8. Build responsive behavior into the component system

9. Protect the deployment path

10. Optimize what users can feel

Review architecture through the MVP question

Building a Production-Ready Stellar Payment App Beyond sendPayment()

Kate Steele — Tue, 06 Oct 2026 08:32:45 +0000

A basic Stellar payment demo can be built quickly. Load an account, create a transaction, add a payment operation, sign it, submit it, and display the hash.

A production application must also prevent duplicate transfers, coordinate sequence numbers, support multiple assets, connect to traditional rails, explain pending states, and reconcile every financial event.

The network transaction is one step in a larger workflow. This article outlines the architecture around it.

Define the account and custody model first

A separate Stellar account for each user provides a distinct on-chain identity and observable balances. It also requires account creation, reserve management, trustlines for issued assets, and a custody model.

A pooled account represents customers in an internal ledger and can differentiate them through mixed accounts or memos. It reduces network account overhead but increases reliance on internal accounting and controls.

A non-custodial wallet lets the user sign. A custodial product signs on the customer’s behalf and therefore needs hardened key management, withdrawal controls, approval policies, and monitoring.

Before implementation, answer:

Who can authorize a transfer?
Where do signing keys live?
Who pays network fees and reserves?
How is an on-chain account linked to a customer?
What happens when credentials are lost?

These decisions cannot be hidden behind a wallet library.

Keep an internal payment record

Never use a transaction hash as the only application record.
Create a payment entity before submitting anything:

payment_id idempotency_key customer_id source_account destination_account send_asset send_amount receive_asset minimum_receive_amount quote_id stellar_transaction_hash status failure_code
The application owns the request and business state. Stellar provides the authoritative network result.

A useful state machine may include:

CREATED AWAITING_SIGNATURE READY_TO_SUBMIT SUBMITTED CONFIRMED PAYOUT_PENDING COMPLETED FAILED REFUND_REQUIRED REFUNDED
Transitions should be explicit. The same idempotency key must return the existing payment. After an uncertain HTTP response, query stored state and the network before building another transaction.

Manage sequence numbers under concurrency

Every Stellar transaction uses a source account sequence number. Two workers that load the same account simultaneously can prepare transactions with the same next sequence. One succeeds and the other fails.

Serialize submissions per source account with a queue or lock. At higher rates, channel accounts can distribute transaction sources where the signing model permits.

Do not trust an in-memory counter. Restarts, timeouts, or external submissions can invalidate it. Reconcile sequence state whenever the result is uncertain.

Add time bounds so a signed transaction cannot remain valid indefinitely. Stellar sequence numbers are consumed by transactions and help enforce ordered, one-time transaction execution.

Build transactions from validated intent

The backend should accept a business command such as “send 50 units of asset A to recipient B,” not an arbitrary serialized transaction from an untrusted client.

Validate:

source authorization;
destination format;
asset code and issuer;
required trustline;
available balance;
transaction limits;
quote expiration;
compliance status;
fee and memo requirements.

Build the transaction deterministically and show the destination, asset, amount, fees, and conversion limits before signing.

When a client constructs the transaction, the signer should still decode and verify every operation instead of approving opaque XDR.

Handle issued assets deliberately

XLM does not require a trustline, while classic issued assets generally do. A recipient without the appropriate trustline cannot receive the asset through a normal payment.

Check this before submission. A wallet can guide trustline creation, while a custodial service may create or sponsor required entries.

Asset identity is the combination of code and issuer, not the code alone. Store and display both. Maintain an allowlist when the product supports a controlled asset catalog.

Authorization-required, revocable, and clawback-enabled assets have different risk and support implications. Make relevant restrictions visible. Stellar assets can also interact with Soroban contracts through the built-in Stellar Asset Contract.

Match path payments to the quote

Path payments allow the source and destination assets to differ.

Choose the operation according to the customer promise:

_PathPaymentStrictSend _fixes the source amount and sets a minimum destination amount.
PathPaymentStrictReceive fixes the destination amount and sets a maximum source amount.

Query paths before building the transaction, then apply limits that match the quoted rate and expiration. Never silently widen the tolerance after approval.

A quote record should preserve the source amount, destination amount, fees, expiry, and route assumptions. When it expires, return to pricing rather than submitting under new economics.

Integrate anchors as durable workflows

Anchors connect Stellar assets with off-chain payment rails. Common Stellar Ecosystem Proposals cover service discovery, authentication, customer information, deposits, withdrawals, cross-border payments, and quotes.

A typical integration may use SEP-1, SEP-10, SEP-12, SEP-6 or SEP-24, SEP-31, and SEP-38.

Use an anchor adapter and persist the provider transaction ID, customer reference, quote, rail, required actions, and status history.

SEP-24 commonly gives the anchor control of an interactive deposit or withdrawal flow. SEP-6 supports a more programmatic experience. The choice affects UX and compliance responsibilities.

Callbacks and polling results may arrive more than once or out of order. Updates must be idempotent, and provider-specific evidence should be retained for support and reconciliation.

Ingest network results independently

The transaction submitter should not be the only component that knows whether a transfer succeeded.

Run an ingestion process that follows relevant accounts, operations, or contract events from a durable cursor. Convert network records into internal events such as:
TransactionConfirmed IncomingPaymentObserved OutgoingPaymentObserved TrustlineChanged

Store the hash, ledger, operation index, asset, amount, source, destination, memo, and cursor. Deduplicate by stable network identifiers.

Independent ingestion catches external submissions and cases in which Stellar accepts a transaction but the client times out.

Advance the cursor only after the related database changes are committed.

Separate confirmation from business completion

A confirmed Stellar transaction may not mean the customer journey is finished.

For a wallet transfer, confirmation may be final. For a withdrawal, an anchor may still need to send funds through a bank or mobile-money rail. For a merchant payment, another service may need to release the order.

Model these stages separately. A payment can be confirmed on Stellar while its payout remains pending. This prevents premature notifications and makes support investigations clearer.

Sponsor fees with policy controls

Fee-bump transactions allow one account to pay the fee for another account’s transaction without changing the inner transaction’s authorization. Sponsored reserves can help cover selected ledger-entry requirements.

Before the sponsor signs, validate the inner transaction, customer, asset, amount, and allowed operations. Add rate limits and abuse monitoring. Sponsorship must not become an arbitrary transaction relay.

Reconcile every view of value

A custodial or hybrid product usually has three views:

the application’s customer ledger;
balances and transactions on Stellar;
balances and statuses held by anchor, banking, or payment partners.

Compare them continuously and classify differences as expected delay, missing callback, duplicate event, failed payout, unmatched deposit, or manual adjustment.

Operator tools should search by internal payment ID, network hash, account, memo, mixed ID, anchor transaction, and bank reference. Corrections need approvals and audit history.

Test uncertainty, not only valid transfers

Cover duplicate API requests, concurrent submissions, expired time bounds, missing trustlines, stale paths, rejected signatures, submission timeouts, duplicate anchor updates, cursor restarts, and payout failure after on-chain settlement.

Use Testnet for integration behavior, but keep deterministic unit tests for construction and state transitions. External service changes should not make the entire suite unreliable.

Expert Stellar blockchain development services should therefore cover more than SDK integration. The critical work lies in account architecture, custody, orchestration, anchor connectivity, reconciliation, observability, and safe operations.

A payment app becomes production-ready when every transfer has one understandable history. Customers see clear status, support can explain events, finance can reconcile value, and engineers can retry safe steps without duplication.

sendPayment() is the visible action. The system around it is the product.

Single-Chain or Cross-Chain DEX? A Product Decision Guide

Kate Steele — Thu, 17 Sep 2026 09:28:22 +0000

A new decentralized exchange often begins with an ambitious idea: let users trade assets from many blockchain networks in one place. The promise is appealing. A broader asset range can attract more communities and reduce the need to move between separate applications.

Cross-chain access, however, is not only an expansion of a single-chain product. It changes the user journey, liquidity strategy, security model, support process, and operating responsibilities.

For some teams, a multi-chain or cross-chain DEX is the right destination. For others, starting with one ecosystem creates a better product and a safer path to growth. The decision should be based on a defined user problem rather than the assumption that more networks automatically mean more value.

Start with the user’s actual obstacle

The first question is not “How many chains should we support?” It is “What prevents the target user from completing the desired trade today?”

Users may face different obstacles:

The assets they need exist on separate networks.
Liquidity is fragmented across several venues.
Moving assets requires too many unfamiliar steps.
The preferred network has limited token variety.
Existing tools do not explain routes and fees clearly.
A community needs one interface for several deployments of the same protocol.

Each obstacle suggests a different product.

If users mainly trade assets within one ecosystem, a focused single-chain exchange may solve the problem. If they regularly move value between ecosystems, cross-chain functionality may be central. If the main issue is fragmented pricing, aggregation could matter more than native deployment on many networks.

The scope should follow the problem.

Why a single-chain launch can be a strength

Supporting one network may look less ambitious, but it gives the team a smaller environment in which to build market quality.

The product can focus its liquidity strategy on selected pairs, optimize wallet and transaction flows for one ecosystem, and create clearer support materials. Monitoring is also easier because the team works with one set of network conditions, token standards, and infrastructure dependencies.

This focus can benefit users. They see fewer network switches, fewer asset versions, and fewer route variations. The interface can use language and defaults suited to the ecosystem rather than presenting a generic experience for everyone.

A single-chain DEX can also develop a recognizable role, such as the preferred market for a specific community, application ecosystem, or category of assets. Depth in one market can be more valuable than weak coverage across many.

When multi-chain deployment makes sense

A protocol may have active communities on several networks. In that case, deploying a similar exchange experience to each network can extend reach without necessarily creating direct cross-chain swaps.

Users trade locally on the chain they have selected. Liquidity remains separate, but the product can provide a consistent interface, shared brand, and unified analytics.

This model can work when each ecosystem has enough users and liquidity to support its own markets. It also allows the team to expand step by step. One deployment can be tested before the next is introduced.

The limitation is fragmentation. The same asset pair may have different prices, liquidity levels, and incentives across networks. Users may still need another mechanism to move assets between them.

The product should not present separate deployments as one unified market.

Cross-chain trading adds another transaction story

A standard swap already involves several concepts: token approval, quoted output, price impact, network fees, and confirmation. A cross-chain action can add source and destination networks, bridging, route providers, multiple fees, longer completion times, and more transaction states.

The interface must turn that process into a story the user can follow.

Before confirmation, the product should show:

which asset leaves the source wallet;
which asset is expected on the destination;
the networks involved;
the estimated total cost;
the expected route and provider;
important timing or liquidity conditions;
what happens if one step cannot be completed.

After confirmation, progress should remain visible. “Pending” is not enough when a transaction passes through several systems. Users need to know which step has completed and whether action is required.

Liquidity is not automatically unified

Adding several networks does not create one deep market by itself. Liquidity may remain divided among pools, bridges, wrapped assets, and external venues.

The routing layer must decide how to connect available sources. The best path may change according to trade size, fees, asset versions, network activity, and available depth. A route that works for a small swap may be unsuitable for a larger one.

The exchange needs rules for evaluating routes, rejecting unreliable options, and explaining trade-offs. The lowest visible fee is not always the best choice if the route is slow or difficult to recover.

Teams should measure execution quality across the complete route rather than evaluating only the swap on the destination network.

Asset identity becomes a product problem

Multi-chain environments often contain native assets, bridged representations, and tokens with similar names or symbols. Users may not understand why two versions of an asset are not interchangeable.

The DEX should provide clear asset metadata and show the network and token identity throughout the journey. Search results, balances, quotes, and transaction history should use consistent labels.

The product also needs a policy for supported assets. Permissionless market creation can coexist with verified token information, but the distinction must be visible. Otherwise, broader access can increase confusion and support cases.

Asset discovery is not a minor interface detail. It is part of transaction safety.

Every added network expands operations

A new chain introduces more than another option in a network selector. The team may need additional nodes or data providers, wallet integrations, token lists, monitoring rules, deployment processes, and support documentation.

Network behavior also varies. Confirmation expectations, fee patterns, transaction formats, and infrastructure reliability may differ. The product must communicate these differences without making the experience feel inconsistent.

Before adding a network, the team should define who owns its operations, how incidents will be detected, and what level of usage would justify continued support. An integration that exists but is rarely monitored can become a long-term liability.

Cross-chain dependencies need explicit risk boundaries

Cross-chain products commonly depend on external bridges, messaging systems, route providers, or liquidity sources. These components may be operated and upgraded independently.

The team needs to know how each dependency is monitored and what happens when it is unavailable. The interface should also identify when another protocol handles part of the route.

A fallback does not always mean silently selecting another provider. Different routes can have different asset representations, costs, and conditions. In some situations, stopping the action and asking the user to review a new quote is safer than automatically changing the path.

Three practical expansion models

Teams can approach broader network access in several ways.

Separate network deployments

The protocol launches on multiple chains, with local liquidity on each. This offers gradual expansion and a familiar experience, but markets remain fragmented.

Aggregated liquidity access

The interface compares or combines routes from existing venues. This can improve asset coverage without requiring the protocol to create every pool, but it increases dependency on external data and execution sources.

Cross-chain transaction flow

The product coordinates movement and exchange across networks in one journey. This offers the most unified experience, but also creates the greatest demands on routing, status tracking, recovery, security, and support.

A roadmap can use more than one model over time. The important point is to describe accurately what the product does at each stage.

Use evidence to choose the second network

The first expansion should be driven by observable demand.

Useful signals include users attempting to connect from another network, repeated requests for specific assets, existing community activity, reliable liquidity partners, and a clear volume opportunity. The team should also assess whether users currently leave the product to complete related actions elsewhere.

An experienced partner in decentralized crypto exchange development can help compare expansion models, map dependencies, and estimate how each option changes the product and operating scope. The goal should be a controlled roadmap, not the largest possible network list.

Expand only when the experience can remain coherent

Single-chain and cross-chain DEXs solve different problems. One offers focus and the opportunity to build deep market quality within an ecosystem. The other can connect fragmented users and assets, but requires stronger routing, communication, monitoring, and recovery design.

A team should begin with the smallest scope that solves the user’s real obstacle. It can then expand when demand, liquidity, and operational readiness support the next step.

The most useful DEX is not necessarily the one connected to the most networks. It is the one that lets users understand what they are doing, complete it reliably, and return with confidence.

How to Build a Data Science MVP Without Creating a Maintenance Nightmare

Kate Steele — Wed, 02 Sep 2026 14:31:11 +0000

A data science MVP is often interpreted as “train a model quickly and show a demo.” That approach can validate that an algorithm finds patterns, but it does not validate whether the product can deliver a useful decision repeatedly.
A better MVP is a thin vertical slice. It uses real input data, runs through a repeatable pipeline, produces an output for a real user or system, and captures enough feedback to judge whether the idea deserves further investment. It stays small without becoming disposable.

Define the smallest useful decision

Start with one user, one decision, and one outcome.
“Predict customer behavior” is not an MVP scope. “Rank accounts for the weekly retention review and measure how many flagged accounts renew after intervention” is bounded enough to test.
Write down:

who receives the result;
when they receive it;
what action they can take;
how many cases they can handle;
what happens when the system is uncertain;
which outcome will be measured. This definition prevents the team from building features that look impressive but do not affect the decision.

Prove value with a baseline first

Before training a complex model, create the simplest credible baseline. It may be a business rule, a rolling average, a linear model, or the current manual process.
A baseline serves three purposes. It confirms that the evaluation pipeline works. It reveals whether sophisticated modeling is necessary. It gives stakeholders a clear comparison.
For example, a demand forecasting MVP might compare:
Baseline A: same value as last week Baseline B: average of the previous four weeks Candidate: model using seasonality, promotions, and stock signals
If the candidate produces only a small technical improvement and requires much more maintenance, the baseline may be the better product. Complexity should earn its place.

Build one vertical slice

The MVP architecture can be compact:
source extract -> validation -> feature generation -> baseline and candidate model -> stored predictions -> lightweight user view or API -> outcome capture
Each arrow should be executable without manual notebook steps. The pipeline may run from a scheduled command, but it should record the input version, configuration, and output location.
Avoid building a general-purpose platform before the first use case is proven. At the same time, do not hide critical logic inside an analyst’s local environment. The goal is a narrow system with clean boundaries.

Keep the repository boring

A predictable structure makes the MVP easier to review and extend:
project/ ├── config/ │ ├── development.yaml │ └── production.yaml ├── data_contracts/ ├── src/ │ ├── ingest.py │ ├── validate.py │ ├── features.py │ ├── train.py │ ├── predict.py │ └── evaluate.py ├── tests/ ├── notebooks/ ├── artifacts/ └── README.md
Notebooks belong in the project, but they should not be the only place where transformations or model logic exist. Move reusable code into modules as soon as it affects a result that others need to reproduce.

Define interfaces before implementations

A stable interface lets the team replace the model without changing every consumer.
from dataclasses import dataclass from typing import Protocol @dataclass(frozen=True) class Prediction: entity_id: str score: float model_version: str status: str class Scorer(Protocol): def score( self, entity_id: str, features: dict[str, float], ) -> Prediction: ...
The first implementation could wrap a rules engine. The second could use a statistical model. The dashboard or API only depends on the Prediction contract.
This separation also makes fallback behavior easier. When the candidate model is unavailable, the application can call the baseline scorer and mark the result accordingly.

Validate data before optimizing models

MVP teams sometimes spend days tuning parameters while the input pipeline quietly produces duplicates, stale records, or inconsistent labels.
Add a small number of high-value checks:

required columns exist;
identifiers are not unexpectedly null;
timestamps fall within an expected range;
key values are unique where required;
target labels follow a documented definition;
row counts and missing-value rates remain plausible.

Failing the pipeline is often better than publishing predictions based on corrupted input. For lower-risk use cases, the system may continue with a warning, but that choice should be explicit.

Use time-aware evaluation

Random train-test splits can create misleading results when behavior changes over time. For many business problems, training on earlier periods and evaluating on later periods better reflects deployment.
The split should also match how predictions will be generated. If the product scores customers every Monday, create historical snapshots that use only information available by each Monday. This avoids leakage from future events.
Do not report only one aggregate metric. Review performance across periods and relevant segments. A model that looks good overall may fail for a region or product line that matters operationally.

Ship the output into an existing workflow

An MVP does not need a full product interface. A scheduled table, a simple internal page, or an API connected to an existing tool may be enough.
The important requirement is that a real user can act on the result. Sending a CSV by email may be acceptable for an early test, but the process should still capture which version generated it and what users did with the recommendations.
The delivery format should expose uncertainty. A ranked queue, confidence band, or “manual review required” status is often more useful than an unexplained yes-or-no label.

Add feedback as a first-class feature

Without outcome and user feedback, the team can measure model behavior but not product value.
Capture whether the recommendation was accepted, overridden, or impossible to use. Record the eventual outcome when it becomes available. Keep feedback fields structured enough to analyze, but allow a short note when context matters.
This reveals whether the problem is the model, the workflow, or the available action. A highly accurate recommendation can still fail if users receive it too late or lack authority to respond.

Automate only what the risk permits

For an MVP, human review is often an advantage. It limits harm, generates labeled feedback, and exposes edge cases. Automation can expand after the team understands failure modes and establishes monitoring.
Define an abstention path. The model should be able to say that it lacks sufficient information. Route those cases to a baseline or a person rather than forcing a low-confidence answer.
When a project requires coordinated discovery, engineering, modeling, integration, and release planning, external data science development services can accelerate the vertical slice. The deliverable should still remain transparent, documented, and transferable to the internal team.

Test the failure paths

Happy-path tests are not enough. An MVP should answer practical questions:
What happens when the source file is empty? What if a category appears that the model has never seen? Can the pipeline be rerun without duplicate predictions? Can the previous model version be restored? Does the consumer know when a score is stale?
A focused test suite can cover:
schema tests feature transformation tests model interface tests pipeline integration tests fallback tests
You do not need exhaustive coverage, but you do need confidence in the parts that could silently change a decision.

Set exit criteria before the demo

An MVP should end with a decision: scale, revise, or stop. Define the criteria before stakeholders see a polished interface.
Useful criteria may include:

improvement over the baseline;
sufficient data quality and coverage;
acceptable operational workload;
evidence that users act on the output;
measurable movement in the target outcome;
a feasible path to security, monitoring, and ownership.

A technically successful experiment may still be stopped if the intervention is too costly or the data can’t be maintained. That is a valuable result because it prevents a larger investment in the wrong system.

What not to build yet

Delay components that do not reduce the main uncertainty. The MVP probably does not need a multi-model feature platform, elaborate real-time serving, automated retraining, or a custom analytics portal.
Add infrastructure when the use case proves that scale, latency, reuse, or governance requires it. Until then, favor simple scheduled jobs, versioned files or tables, clear contracts, and visible logs.
Small does not mean careless. It means every component exists for a reason.

Conclusion

A maintainable data science MVP is not a miniature enterprise platform and not a disposable notebook. It is a narrow, repeatable decision loop built with enough engineering discipline to produce trustworthy evidence.
Start with a baseline, ship one vertical slice, capture real feedback, and test failure behavior. Then scale only the parts that the evidence says are worth keeping.

A Practical Guide to Database Migration Without Operational Chaos

Kate Steele — Tue, 01 Sep 2026 14:40:29 +0000

Database migration is often summarized as moving data from one system to another. That description is technically correct and operationally incomplete.

A database carries business meaning: relationships, permissions, historical states, and assumptions accumulated over years. The main risk is not failed copying, but a complete-looking system that interprets the business differently.

A reliable migration is a controlled product transition, not a single technical event.

Begin with the business reason

Migration is disruptive. The team should be able to explain why the current system no longer supports the business and what the new one must improve.

Common reasons include performance limits, difficult scaling, unsupported technology, rising maintenance cost, weak access control, poor integration options, inconsistent data, or the need to support a redesigned product.

The reason determines the plan. A migration driven by security risk may prioritize rapid retirement of the old platform. A migration driven by a new data model may require more time for interpretation and business validation. A performance project may keep much of the existing structure while changing the operating environment.

Success criteria should be measurable. “Move to the cloud” describes an activity. “Reduce failed peak-hour transactions while preserving reconciliation accuracy and shortening recovery time” describes an outcome.

Without a clear reason, the migration can become a long program that reproduces the old system without improving it.

Inventory every dependency, not only every table

Applications rarely access a database through one clean path. Customer-facing services, internal tools, scheduled reports, scripts, integrations, exports, monitoring jobs, and individual analysts may all depend on it.

Some dependencies are documented; others appear only when they fail.

The inventory should identify systems that read or write data, their owners, reports, scheduled processes, permission dependencies, external partners, availability requirements, and business periods when disruption is especially costly.

Connection logs, code search, interviews, and operational observation can all reveal dependencies.

The goal is not merely to produce a list. Each dependency needs a migration decision: update, replace, retire, or temporarily support through a compatibility layer.

Define the source of truth during every phase

A migration creates a period in which old and new systems may both contain current information. Unless ownership is explicit, teams can update different versions and create conflicts that are difficult to reconcile.

For each stage, define which system is authoritative for each record or workflow. The old platform may remain the source of truth while data is copied to the new one for testing. Later, one customer group or process may switch while the rest stays on the old system.

Dual writing should be treated carefully. Sending every update to two databases sounds reassuring, but partial failures can produce divergence. If dual writing is necessary, the team needs an operation log, retry strategy, reconciliation process, and a clear rule for resolving conflicts.

Temporary architecture still needs production-level discipline because the business will depend on it during the most sensitive part of the transition.

Clean data before structure hides the problem

Legacy databases contain valid history alongside duplicates, missing identifiers, outdated categories, and records that no longer match current rules. Moving all of this unchanged may make the technical cutover faster while preserving the reasons the old system was difficult to use.

Migration creates a valuable opportunity to classify data.

Some records should be transformed into the new model. Some should be corrected or merged. Some should remain in an archive. Some may need to be deleted under retention rules. Ambiguous cases should be reviewed by business owners rather than decided through a generic script.

Cleaning does not mean rewriting history to make it look tidy. Corrections should remain traceable, especially for financial, regulated, or customer-impacting information.

Transformation rules and exceptions should be recorded so migrated values can be explained later.

Map meaning, not only fields

Field-to-field mapping is necessary but insufficient.

The same label may have different meanings in two systems. A legacy “active” status might include customers who are trialing, paying, paused, or awaiting review. The new model may separate those states. One old customer record may become an organization, several contacts, multiple subscriptions, and a billing profile.

This is not a copying decision. It is a domain decision.

Mappings should describe business interpretation, defaults, required enrichment, exceptions, and how relationships and historical states will be preserved.

When teams focus only on column names and data types, they risk delivering technically valid records that produce incorrect workflows.

Build migration runs to be repeatable

The first migration run should not be the production cutover. Teams need repeated rehearsals using representative data and the same procedures intended for launch.

A repeatable process extracts a defined source version, applies documented transformations, loads the target environment, produces validation results, and records rejected or changed data.

Rerunning it should not create duplicates or depend on hidden manual steps. Necessary manual decisions should be captured so they can be reapplied.

Repeated runs reveal how long the move takes, where failures cluster, and whether the process behaves consistently as source data changes. They also make rollback and recovery more credible because the team has practiced the operation.

Validate business behavior, not only data volume

Matching row counts can prove that a certain number of records arrived. It cannot prove that the new system behaves correctly.

Validation should cover several layers.

Structural checks confirm required fields, types, identifiers, and relationships. Reconciliation checks compare balances, totals, status distributions, and record counts. Application checks confirm that important workflows can read and update the migrated data. Permission checks verify that users see only what their roles allow. Historical checks confirm that past states and timestamps still mean what the business expects.

Business users should review representative and difficult cases. They know which exceptions are meaningful and which reports must agree before the system can be trusted.

Validation thresholds should be defined before the final run. Otherwise, teams may lower expectations under launch pressure.

Plan the cutover as a sequence

The cutover plan should specify what happens before, during, and after traffic moves.

Before the cutover, the team may freeze selected changes, complete a final incremental copy, verify system health, and confirm that support teams are ready. During the transition, traffic may move by customer segment, geography, workflow, or percentage. Afterward, the team monitors application behavior, data reconciliation, and user-reported issues.

Some systems require a single switch. Even then, the operation should be divided into checkpoints with explicit go or no-go decisions.

Employees and partners need to know which actions are unavailable, when data may appear delayed, and where to report unexpected behavior.

Rollback must restore a coherent business state

“Switch back if something fails” is not a complete rollback plan.

If the new system has accepted writes, the old system may no longer contain the latest state. Returning traffic without reconciling those changes can lose orders, approvals, payments, or user updates.

The rollback design should answer:

What conditions trigger rollback?
Who has authority to make the decision?
How are writes captured during the transition?
Can new records be replayed into the old system?
Which integrations must switch back?
How will users be informed?
How will the team confirm that the restored state is coherent?

Sometimes the safer response is to pause a workflow while the team repairs the new system rather than return to an outdated source. The correct choice depends on the business process and must be decided before the incident.

Security should improve during the move

Migration can temporarily increase exposure because teams create exports, staging environments, logs, backups, and access paths that do not exist during normal operations.

Sensitive data should remain protected throughout extraction, transfer, testing, and disposal. Access should be limited to the people and services that need it. Test environments should not receive production data by default without appropriate controls. Temporary copies should have owners and deletion dates.

The target system’s roles and permissions need direct validation. A technically successful migration that grants broader access is not successful.

Auditability is also important. Teams should know who initiated runs, which data version was used, what transformations occurred, and which records required manual intervention.

Choose outside support for migration discipline

A database development company can be valuable when the transition combines schema redesign, data cleansing, system integration, performance requirements, and high availability. The selection should focus on the migration method, not only familiarity with the target technology.

A capable partner should ask about business-critical periods, hidden consumers, historical requirements, access rules, rollback, reconciliation, and post-launch ownership. It should be able to explain how it will rehearse the migration and prove that the result is correct.

The organization should retain ownership of business definitions and acceptance criteria. External engineers cannot resolve an ambiguous status or duplicate customer without domain input.

Documentation and knowledge transfer should be included from the beginning so that the new platform does not become another system understood only by a small external group.

Do not retire the old system immediately

After cutover, the legacy database may need to remain available in a controlled, read-only state while teams verify historical queries, reports, and edge cases.

This period should have a defined end. Keeping the old system indefinitely creates cost and uncertainty. Retirement should cover final backups, access removal, integration shutdown, and disposal of temporary migration data.

Post-migration work should also include performance tuning based on real usage, review of failed or manually transformed records, and confirmation that monitoring and support procedures operate as expected.

The migration is complete when the organization can run the new system confidently, not simply when the first production request succeeds.

Conclusion

Database migration is difficult because data is inseparable from the business processes that created it. A reliable transition must preserve relationships, history, permissions, and operational continuity while introducing a new foundation.

The safest programs define the business reason, inventory dependencies, rehearse migration runs, validate real workflows, and prepare a rollback that protects the current state.

Treating migration as a controlled sequence of decisions turns it from a high-risk weekend event into an evidence-based product change.

Web3 Application Development: Designing Products Around Ownership, Composability, and Verifiable State

Kate Steele — Mon, 31 Aug 2026 14:10:46 +0000

Traditional applications usually rely on one company-controlled database. Web3 changes that model by letting important state live on a blockchain, where ownership and transaction rules can be verified independently.

That makes Web3 application development more than ordinary software with smart contracts added underneath. The real challenge is deciding what belongs on-chain, what should remain in conventional infrastructure, and how those layers can work together without making the product difficult to use.

What changes when application state moves on-chain

The biggest difference between Web2 and Web3 is not the wallet. It is where authority over important state lives. Once ownership or transaction logic becomes on-chain, users and other applications can rely on the same public record.

That creates stronger transparency, but it also makes product rules harder to change casually. The development team has to think about long-term behavior much earlier.

Ownership becomes part of the architecture

In a conventional application, an asset usually belongs to someone because the company database says so. In Web3, ownership can be represented through a blockchain token that other compatible applications can also recognize.

This changes the product model. The team needs to define what ownership actually allows, whether assets can move freely, and what happens when their state changes. Putting ownership on-chain gives users stronger control, but it also makes those rules more visible and harder to rewrite later.

Public state changes how rules are designed

When a smart contract controls rewards or transfers, users can verify the same logic the application relies on. That makes some product behavior more transparent than it would be inside a private backend.

Not every rule needs this treatment. Frequently changing features can remain off-chain when blockchain adds little value. The strongest architecture puts only the rules that benefit from independent verification on-chain.

Composability turns protocols into building blocks

Web3 applications can interact with existing smart contracts instead of rebuilding every capability. A new product may connect to an established exchange protocol or use another protocol for lending or asset management.

This can speed up development, but it also creates dependency risk. If an external protocol changes or becomes unavailable, the application may be affected even when its own code remains stable.

The architecture behind a Web3 application

Most Web3 applications are hybrid systems. Smart contracts provide the trusted execution layer, while other infrastructure handles faster data access and ordinary product logic.

The important part is keeping those responsibilities clear. Supporting services can improve usability without replacing the blockchain as the authoritative source for on-chain state.

Smart contracts act as the rule engine

Smart contracts should control the behavior that users need to verify independently. This may include ownership or settlement logic, while high-frequency application behavior can remain elsewhere.

Keeping the contract layer focused makes it easier to test and maintain. It also reduces the number of actions that require blockchain transactions, which can improve both cost and user experience.

Indexing turns blockchain history into usable data

Blockchains preserve transaction history well, but applications often need faster ways to query that information. Indexing systems process events into structured data so the frontend can display balances or activity without rebuilding everything directly from the chain.

The index should improve access, not become a competing source of truth. If it falls behind, the application should recognize that delay instead of showing stale information as confirmed.

Decentralized storage extends the ownership model

Large media files or detailed metadata are usually impractical to store directly on-chain. Content-addressed storage can provide a better fit by linking blockchain assets to information stored outside the chain.

This still requires planning. A content identifier proves which file is referenced, but the file also needs to remain available. Persistence therefore remains an operational responsibility even when storage is decentralized.

Product decisions unique to Web3 applications

Architecture choices eventually become user experience. Account design changes onboarding, while transaction design determines how much blockchain friction users actually feel.

A strong Web3 product hides unnecessary complexity without hiding the consequences of important actions.

Account design shapes onboarding

Crypto-native applications can rely on users bringing their own wallets. Mainstream products may need smart-account models that reduce dependence on seed phrases and native gas tokens.

Account abstraction can support capabilities such as gas sponsorship and transaction batching. The best model depends on the audience. A trading tool for experienced users and a consumer app should not automatically use the same onboarding flow.

Transactions need a visible lifecycle

Signing a transaction is not the same as confirmation. A transaction can remain pending before it succeeds or fails, so the interface needs to communicate those states clearly.

This is especially important when value is involved. Users should know whether an action is still waiting for the network or whether it has reached its final state, otherwise they may repeat the same transaction unnecessarily.

Permissions need to be understandable

Web3 users may grant token allowances or approve smart-account permissions. Contracts can also contain administrative roles that affect the system after launch.

The application should explain these permissions in product language. A technically correct signing request is not enough if the user cannot understand what authority they are giving away or whether it can later be revoked.

Decentralization should be selective

A Web3 application does not need every service to be decentralized. The useful question is which form of control users benefit from moving away from one central operator.

For some products, that is custody. For others, it is transparent settlement. Selective decentralization usually produces a cleaner product than moving routine functionality on-chain simply to make the architecture appear more Web3-native.

Composability and long-term evolution

Open protocols give Web3 applications access to infrastructure they did not build themselves. That can shorten development dramatically, but it also means the product inherits some of the assumptions of every external protocol it uses.

The application also needs a plan for change. Smart contracts may be difficult to replace once other users or applications depend on them.

External protocols need their own risk model

Integrating an existing protocol can be more efficient than rebuilding the same functionality internally. The team still needs to understand what the application depends on and what happens if that dependency changes.

This is particularly important with upgradeable contracts or liquidity-dependent services. The integration may remain technically available while the economic conditions around it become unsuitable for users.

Cross-chain features multiply application states

A single-chain application has one primary execution environment. Cross-chain functionality introduces periods where one network has completed an action while another is still waiting.

The application needs to represent that intermediate state clearly. Developers also need protection against duplicate execution and out-of-order messages, making multichain development a state-management problem rather than simply a matter of adding more RPC endpoints.

Upgrades need an explicit trust model

Upgradeable contracts make future fixes possible, but someone must retain authority to change the implementation. That power should be visible and governed deliberately.

Immutable contracts avoid that authority but may eventually need migration to a replacement. Either way, the product needs a clear plan for how users and connected applications will move when the system changes.

Production monitoring has to cover the full stack

A Web3 application can appear broken even when its smart contracts are healthy. RPC services can fail, indexers can lag, and external dependencies can stop responding.

Monitoring should therefore connect blockchain activity with application behavior. The team needs to distinguish a contract problem from an infrastructure problem quickly, especially when users depend on the system for valuable assets or transactions.

Conclusion

Web3 application development is fundamentally about deciding where trust should live. Smart contracts can make ownership and execution independently verifiable, while composability lets products build on open infrastructure that already exists.

The strongest applications use those properties selectively. Blockchain remains authoritative where verification matters, while conventional software keeps the product fast and flexible. When account design, dependencies, and long-term evolution are planned together, Web3 becomes a coherent application architecture rather than a collection of blockchain features.