DEV Community: 4thwithme

A/B testing: when the standard test is the wrong tool

4thwithme — Mon, 07 Sep 2026 07:44:07 +0000

Last week was the standard fixed-horizon test: split traffic 50/50, wait for your pre-calculated sample size, look once, done. That's still the right default most of the time. It's not the only tool, though, and treating it as the only one gets expensive once its assumptions stop matching your situation.

Why the default isn't always right

A fixed-horizon test answers one question: is there a detectable difference between two fixed groups, at a preset error tolerance, once enough data has come in. That's the whole job. It doesn't answer "how do I make the most money while this test runs," and it assumes your two groups don't affect each other. Both assumptions can quietly fail.

Here's a real version of why that matters. A subscription business earning roughly $1M a month runs a pricing test: 10% baseline conversion, a 5-week horizon at the standard 5%/80% thresholds. One week in, the test group converts at 15% against the control's 10% - a trend that, if it held, would add roughly $500K to that month's revenue. The textbook says don't stop, the sample size isn't there yet.

That's not a math problem, it's the wrong question. The test was built to answer "is there a detectable difference at a fixed error rate," not "how do I make the most money while this runs." Point it at the second question and it keeps giving you the wrong kind of answer.

Multi-armed bandits

Multi-armed bandit - instead of holding a fixed split for a fixed duration, you continuously shift more traffic toward whichever variant is currently winning, while still sending a smaller slice to the others so you don't lock in a lucky early read. The name comes from a gambler in front of a row of slot machines ("one-armed bandits"), trying to find the one that pays out best while losing as little as possible to the rest.

That tension has a name: the exploration-exploitation tradeoff. Exploit too early and you lock in the wrong winner off a noisy first week; explore forever and you never cash in on knowing which one wins. Studied by Allied scientists in World War II, and found so hard that mathematician Peter Whittle joked it should be dropped over Germany so enemy scientists could waste their time on it too, before Herbert Robbins gave it a serious mathematical treatment in 1952. Three practical strategies still run in production today: epsilon-greedy (mostly exploit, explore at random sometimes), UCB (favor whatever you're least certain about), and Thompson sampling (a Bayesian approach that updates its belief as data comes in).

Run as a bandit, the pricing test above looks different: most traffic goes to whichever price is winning, a smaller slice keeps testing the alternative, and the split keeps moving as the picture updates, instead of freezing at 50/50 for five weeks.

the losing variant never gets shut off entirely, it just gets starved of traffic

The trade-off: a bandit chases money made during the test, not a clean significance result, and usually needs longer to reach the same confidence one calculated sample size gets you. Use it when running a worse variant a little longer costs more than a tidy p-value is worth.

In simple words - Three ice cream stands, and you want to know which one people like best. A regular test sends exactly one-third of people to each stand for a whole month, even once it's obvious stand #2 is winning. A bandit is smarter: it quietly sends more people to stand #2, while still sending a few to the others, just in case they get better later.

Switchback tests

Switchback test - instead of splitting your population into two groups, split time into blocks and alternate the whole population between A and B as blocks pass: this hour everyone gets A, next hour everyone gets B, on a randomized schedule.

The case for this shows up once your users stop being independent of each other. A standard split assumes what happens to user A doesn't leak into user B's experience. That breaks in a marketplace: if half of riders see one price and half see another, drivers respond to whichever price is in front of them, and both groups end up competing for the same pool of drivers. The "control" group is now polluted by the treatment group's effect on shared supply. Alternating the whole market by time block avoids that: "this city under A" versus "the same city under B," not two artificially split groups sharing the same supply.

There's no single textbook citation for this the way "multi-armed bandit" has one; the closest formal relative is what statisticians call crossover design, adapted for markets instead of individual subjects. Uber and Lyft have both published on using it for dynamic-pricing and dispatch experiments, exactly because their two-sided marketplaces make a standard user-level split unreliable.

every time block is its own control for the block right next to it

In simple words - One playground, one set of kids. You can't fairly split them "half get the red slide, half get the blue slide," because they all play together and one group's fun affects the other's. So instead: everyone gets red on Monday, blue on Tuesday, red again on Wednesday - same kids, same playground, just switching what they try on different days.

Hold-out tests

A hold-out test is the simplest design here: keep a slice of your population permanently untouched by the change, and compare how their behavior evolves against everyone else's over time. No alternating, no traffic-shifting, just a group frozen in the "before" state as a long-running reference point. It earns its place for effects a short test can't see - trust, habit, long-term retention - anything that shows up as a slow drift over months, not a clean signal within weeks. The cost: you're deliberately holding back a possibly-better experience from real people, for a long time, just to keep a clean comparison available later.

A real one: Uber's rider-acquisition team saw wild week-to-week swings in Meta ad cost-per-acquisition that looked more like noise than signal, so they ran a 3-month hold-out - just turned Meta ads off for a slice of the market and watched. Nothing happened. Signups held steady, meaning the ads were just taking credit for people who'd have signed up anyway. That freed up roughly $35M a year for channels that actually moved the number - a finding a two-week test would never have had the runway to catch.

Pre/post & synthetic control

Sometimes there's no way to randomize at all, because there's only one of you: one product launching a redesign, one market entering a new pricing policy. You can't hold half a country in a control group. The blunt option is a pre/post comparison - metric before, metric after, call the difference the effect. It's also the weakest design here: everything else moving in that window (seasonality, a competitor's move, the economy) gets mixed into your "effect" with no way to pull it back out.

Synthetic control - when you have exactly one treated unit and no real control group, build a fake one: a weighted blend of other similar untreated units, chosen so the blend tracks your unit's trend closely before the change. After the change, the gap between what actually happened and what the blend predicts is your estimated effect.

This isn't an internet-testing invention, it comes from economics. Alberto Abadie and Javier Gardeazabal introduced it in 2003, building a "synthetic Basque Country" out of a blend of other Spanish regions to estimate what its economy would have done without a decades-long conflict. The method's Wikipedia page is a good primer; the idea carries over cleanly to product work, where "other regions" become other markets, stores, or cohorts that never saw your change.

before the change, the lines sit on top of each other - that's what makes the later gap meaningful

In simple words - One plant, and you want to know if a new fertilizer helped it grow. No second identical plant to compare against. So you blend ten other, similar-but-not-identical plants into one imaginary "average plant" that grew just like yours did before the fertilizer. If your real plant suddenly grows taller than that imaginary one, the gap is probably the fertilizer.

Don't reach for A/B/C/D early

Testing four or five variants at once is tempting when you have a backlog and want an answer fast. Resist it. A clean multi-variant test needs roughly the same sample size as a sequence of pairwise tests, but it's far harder to tell which comparison drove the result, and far easier to p-hack yourself by checking every pair until one clears the bar. A sequence costs calendar time, not data, and buys back your ability to actually interpret what happened.

How to actually choose

No universal method fits every situation - last week's flow solves one problem, nothing beyond that. You'll hit other limits too: overlapping tests, insufficient traffic, a goal about maximizing an outcome instead of detecting a difference. A rough guide:

Your situation	Reach for
Can randomize independently, want a calibrated error rate	Standard fixed-horizon A/B test
Users interact with each other (marketplace, shared supply)	Switchback test
Goal is maximizing outcome during the test, not a calibrated verdict	Multi-armed bandit
Only one treated unit exists, no randomization possible	Pre/post, ideally with synthetic control
Effect is slow, long-term, or compounding	Hold-out test
More than two variants on the table	Decompose into a sequence of A/B tests

None of these replace the fixed-horizon test as a default. Reach for them once you can name the specific way your situation breaks its assumptions - not before.

Elsewhere

GitHub · LinkedIn · Dev.to · Substack · 4thwithme.dev/blog

May the --force be with you. See you next week.

A/B testing: the stat-sig problem

4thwithme — Sun, 30 Aug 2026 22:54:41 +0000

Post source: https://4thwithme.dev/blog/ab-testing-stat-sig-problem/

[ THE SETUP ]

Generic scenario, no company specifics: you run an ecommerce storefront. Someone ships a change, a re-ranked product grid, a new checkout button color, doesn't matter. You split traffic 50/50, wait for data, and want to know: did this actually help, or did it just look like it helped?

That's the whole job of an A/B test. It answers exactly one question: is there a difference between the two groups, yes or no. It does not tell you why. It does not reliably tell you how much the metric will move once you roll out to everyone. It rejects a null hypothesis or it fails to reject it. That's the entire output. Everything else people read into a test result, they're reading in themselves.

[ DEF.1 ]

Null hypothesis (H₀) — the default, skeptical assumption that nothing changed: there is no real difference between A and B. An A/B test is built entirely to argue against this one claim. It either rejects it (evidence a difference exists) or fails to reject it (not enough evidence either way). It never proves the opposite outright, it just runs out of reasons to doubt it.

the assumption every A/B test starts by trying to disprove

[ WHAT "STAT SIG" ACTUALLY MEANS ]

Here's the misunderstanding almost everyone ships with: a p-value is not "the probability that B is better than A." It's the probability of seeing data this extreme (or more extreme) if there were actually no difference at all. It's a statement about the data given no effect, not a statement about the effect given the data. Those sound similar. They are not the same claim, and the difference matters every time you're deciding whether to trust a result.

Put another way: a p-value doesn't tell you "B is probably better." It tells you "if B and A were actually identical, how surprising would this result be?" A low p-value means the result would be surprising under the assumption of no real difference — that assumption starts to look shaky, so it's worth trusting. It does not tell you the odds that B is actually better; that's a different question the test never answers. It's the same logic as saying "if this coin were fair, getting 9 heads in 10 flips would be surprising" — that doesn't prove the coin is rigged, it just means the fair-coin explanation is hard to believe.

[ DEF.2 ]

P-value — the probability of seeing a result at least this extreme, assuming the null hypothesis is true, assuming there's actually no difference between A and B. It is not the probability that B beats A, and it is not the probability that the null hypothesis itself is true. A small p-value just means "this would be a strange coincidence if nothing had actually changed," nothing more.

a statement about the data given no effect, not about the effect given the data

To get to a p-value you first have to set up two competing claims:

Null hypothesis (H₀): there is no difference between A and B. This is always the same claim, every single time.
Alternative hypothesis (H₁): there is a difference.

An A/B test never proves H₁. It only ever rejects H₀ or fails to reject it. Failing to reject H₀ doesn't mean "there's no effect," it means "we didn't detect one at the sensitivity we set up for." That distinction alone would save a lot of bad blog posts about "our test failed, this feature doesn't work."

Then there are two ways to be wrong, and they have genuinely useful mnemonics:

[ DEF.3 ]

Type I error (false positive) — you see a difference that isn't really there. Telling grandpa he's pregnant.
Type II error (false negative) — you miss a difference that is really there. Telling a pregnant woman she isn't.

the two ways an A/B test can lie to you

The industry default is 5% tolerance for Type I (α = 0.05) and 20% tolerance for Type II (β = 0.20, i.e. 80% power). Neither number is handed down by nature. They're both choices you make based on the cost of being wrong. A drug trial might demand α with eight zeroes before the first significant digit. A button-color test where you genuinely don't care much either way could reasonably run at α = 0.20. The 5% convention is a convention, not a law.

Here's how the p-value and alpha actually connect: alpha is the surprise threshold you commit to before you look at any data — your pre-agreed tolerance for crying wolf. The p-value is how surprising your actual result turned out to be. The decision rule is just a comparison: p-value < alpha means "significant." That's it. And alpha isn't just a cutoff, it IS your Type I error rate — a 5% alpha means you've pre-accepted a 5% chance of telling grandpa he's pregnant, across all the times you'd run this test. Type II (missing a real difference) isn't controlled by alpha at all — that's a separate knob, beta, driven mostly by how much data you collect.

Beta works the opposite way from alpha: it's the rate at which you fail to catch a difference that's actually there — telling the pregnant woman she isn't. Beta isn't a tolerance you set directly like alpha; it falls out of how much data you collect, given the effect size you're trying to detect. Small sample, small true effect: beta is high, meaning you'll miss it most of the time and walk away concluding "no difference" when there really was one. More data pulls beta down. Power is just the flip side of the same number: power = 1 − β, so the industry-default β = 0.20 is the same fact stated as "80% power" — an 80% chance of actually seeing the effect if it's real, and a 20% chance of missing it outright.

[ THE FOUR NUMBERS THAT SET THE BAR ]

Before you run anything, four inputs determine how much data you actually need: your alpha threshold, your power, your baseline conversion rate, and your MDE, minimum detectable effect, the smallest lift you actually care about being able to see.

[ DEF.4 ]

MDE (minimum detectable effect) — the smallest lift you actually care about being able to see. It's not a property of the data, it's a choice you make going in: below this size, a real effect might exist and your test still won't reliably catch it. Set it too small and your required sample size balloons past what your traffic can deliver in a reasonable timeframe; set it too large and you'll miss smaller wins that were real.

the smallest lift you've decided is worth being able to see

Change any one of these and your required sample size moves, usually a lot more than intuition suggests. Sample size explodes as your baseline moves away from 50%, or as your MDE shrinks. Concretely, at a 10% baseline conversion rate with the standard 5%/80% thresholds:

Detecting this relative lift	Sample size needed, per arm
10% (10% → 11%)	~14,313
5% (10% → 10.5%)	~56,000
1% (10% → 10.01%)	~1,414,681

FIG.1 — going from "detect a 10% lift" to "detect a 1% lift" costs you 100x the traffic

Going from detecting a 10% lift to detecting a 1% lift costs you roughly one hundred times the traffic. This is why small-traffic products shouldn't be running color-tweak A/B tests: they simply don't have the volume to detect small effects reliably, and running one anyway doesn't make the math work, it just produces a noisy coin flip dressed up as a decision. Where does MDE actually come from in practice? There's no clean formula. In order of how people actually do it: gut feel on plausibility, historical results from similar past tests, or, the more honest method, work backwards from your constraints. How much traffic do you actually have, how many hypotheses are sitting in the backlog, how fast do you need to decide. Set your MDE from what's actually testable in your time and traffic budget, not from an abstract target someone wrote on a roadmap slide.

[ USING THE CALCULATOR: TWO EXAMPLES ]

You don't need to memorize the sample size formula to use any of this. Evan Miller's sample size calculator does the arithmetic for you: plug in your baseline conversion rate and your minimum detectable effect, and it hands you a number per variation. The one setting that trips people up is the toggle between Absolute and Relative. Absolute means "detect a change of this many percentage points" (a 20% baseline with a 5-point absolute MDE means detecting 20% → 25%). Relative means "detect a change of this many percent of the baseline itself" (a 20% baseline with a 10% relative MDE means detecting 20% → 22%, since 10% of 20 is 2). Relative is almost always the more honest way to think about it, because the same absolute point-move means something very different at a 2% baseline than at a 50% one. Here are two examples run through the same tool, same 5%/80% thresholds, to show how differently the same math treats two real spots on a storefront.

Example 1: top of the funnel. Say you're changing the color of the main call-to-action button on the homepage, high-traffic real estate that basically every visitor sees. Baseline click-through is 20%, and you want to detect at least a 10% relative lift, meaning you'd notice a move to roughly 22% or higher.

FIG.2 — 20% baseline, 10% relative MDE → 6,347 per variation

That's 6,347 per variation, 12,694 total. If the homepage does something like 3,000 visits a day, split 50/50, that's 1,500 per arm per day, so you'd clear the bar in well under a week. The strategy here is straightforward because the traffic supports it: pick a real, meaningful MDE, run a standard fixed-horizon test, wait for the number, done.

Example 2: deep in the site, low traffic. Now say the change is on an advanced export feature buried three clicks into account settings, the kind of thing only a small slice of engaged users ever reaches. Baseline conversion on the action you're watching (say, clicking "export") is 3%, and because the effect would need to be large to matter at this baseline, you set a generous 20% relative MDE, detecting a move to roughly 3.6% or higher.

FIG.3 — 3% baseline, 20% relative MDE (already a generous ask) → 13,050 per variation

Even with an MDE more than double the first example's, you need 13,050 per variation, more than double the homepage number, because the low baseline is working against you the whole time. If that settings page gets 300 visits a week, that's 150 per arm per week. Reaching 13,050 per arm at that rate takes roughly 87 weeks, about a year and a half. That's not a test, that's a career milestone.

The two examples share a calculator and a formula. They don't share a strategy. For the homepage button, run the standard test, it's cheap and fast. For the deep feature, a fixed-horizon 50/50 split is very likely the wrong tool entirely: either loosen your MDE further until the sample size matches your actual traffic (accepting you can only detect huge swings), or stop trying to force a classic A/B test onto traffic that can't support one, and reach for one of the alternatives from the next post in this series instead, a long-running hold-out, a qualitative read, or a Bayesian approach that doesn't demand a pre-fixed sample size to say something useful early.

[ THE PROBLEM: PEEKING ]

Here's where the dashboard-refreshing habit turns into an actual statistical problem, not just an impatience problem. Refreshing the page doesn't corrupt the underlying data, the numbers are whatever they are regardless of who's watching, but it does change your stopping rule. "Stop and declare a winner the moment p < 0.05" isn't one test, it's a test you get to retry every time you check.

And "check" has a specific meaning here: computing the p-value and using it to decide whether to stop, not just glancing at a chart. Opening the dashboard to see where the numbers stand is harmless on its own. It's treating that p-value as a decision point, "is this under 0.05, should I call it," that counts as a look.

Your 5% alpha threshold is a guarantee about exactly one such decision point, made after your pre-calculated sample size is reached. Every time you check the result before that point and treat "p < 0.05 right now" as a stopping signal, you're giving yourself another shot at the threshold, another roll of the dice for a false alarm, no new data required. Do that ten times over the course of an experiment and your real false-positive rate isn't 5% anymore. It's roughly 4x that, around 20%.

Here's the simple version of why. A single 5%-threshold look means a 5% chance of a false alarm, and a 95% chance of correctly seeing nothing when there's nothing there. That 95% is your "safe" probability for one look. Now take another look. If those looks were fully independent, both looks correctly showing nothing would need 95% and then 95% again, multiplying down toward zero the more times you check, ten independent looks would already put you near a 40% false-alarm rate.

Real peeking lands lower than that naive multiplication, because your looks aren't independent, each one is checking the same accumulating data as the last, just with a few more rows added. But the direction holds and the size is still large: more looks, more chances to get unlucky, and nobody told the dashboard to warn you which chance you're on.

FIG.4 — your chosen 5% only holds if you look exactly once

Intuitively: a p-value during a running test isn't a fixed property of your data, it's a noisy quantity that jumps around as samples accumulate. It can dip under 0.05 early by pure chance, climb back over, dip again. If you stop the moment it happens to be favorable, you're selectively sampling the noise, not the effect. The p-value only reflects the error rate you actually chose once the pre-calculated full sample size is reached. Everything before that is a preview, not a verdict, no matter how green the dashboard looks.

This doesn't mean the textbook rule, "never look, always wait for full sample size," is the only correct answer. It's correct if you need that specific 5%/20% guarantee, which you usually do in regulated or high-stakes contexts. It's adjustable if you're willing to trade away some of that precision on purpose: agree on a looser threshold up front, or use a testing method actually designed to handle repeated looks (more on that in a minute). What's not adjustable is pretending you didn't peek when you did, and reporting the original 5% anyway.

[ WHY IT KEEPS HAPPENING ]

This isn't a knowledge problem. Most people who work near experimentation can recite "don't peek" if you ask them directly. It keeps happening anyway, because the incentives in the room are almost never aligned with statistical patience.

A test trending toward a strong result after one week, against a five-week pre-registered horizon, puts real pressure on the room: waiting the textbook full duration for rigor has an actual cost in forgone revenue or forgone learning speed. A test trending the wrong direction creates the mirror pressure: cutting losses early feels responsible, and also inflates your error rate exactly the same way a favorable early peek does. Shipping pressure doesn't care which direction the needle is moving. It pushes toward stopping early either way.

The organizational failure mode compounds from here. A single test tells you a difference was detected, not the true magnitude of that difference. Teams that build quarterly OKRs by summing the observed uplifts of "winning" tests are stacking a series of noisy, possibly-inflated point estimates and treating the sum as a forecast. It rarely survives contact with the next two quarters. The statistics didn't fail here. The organization asked the statistics a question they were never built to answer.

[ ALTERNATIVES, BRIEFLY ]

None of this means fixed-horizon testing is broken or that peeking is unforgivable. It means the standard frequentist test answers one specific question, "is there a detectable difference at a preset error tolerance," and if that's not actually the question you're trying to answer, there are other tools built for the question you do have.

Sequential testing and always-valid p-values (mSPRT, group sequential designs) are built specifically to let you look as often as you want without the error-rate blowup, by spending your error budget across looks instead of assuming a single look. Bayesian A/B testing reframes the whole question away from p-values entirely, toward "what's the probability B is actually better, and by how much," and it doesn't require a fixed sample size to interpret at all. Both of these are real, useful, and deserve more than a paragraph each, which is exactly why they're getting their own posts in this series rather than a rushed footnote here.

[ THE MANAGERIAL PROBLEM ]

Even once the math is right, there's a second problem that's arguably harder: communicating a probabilistic result to a stakeholder who wants a yes-or-no answer by end of day.

"We're 87% confident this is an improvement, with a plausible range of 0 to 4%" is a true, useful sentence. It is also not the sentence most rooms want to hear. The fix isn't to round it down into a fake yes-or-no, it's to make the underlying decision structure explicit before you ever run the test, so uncertainty has somewhere to land. A workable hypothesis has four parts: if we do X, then Y will change by Z, because W. X is the change, Y is the metric you're watching, Z is the effect size you actually expect (which is also what feeds your sample size calculation), and W is your causal reasoning for why you expect it. W is the part almost everyone skips, and it's the part that compounds: a single test only tells you that something changed. Your working theory of why is what turns one test into a body of knowledge instead of an isolated anecdote you can't reuse.

[ WHAT WE ACTUALLY DO NOW ]

The practical version of all of this, generalized, not tied to any one team's exact setup:

Pre-register your sample size before you look at results, driven by your actual alpha, power, baseline, and MDE, not by vibes
If you must look early, decide that up front and either use a method designed for it (sequential testing, Bayesian) or explicitly accept a looser real error rate
Run continuous A/A tests on any traffic that isn't allocated to something real, they're nearly free and they catch splitter and tracking bugs before those bugs corrupt an actual test
Write the X/Y/Z/W hypothesis down before launch, so "why" survives even when the test result is ambiguous
Treat a null result as information, not failure, log it and move to the next hypothesis instead of quietly re-running until something turns green

None of this eliminates uncertainty. It just makes sure the uncertainty you're shipping with is the uncertainty you actually chose, instead of one that quietly grew while nobody was looking.

[ USEFUL LINKS ]

Evan Miller's A/B testing sample size calculator — the fastest gut-check on whether you have anywhere near enough traffic before you run anything
"The ASA Statement on p-Values: Context, Process, and Purpose" (Wasserstein & Lazar, The American Statistician, 2016) — the closest thing statistics has to an official correction of the p-value misunderstanding in this post, worth searching up directly
mSPRT / "always-valid p-values" (Johari, Koomen, Pekelis, Walsh) — the paper behind modern sequential testing at scale, more detail in the next post in this series

[ ELSEWHERE ]

GitHub · LinkedIn · Dev.to · Substack · 4thwithme.dev/blog

May the --force be with you. See you next week.

My dev setup

4thwithme — Sun, 23 Aug 2026 22:04:53 +0000

[ HARDWARE ]

[ Ecosystem ]

Let's start from ecosystem. Windows, Linux, and macOS all have their strengths, but my main requirements are:

Stability and reliability, so I can focus on work instead of rebooting or troubleshooting Wi-Fi, Sound, or Bluetooth drivers - so Linux is out.
Well-supported, and well-documented UNIX development environment - so Windows is out.

So macOS it is.

[ Hardware Requirements ]

Let's go to the hardware requirements. I need a machine that can run multiple tools at once without slowing down, and that can handle large codebases and datasets. Supporting Machine Learning and AI workloads is also a plus.

In 2026, to run all the tools I need, 16GB RAM is the absolute minimum.
SSD storage is a must for fast boot times and quick access to files.
A high-resolution display is important for productivity.
GPU support is a plus for machine learning and AI workloads.
Long battery life is important for working on the go.

Right now I work on a MacBook Pro 14" with M4 Pro chip and 24GB RAM, which fulfills all the requirements above. I also have a 4K external monitor, mechanical keyboard, and mouse for a more comfortable and efficient workflow.

Split is roughly 60% external monitor + mechanical keyboard + mouse
40% laptop as-is. No trackball, no split keyboard — I have many sides, but not that many.

[ Karabiner ]

Where the OS actually gets tuned is keyboard shortcuts.
It lets me add custom shortcuts, remap keys, and create complex modifications. For example, I have a rule that allows me to switch between English and Ukrainian keyboard layouts with a single keypress.

Language switching is bound directly. No cycling, no counting presses — you press the key for the language you want and you're already there.

Ctrl+E jumps straight to [E]nglish
Ctrl+U straight to [U]krainian.

FIG.1 — one keybind, one destination — no rotation involved

{
  "description": "Switch to English input source with Control + E",
  "manipulators": [
    {
      "from": {
        "key_code": "e",
        "modifiers": { "mandatory": ["control"] }
      },
      "to": [{ "select_input_source": { "language": "en" } }],
      "type": "basic"
    }
  ]
}

[ Mission Control ]

The same logic runs the desktop layout. I keep six Spaces, each hard-assigned to one thing. Ctrl+1 - Ctrl+6 jump straight to the matching Space. No "next/previous desktop" cycling here either, same reasoning as the language switch: a direct destination beats a direction you have to count.

browser
IDE - my main editor, where I do the bulk of my coding and debugging
terminal - my main workspace, where I run Claude Code, git, and other CLI tools
free space for DB/Redis GUIs and OrbStack - my dev tools, local servers, and containers
Slack - work chat, async comms, and incident response
Obsidian - my 2nd brain, where I keep my notes and plans.

FIG.2 — Ctrl+[number] beats Ctrl+left/right, every single time

Ctrl+A pulls up the Mission Control overview when I actually need to see all six at once. Every window fills its Space at 100%, but I keep them in windowed-fill mode rather than native macOS fullscreen — same usable area, but the menu bar and dock stay one motion away instead of a whole gesture away.

FIG.3 — six desktops, six fixed jobs — the same one is always in the same place

[ TERMINAL ]

My main working tool and my main terminal is Warp. No tmux, no zellij, no herdr, no multiplexer at all, even though I've used a few of them and they all worked fine. But we are living in 2026, not in the wild 2010s. Warp already has tabs, workspaces, and project-grouped sessions natively, with an AI layer on top. I have no nostalgia for multiplexers, sorry ;)

FIG.4 — tabs grouped by project, not by panes I have to remember the layout of (branch names blurred, not a leak)

One deliberate exception: Neovim gets its own terminal app entirely — Rio or Ghostty, kept separate from Warp. Not because Warp can't run it fine, but because I don't want my editor's session sharing chrome, history, and AI panels with everything else I'm doing. It's not the 90s, my machine can afford a second terminal app just so nvim gets a clean box to live in.

[ EDITOR, BRIEFLY ]

Three editors plus one thing that isn't really an editor. We will dive deep into Claude Code and nvim in future posts. In the meantime, here's the rough split of my time:

Neovim gets 20% of my time.
Zed - 15% of time, because it's fast and occasionally I have a mood for it.
My old buddy VS Code - 15%. It was my first IDE and some habits just don't leave.
The other 50% is Claude Code, and for that half of my week I don't open an editor at all. The work happens in the terminal, in diffs, in review — the editor becomes optional infrastructure instead of the default starting point - this is the reason why terminal has to be modern, fast, and AI-enabled.

FIG.5 — nvim, mid-config — fittingly, the plugin on screen is the one that hides my secrets

Almost all my IDEs are configured with the same philosophy: minimal, fast, and keyboard-driven. I don't need a million panels and toolbars, no terminal embedded in the IDE, no DB connectors, no Git GUI, no debugger panels — I have other tools for all of that. I just need a fast editor that can handle large files and projects. If I cannot find a plugin or theme that fits my needs, I write one myself. I have a few of those.

[ AI TOOLING ]

I've been using Claude Code for almost two years now — long enough that opening a traditional IDE feels like the exception, not the default. I run it straight in the terminal, doing the actual work: planning, multi-file changes, the stuff that used to mean opening an IDE and doing it by hand. To stay aware of what a session is actually doing, I use the ccstatusline plugin, which keeps session cost, model, and context usage visible so I'm not flying blind on a long-running task. GitHub Copilot's inline autocomplete still runs alongside it for the moments I'm typing directly — the two aren't competing, they're solving different-sized problems: Copilot finishes your line, Claude Code finishes your ticket.

[ THE ACTUAL WORKFLOW ]

My working day:

Review Claude Code — whatever overnight/async sessions did while I wasn't looking, leaving feedback or new instructions where needed.
Then Slack: blockers, incidents, and whatever business or engineering questions came in overnight.
Then New Relic and Rollbar for anything performance- or error-shaped, plus a daily AI-generated report that's already combed through the logs for me so I'm reading a summary instead of raw noise.
Then PR review. I prefer to do it in the browser default GitHub format. When I see GitHub, my brain switches to "code review mode" and I can focus on the diff, and it is easier for me to think about bottlenecks, edge cases, and the overall quality of the code.
Then JIRA for tickets, planning, and triage.
Then finally I check my own plan for the day written in Obsidian — which is a whole separate post, because "second brain" is a real thing, and it deserves a real writeup.

[ EVOLUTION ]

My setup has been stable for years at its core, but I'm not precious about it — I'll happily give a new tool a real two-to-three month trial to see if it actually brings something new and cool to the workflow. Most of them don't survive contact with what I already have. Raycast got a fair shot and never earned a place — Spotlight already does what I need, and a second launcher just for the sake of having a "better" one wasn't worth the context-switching. Same story with a git GUI — not VS Code's, not LazyVim's — terminal git stays, mostly because I trust what I can see happening over what a panel summarizes for me. My latest churn: Rio → Ghostty → Rio, right back to where I started after giving Ghostty a genuine try. That's the pattern — I'm open to switching, the current setup just keeps winning.

[ ELSEWHERE ]

GitHub · LinkedIn · Dev.to · Substack · 4thwithme.dev/blog

May the --force be with you. See you next week.

Starting my weekly blog (ML, agentic dev)

4thwithme — Mon, 17 Aug 2026 10:01:42 +0000

Post source: https://4thwithme.dev/blog/intro/

Nine years ago I was calculating neutron flux in a reactor core simulation. Today I spend my mornings in 1:1s and my evenings building MVPs, and ML recommendations models. Somewhere between those two sentences is the short version of how I got here. The reactor, for the record, did not explode.

I'm Andrii - an Engineering Manager based in Barcelona, currently leading a team of engineers at a promo-products ecommerce company. My path here was not exactly linear, it started with a physics degree instead of a computer science one: a Master's in Nuclear Power Engineering, a couple of years as a neutron physics engineer running Fortran and Python for reactor calculations, which is where the programming itch actually started, at the same time a few years teaching robotics and programming to kids who debugged with more patience than most senior engineers I've since worked with. Then a full pivot into software, no looking back, mostly because there was nothing to look back at except radiation shielding calculations.

Who I am

If you want the honest ratio: I'm about 70% engineer, 30% manager. I still want to be in the code, still want to know why the model regressed and not just that it did, and I'd rather spend an evening tinkering with a training run or working through the math behind it than sit through another roadmap slide. Management is real work and I take it seriously, but the itch that got me here in the first place is still curiosity about how things actually work, down to the math.

day job

I lead the team responsible for recommendation and search on a storefront that has strong opinions about branded mugs, the kind of place you order 200 branded pens from and never think about again until the next company retreat. Lately my job is less "ship the feature" and more "figure out how a whole engineering org adopts AI tooling without either turning it into a personality or pretending it doesn't exist." Both camps are loud. I try to be the annoying third option that just ships things.

outside the day job

I'm an AI/ML enthusiast in the fullest, most unqualified-to-stop-talking-about-it sense, hands-on with code, and hands-on with hardware too. I like tinkering with models, sitting with the math until it actually clicks instead of just pattern-matching the notation, and reading research papers slower than is efficient because skimming defeats the point. I write Neovim plugins nobody asked for, I'm working through computer vision fundamentals, and I have a Three.js habit that has produced zero shipped products and several very nice spinning cubes.

Building / learning

Computer vision, slowly, deliberately, mostly on Sundays
The math under the models I use daily, not just the API surface
A Neovim plugin nobody asked for (ss.nvim)
An unreasonable Three.js habit with a 0% shipped-product conversion rate
Whatever this blog forces me to actually finish, since public commitment works better than a private todo list

Why this blog

"Cold start" is a recommender-systems term. No data on a new user or item, and you still have to guess something useful, confidently, with nothing to go on. That's also just what most Mondays feel like. So the rule is simple: one honest post a week, no excuses, a threat I'm making primarily to myself, and no quiet retirement of this blog in week six like every New Year's resolution I've ever made.

Expect posts on ML, agentic development, Claude Code, nvim, new tools I'm evaluating, and the occasional controversial take I'll defend confidently for about a week.

Elsewhere

GitHub · LinkedIn · Substack · 4thwithme.dev/blog

May the --force be with you. See you next week.