DEV Community: jguillaumesio

My fine-tuned model scored 100%... The benchmark was lying

jguillaumesio — Tue, 11 Aug 2026 20:09:08 +0000

I fine-tuned Mistral 7B on my laptop to detect personal data in log lines and support messages. On my first test set it scored 100%. Perfect. Every single line classified correctly.

I did not publish that number, because the same test set gave few-shot prompting 94%, and a six-point gap over a prompt you can write in five minutes is not a reason to fine-tune anything. The honest conclusion looked like "this was a waste of an afternoon."

Then I threw my test set away and rebuilt it from real public data. The fine-tune dropped to 95%. Prompting collapsed to 66%.

Same model, same code, same training recipe. A 6 point gap became a 29 point gap, and the conclusion flipped completely. My benchmark had been choosing my answer for me, and it had chosen wrong.

This article is the whole run: what LoRA actually does, how to build a dataset that does not lie to you, the exact commands, and the measured results. Everything is reproducible from the repository on any Apple Silicon Mac.

Is LoRA real fine-tuning?

Worth settling first, because "LoRA is not really fine-tuning" comes up constantly.

It is. LoRA trains the model with gradient descent on your data exactly like full fine-tuning. The difference is which weights move. Instead of updating all 7 billion parameters, it freezes them and learns two small low-rank matrices per targeted layer. Their product approximates the weight update that full fine-tuning would have made, and it can be merged back into the base weights afterwards, giving you a genuinely different model.

In my run, that meant 0.145% of the parameters were trainable: 10.5 million out of 7.25 billion. That is the entire reason this fits on a laptop, and why the output is a 42 MB adapter file instead of a new 4 GB model.

What it is not is prompting or retrieval. The weights actually change. The honest caveat is that full fine-tuning can push a bit further on hard tasks, at ten to a hundred times the memory cost. For teaching a model a format, a taxonomy, or a behaviour, LoRA is what practitioners actually ship, and it is what the fine-tuning APIs from the major providers run under the hood.

The task and the hardware

The model gets one line of text, in English or French, and must answer with strict JSON:

{"pii": true, "types": ["email", "name"]}

Six types: email, phone, name, iban, address, dob. Empty list when there is nothing.

This is a real problem, not a toy. Personal data leaks into logs, staging dumps and exports far beyond your users table, which I wrote about in PII and data masking, and knowing where it is is the precondition for retention and deletion that actually works.

Hardware: a MacBook with an Apple M5 and 16 GB of unified memory. Base model mlx-community/Mistral-7B-Instruct-v0.3-4bit, running on MLX, Apple's array framework. No cloud, no API keys, total cost 0 EUR.

The part nobody writes about: the dataset

Every tutorial shows you mlx_lm.lora --train. That command is four minutes of work. The dataset is the other three hours, and it is where the result is actually decided.

Attempt one, and why it lied

I generated 800 examples from my own templates: log lines with fake emails, support messages with fake IBANs, plus hard negatives full of UUIDs and invoice numbers so the model could not just flag anything that looks like an identifier.

It gave the fine-tune a perfect score. The reason is obvious in hindsight: the test set came from the same templates as the training set. The model had seen every sentence shape before. I was measuring memorisation of my own imagination.

If your fine-tune scores 100%, your test set is too easy. That is not a nice problem to have, it is a broken measurement.

Attempt two: real data, and a licensing trap

So I went looking for public corpora, and immediately walked into two problems that are worth more than the rest of this article.

The obvious choice is ai4privacy/pii-masking-200k, which everybody cites. Its licence is dual: free for individuals, non-profits, and companies with three staff or fewer, paid otherwise. A blog that markets a consulting practice is commercial use. Depending on your situation you may be fine, but "everybody uses it" is not a licence.

For negatives I grabbed LogHub, 32,000 lines of real production logs from 16 systems. Two problems. Its licence covers "research or academic work" only. And, far worse for an article about detecting personal data, several of those systems contain real personal data:

BGL exposes real researcher account names in /home/ paths from national laboratories
Mac logs contain a real email address and a real home directory
Linux and OpenSSH carry real usernames from live attack traffic

My screening regex caught four email addresses and missed all of it, because a username inside a file path does not look like contact data. I had built a training set that teaches a model that real people's names are not personal data. That is worse than no model at all.

Both sources were thrown out. The final pipeline uses two Apache-2.0 corpora: kiji for positives, which covers all six types including IBAN and date of birth in English and French and ships its own train/test split, and witfoo syslog for negatives, 155,000 lines of real firewall and system logs.

The shortcut that would have faked the score again

Here is the trap that would have quietly ruined everything. Positives are business prose. Negatives are raw syslog. A model can separate those two by writing style and score brilliantly without ever learning what personal data is.

So every quadrant has to exist:

	Positive (has PII)	Negative (no PII)
Prose	kiji sentences	same sentences, PII replaced by role words
Log	real syslog with real PII injected	real syslog, untouched

The prose negatives are built by rewriting each annotated span into a generic role word, so "contact Alice Dupont at alice@example.fr" becomes "contact the customer at the support address". Same sentence structure, same vocabulary, same language, no personal data.

Then I found the leak inside my own fix. Filler phrases like "the customer" appeared only in negatives, so they became a perfect giveaway. The model could learn my filler vocabulary instead of the task. So 10% of the rows are hybrids: role words everywhere, except one real value left in. Now the filler carries both labels and is useless as a signal.

Four defects in the public data

Public datasets are not clean. Screening mine turned up:

Blanket filtering costs three quarters of the corpus. Most kiji rows mention a passport or an SSN somewhere, types I do not model. Dropping those rows left 4,180 usable. Replacing just those spans with role words instead left 15,810.

The annotations are incomplete. Kiji has a coreferences field, and it is empty on every single row. So a later mention survives: "Alice Dubois" is annotated, but "Dubois a signé" three sentences down is not, and that real surname lands inside a "no personal data" example. Fix: screen every negative against a 1,519-token name vocabulary harvested from the corpus itself.

4.7% of the French rows are encoding-corrupted. Accented characters arrive as NUL bytes, so étude is stored as \x00tude. Train on those and you are teaching the model mojibake French. 406 rows dropped.

4% of the syslog carries account names in prose, like Accepted password for johndoe. Too few to be worth parsing, so they are dropped wholesale rather than mislabelled as clean.

Final dataset: 8,000 train, 448 validation, 800 test. 4,572 positives against 4,676 negatives. 2,544 French rows. Zero overlap between splits, verified.

The Mistral gotcha that costs you an afternoon

My first smoke test crashed:

jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...

Mistral's chat template rejects a standalone system role. Unlike Llama or Qwen, it wants strictly alternating user and assistant turns. So the instructions have to ride on the first user turn:

{
  "messages": [
    {"role": "user", "content": f"{SYSTEM_PROMPT}\n\nLine: {text}"},
    {"role": "assistant", "content": '{"pii":true,"types":["email"]}'}
  ]
}

The dangerous part is not the crash. It is that if you fix this in your evaluation script and forget your dataset builder, training and inference use different prompt shapes, and your adapter looks broken for no visible reason. Build both from the same function.

Training

python -m mlx_lm lora \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --train --data ./data \
  --fine-tune-type lora \
  --batch-size 4 --num-layers 16 --iters 500 \
  --learning-rate 1e-5 --max-seq-length 512 \
  --mask-prompt --grad-checkpoint \
  --steps-per-report 50 --steps-per-eval 250 --save-every 500 \
  --val-batches 25 --seed 42 \
  --adapter-path ./adapters 2>&1 | tee training.log

Why these values:

--mask-prompt is the one you must not skip. It computes the loss on the answer only, not on the input line. Without it, most of the tokens the model is learning to predict are the log line itself, which is not the task.

--batch-size 4 --num-layers 16 came from measurement, not guesswork. A first run at batch 1 with 8 layers peaked at 4.8 GB on a 16 GB machine, so there was room to double both.

--learning-rate 1e-5 is the consensus for small datasets. 1e-4 oscillates, 1e-6 barely moves.

--grad-checkpoint trades compute for memory, and --save-every matters because you want the best checkpoint, not the last one.

The loss curve is the interesting part:

Iteration	Validation loss
1	3.537
250	0.508
500	0.491

Almost everything happens in the first 250 iterations. The next 250 bought a 3% improvement, so I stopped there rather than running the 2,000 I had planned. If you take one operational lesson: watch validation loss and stop when it flattens, because "train longer" is mostly a way to spend electricity.

Final cost: 45 minutes, 6.0 GB peak memory, a 42 MB adapter, 0 EUR.

Results

Every mode uses the same prompts, the same test set and temperature 0. "Few-shot" means six worked examples in the prompt, which is what a sensible engineer tries before reaching for training.

400 held-out rows

Metric	Zero-shot	Few-shot (6)	LoRA
Accuracy	66%	66%	95%
Precision	0.639	0.610	0.926
Recall	0.686	0.840	0.974
F1	0.662	0.707	0.950
False positives	75	104	15
Missed PII	61	31	5
Valid JSON	100%	100%	100%
Seconds per line	0.83	1.67	0.91

Look at few-shot's precision: adding six examples made it worse than zero-shot, 104 false positives against 75. It found more personal data and cried wolf far more often.

The per-type breakdown shows why:

Type	Zero-shot	Few-shot	LoRA
email	0.745	0.782	1.000
phone	0.628	0.575	0.983
name	0.531	0.663	0.855
iban	0.358	0.194	0.950
address	0.500	0.597	0.914
dob	0.383	0.366	0.875

IBAN is the story. Six examples cannot teach a model the boundary between an IBAN, an invoice reference and a whsec_ webhook secret across two languages. Few-shot scores 0.194 there, worse than saying nothing. The fine-tune reaches 0.950, because 4,000 examples of that boundary is what it takes.

30 hand-written lines the model has never seen

I also wrote 30 lines by hand, in phrasings the corpora never produced, as a final honesty check:

Metric	Zero-shot	Few-shot	LoRA
Accuracy	80%	90%	100%
False positives	2	1	0
Missed PII	4	2	0

The fine-tune got all 30. Small sample, so I would not put "100%" on a slide, but it did not fall apart off-distribution, which was the real question.

It is also cheaper to run

The fine-tune is 1.8x faster per line than few-shot, 0.91 seconds against 1.67. The six examples are gone from the prompt, so every single inference is shorter, forever. Better and cheaper is a rare combination.

When you should not do this

Fine-tuning teaches behaviour, format and taxonomy. It does not teach facts.

If your problem is "the model does not know our internal documentation", fine-tuning is the wrong tool and retrieval is the right one. Facts change; weights do not. You will retrain forever and still get confident wrong answers.

And prompting deserves a fair trial first. On my synthetic dataset it genuinely tied the fine-tune. It only collapsed when the task got hard enough that six examples could not express the rules. That threshold is the actual decision point, and you cannot find it by reading blog posts, including this one. You find it by measuring both, which costs an afternoon.

The lesson that generalises

The technical work here was easy. MLX is excellent, the command is one line, and it ran on a laptop while I did something else.

The hard part, and the part that decided the outcome, was the data: the licence that quietly excludes commercial use, the "clean" log corpus full of real usernames, the empty coreference field, the NUL-corrupted French, and the style shortcut that would have handed me a beautiful meaningless number.

I got two completely different answers to the same question, on the same model, on the same day. The only thing that changed was the quality of what I measured against. Before you trust any fine-tuning result, including the ones in this article, ask what the test set is made of.

Reproduce it

Everything is public, including the training log and the raw predictions:

git clone https://github.com/jguillaumesio/lora-pii-detection-mlx
cd lora-pii-detection-mlx
python3 -m venv .venv && .venv/bin/pip install mlx-lm datasets
.venv/bin/python build_dataset.py

The dataset is not committed. It rebuilds from the two Apache-2.0 corpora with that one command, so nothing is redistributed that should not be. The 30 hand-written test lines, the results and the training log are all in the repo.

Originally published on jguillaumesio.com. I write about payments, AI agents in production, and running SaaS infrastructure without a platform team.

GDPR, PII and data masking: how to stop leaking personal data everywhere

jguillaumesio — Sun, 09 Aug 2026 09:34:20 +0000

The hard part of protecting personal data is not the users table. Everyone knows that holds PII. The problem is everywhere else it quietly ends up: an email address in an error log, a full production database copied into staging, a customer name in an analytics event, a phone number in a support export, a request body pasted into an LLM prompt.

PII (personally identifiable information) leaks by default, because copying data is convenient and nobody stops to ask whether the copy still needs to be personal. This is a practical guide to masking it: the techniques, where to apply each, and the one distinction that decides whether you have actually reduced your risk or just moved it.

This pairs with the guide on GDPR data retention and deletion. That one is about how long you keep data and how to delete it. This one is about making the copies you do keep stop being personal data in the first place.

Usual disclaimer: I am an engineer, not a lawyer. This is implementation guidance, not legal advice.

The distinction that actually matters

Three words get used interchangeably and are not the same thing. Getting them straight is the whole game, because they have different legal consequences under GDPR.

Masking / redaction: replacing the value with something non-sensitive, usually irreversibly, for display or storage. john@acme.com becomes j***@acme.com or [REDACTED]. The original is gone from that copy.
Pseudonymization: replacing identifiers with a reversible token, where a separate key can map back to the real value. john@acme.com becomes user_8f3a, and a lookup table or key can reverse it. GDPR encourages this, but it is critical to understand: pseudonymized data is still personal data. You still hold the mapping, so it is still in scope for retention, erasure, and breach rules.
Anonymization: transforming data so that no one, including you, can link it back to a person, even by combining it with other data. Truly anonymized data is no longer personal data and falls outside GDPR. This is much harder than it sounds, because re-identification from "anonymous" datasets is a well-documented attack.

The trap is to do pseudonymization, call it anonymization, and assume you are out of scope. You are not. If you kept a way back, it is pseudonymization, and the data is still regulated. Only reach for "anonymized" when you have genuinely thrown away the ability to re-identify.

Where to mask: static vs dynamic

There are two moments you can mask, and they solve different problems.

Static masking transforms data at rest, producing a sanitized copy. This is the one for non-production environments. The classic GDPR failure is cloning the production database into staging so developers have "realistic data", which means every engineer, every CI run, and every less-secured staging box now holds real customer PII. Instead, mask on the way out:

-- build a staging dump with names and emails masked
UPDATE users SET
  email = 'user' || id || '@example.com',
  full_name = 'User ' || id,
  phone = NULL;

Better still, generate a synthetic seed that looks real but describes no actual person, so staging never contains production PII at all. Either way, real production data in a non-production environment is the single most common avoidable exposure.

Dynamic masking transforms data at read time, based on who is asking. A support agent sees **** 4242, a payment service sees the full number. The stored value is intact; the masking happens in the query layer or the API response. Use this when different roles legitimately need different views of the same record, and never rely on the frontend to do it: mask server-side, because anything the client hides, the client can also reveal.

Masking techniques, and their pitfalls

Not all masking is equal, and the naive version breaks things:

Random masking replaces each value with a random one. Safe, but it destroys relationships: the same user gets different masked values in two tables, so your staging data no longer joins.
Deterministic masking maps each input to the same output every time (usually a keyed hash). john@acme.com always becomes user_8f3a, everywhere. This preserves joins and uniqueness, which is what makes masked data actually usable. The catch: deterministic masking is pseudonymization, not anonymization, and it is vulnerable to correlation, so keep the key secret and treat the output as still-personal.
Format-preserving masking keeps the shape so validators and column types still pass: a masked credit card is still 16 digits, a masked email still has an @. Necessary when downstream code checks format.

Do not log PII in the first place

Logs deserve their own section because they are the leak nobody notices until a breach. An email in a stack trace, an IP in an access log, a full request body captured "temporarily" for debugging: all of it is personal data, sitting in a system designed to be append-only and widely readable.

You will not run per-user deletes across a log firehose, so the control has to be at write time:

Structured logging with an allowlist. Log explicit fields, never whole objects. logger.info({ userId, action }), not logger.info(user), which dumps the email, name, and everything else the moment someone logs the object.
A redaction layer. Most logging libraries support redaction paths that strip known-sensitive keys (password, email, authorization, token) before anything is written. Configure it once, centrally.
Detection for free text. Structured redaction misses PII embedded in free-form strings (an email typed into a message body). For that, a detection library like Microsoft Presidio can find and redact common PII patterns (emails, phone numbers, names) before logging or before data leaves your system.

The cheapest control by far is not writing the PII down. Redaction after the fact is always more expensive and never complete.

A practical approach

You do not need a data-masking platform to start. In order of impact:

Stop cloning prod into non-prod. Mask the dump, or seed synthetic data. This kills the biggest exposure immediately.
Redact PII at the logging layer. Allowlist fields, strip sensitive keys, add free-text detection where you log user content.
Use deterministic masking when masked data must stay joinable, and remember it is still pseudonymized (keep the key safe).
Mask dynamically for role-based views, always server-side.
Be honest about anonymized vs pseudonymized. If you kept a way back, it is still personal data, with all the retention and erasure duties that implies.

None of this is exotic. It is mostly deciding, per copy of the data, whether that copy needs to identify a real person, and if not, removing the ability for it to do so. The systems that get breached are rarely breached through the users table. They are breached through the fourth staging box and the two-year log archive that everyone forgot were full of real people.

Originally published on jguillaumesio.com

GDPR data retention and deletion: a practical guide for developers

jguillaumesio — Sun, 09 Aug 2026 09:32:40 +0000

Most GDPR work that lands on engineers is not about cookie banners or privacy policies. It comes down to two duties that are easy to state and hard to implement: do not keep personal data longer than you need it, and delete it properly when you are supposed to.

Both are retention and deletion problems, and both are where real systems quietly fall out of compliance. This is a practical guide to getting them right, including the part almost every plan forgets: your backups.

One disclaimer up front. I am an engineer, not a lawyer, and this is implementation guidance, not legal advice. The exact retention periods and legal bases for your product are a question for your DPO or counsel. What follows is how to build the machinery once those decisions are made.

Duty one: storage limitation (stop keeping data forever)

GDPR's storage limitation principle says personal data should be kept only as long as it serves the purpose you collected it for. In engineering terms: every category of personal data needs a defined lifetime and something that enforces it.

The trap is that systems default to keeping everything forever. Disk is cheap, deletes feel risky, and "we might need it later" wins every argument. So the first, unglamorous step is a retention schedule: a table of what personal data you hold, why, and for how long.

Data	Purpose	Retention
Account profile	Provide the service	Life of account + 30 days
Order history	Legal and tax obligation	6 to 10 years (jurisdiction dependent)
Support messages	Customer service	2 years
Server and access logs	Security and debugging	30 to 90 days
Analytics events	Product analytics	14 months

The retention column is a legal decision. The point for engineers is that once it exists, it becomes a spec you can implement and test against.

Enforcing it: automated purging

A retention policy that relies on someone remembering to delete things is not a policy. Automate it:

A scheduled job (nightly or weekly) that hard-deletes records past their retention window.
For time-series data (logs, events, analytics), use the storage engine's own TTL or partition dropping rather than row-by-row deletes. Drop the old partition, expire the index, set the bucket lifecycle rule. It is faster and it cannot silently miss rows.

-- example: purge support messages older than 2 years
DELETE FROM support_messages
WHERE created_at < now() - interval '2 years';

Soft delete is not deletion

A common mistake: marking a row deleted_at = now() and calling it done. A soft delete hides data from the application, but the personal data is still there, fully readable in the database. That is fine as an intermediate state (a grace period, an undo window), but it does not satisfy retention or erasure on its own. Something has to hard-delete it eventually.

Duty two: the right to erasure

When a user exercises their right to erasure (the "right to be forgotten"), you generally have 30 days to actually remove their personal data. The difficulty is almost never the main users row. It is that personal data has spread.

By the time a product is real, a single user's data lives in many places:

the primary database (and its read replicas)
a search index (Elasticsearch, Algolia, OpenSearch)
caches (Redis, a CDN)
object storage (avatars, uploads, exports)
the analytics pipeline and warehouse
third-party processors (email, payments, support, error tracking)
log storage
backups

Erasure means cascading the delete across all of these, not just the row. Two things make this survivable:

A data map. You cannot delete what you have not written down. Maintain a list of every system that stores personal data and how to delete from each. This is the same map your retention schedule needs.
Delegate to your processors. For third parties, you usually do not delete their copy yourself, you call their deletion API or rely on their contractual retention. Stripe, your email provider, your error tracker each have a deletion mechanism. Erasure includes triggering theirs.

A clean implementation is an erasure workflow: one job that fans out deletion to every system on the map, records what it did, and is idempotent so it can be re-run if one downstream call fails.

The part everyone forgets: backups

Here is the question that stops most erasure plans cold. You have deleted the user from production, from the search index, from object storage, from every processor. But last night's database backup still contains them. So does the one from the night before, and every one going back weeks. Have you actually complied?

You cannot realistically open each backup, surgically remove one user, and repackage it. That would defeat the purpose of backups and risk corrupting them. Regulators know this, and the accepted engineering answers are these:

Bounded backup retention. If your backups rotate on a defined schedule (say 30 days) and then are destroyed, the deleted user's data ages out of the backup set within that window. Document this. "Backups are retained for 30 days, after which erased data is permanently gone" is a defensible, common position. The key is that the window is finite and enforced, not "we keep backups forever."
Crypto-shredding. Encrypt each user's personal data with a per-user key. To erase the user, you delete their key. The ciphertext may still sit in old backups, but without the key it is unrecoverable, which for practical and regulatory purposes is deletion. This is the strongest approach and the one to reach for when your backup retention is long or legally required to be.
A restore-and-re-erase procedure. Accept that backups still contain the data, and commit in writing that if you ever restore a backup, you immediately re-run the erasure job for anyone who requested deletion in the meantime. Keep the list of erasure requests so this is possible. Weaker than the first two, but honest and workable.

The wrong answer is to have no answer. "We delete from the database" while unbounded backups quietly retain everyone forever is the gap auditors and breach investigations find first.

Do not forget your logs

Logs are the most overlooked store of personal data. An email address in an error message, an IP in an access log, a full request body captured during debugging: all of it is personal data, subject to the same retention and erasure duties, and scattered across a log system that is designed to be append-only.

You will not run per-user deletes across a log firehose, so handle logs at the source:

Do not log PII in the first place. Redact emails, tokens, and request bodies before they are written. This is by far the cheapest control.
Set a short retention. 30 to 90 days covers almost all security and debugging needs. Configure it and let old logs expire automatically.
Treat "we can search two years of logs" as a liability, not a feature.

A short checklist

A written retention schedule: every category of personal data, its purpose, its lifetime.
Automated purging that enforces the schedule (TTLs, partition drops, scheduled jobs).
A data map: every system that stores personal data and how to delete from each.
An idempotent erasure workflow that cascades across all of them, including processors.
A documented, defensible answer for backups: bounded retention, crypto-shredding, or restore-and-re-erase.
No PII in logs, and short log retention.

None of this is exotic engineering. It is mostly bookkeeping (knowing where the data is) plus a few scheduled jobs. The reason it so often fails is not technical difficulty, it is that nobody owns the map. Write the map down, automate the deletes, and decide your backup story on purpose rather than by accident. That is the difference between a system that is compliant and one that merely looks compliant until someone asks.

Originally published on jguillaumesio.com

Docker never restarted my database: the price of restart: no

jguillaumesio — Sun, 09 Aug 2026 09:30:56 +0000

Something died in the middle of the night. I found out in the morning, from a user, not from a machine.

The container had exited. It stayed exited. Docker had watched it die and done nothing, precisely because I had told it to do nothing. The line responsible was three characters long, repeated on every service in my compose file.

This is part 5 of a series on hardening a solo-built SaaS in production. The setup and the full list of weak spots are in the pillar article. This one is about the difference between a container that crashes and a system that recovers.

The line, on all seven services

restart: no

Every service had it: nginx, the API, Postgres, Redis, MinIO, the dashboard build, the workspace helper. Seven out of seven.

restart: no is Docker's default, and it means exactly what it says. If the process exits, for any reason, the container stays dead. There is no supervisor, no retry, no backoff. It also means something worse that I had not thought through: if the VPS reboots, nothing comes back up. The machine boots, the Docker daemon starts, and every container sits there stopped, waiting for me to SSH in and run make up. A host reboot after a kernel update would have taken the whole product offline until I noticed by hand.

For a laptop, restart: no is the sane default: you do not want yesterday's experiment relaunching itself when you reboot. On a production server, it is a decision to have no automatic recovery at all.

The part that surprises people: I had healthchecks

Here is what makes this worth writing about. It was not that I had no monitoring at the container level. Two services had real, well-written healthchecks:

db:
  healthcheck:
    test: ['CMD-SHELL', 'pg_isready -U ${DB_USER} -d ${DB_NAME}']
    interval: 10s
    timeout: 5s
    retries: 5

api:
  healthcheck:
    test: ['CMD', 'bun', '-e', "fetch('http://localhost:8000/health')..."]
    interval: 30s
    timeout: 10s
    retries: 3
    start_period: 40s

And the API even waits for the database properly, which is the right pattern:

api:
  depends_on:
    db:
      condition: service_healthy    # not just "started"

So why did none of that help? Because of a fact that catches almost everyone coming from Kubernetes:

In plain Docker Compose, a failing healthcheck does not restart anything. It flips the container's status to unhealthy and stops there. There is no liveness probe semantics, no automatic replacement. Unlike a Kubernetes liveness probe or Docker Swarm, Compose has no built-in actor that reacts to unhealthy.

So a healthcheck without a restart policy produces a container that is beautifully, accurately labelled as broken, that nobody restarts and nobody is watching.

Two mechanisms, two different failure modes

Once you see it this way, the design becomes obvious. They are not alternatives, they cover different failures:

Failure	What catches it
The process crashes or exits	Restart policy (`restart:`)
The process is alive but wedged, deadlocked, or not serving	Healthcheck plus something that acts on it

I had the second half of the second row missing, and the entire first row missing. Which is how a crash at 3am became a morning outage.

Choosing a restart policy

There are four values, and only one is usually right for a production service:

Policy	Restarts on crash	Survives host reboot	Respects a manual stop
`no`	no	no	n/a
`on-failure[:N]`	only on non-zero exit	no	yes
`always`	yes	yes	no, it comes back after a deliberate stop
`unless-stopped`	yes	yes	yes

unless-stopped is the one you want for long-running services. It restarts the container on crash, brings it back after the host reboots, and still lets you deliberately docker compose stop a service without Docker second-guessing you the next time the daemon starts. always looks similar but will resurrect a container you intentionally stopped, which is maddening during an incident.

db:
  restart: unless-stopped
api:
  restart: unless-stopped
nginx:
  restart: unless-stopped

Note that this applies to long-running services only. One-shot containers (a build step, a migration runner, the workspace helper that just runs an install) should stay no, because restarting a job that already completed is not recovery, it is a loop.

Making "unhealthy" actually do something

Restart policies handle crashes. They do nothing for the nastier case: the process is still running, so Docker is happy, but it stopped serving. That is what healthchecks detect, and in Compose you need to add the actor yourself. Two honest options:

A small autoheal sidecar: a container with access to the Docker socket that watches for unhealthy status and restarts those containers. It is a handful of lines in the compose file and it closes the gap without introducing an orchestrator.
External monitoring that alerts you, so a human decides. Slower, but it also catches the failures a restart cannot fix (a full disk, a dead dependency, an expired certificate).

The second one is not optional even if you add the first, because "restart it" is not always the right answer, and a service that silently restart-loops all night is its own kind of outage. Getting that visibility is the next article in this series.

While you are in there, close the smaller gaps too. In my file, only the database and the API had healthchecks: Redis, MinIO and nginx had none. And nginx used the short form of depends_on, which only waits for the other containers to have started, not to be healthy:

# before: nginx starts as soon as api exists, ready or not
nginx:
  depends_on: [storage, api]

# after: nginx waits until they can actually serve
nginx:
  depends_on:
    api:
      condition: service_healthy
    storage:
      condition: service_healthy

What this does not fix

Restart policies keep you up through crashes and reboots. They do not give you zero-downtime deploys. My deploys still run docker compose up -d --force-recreate, which stops the container and starts a new one, so every deploy is a few seconds where requests hit nothing. That is a different problem with a different fix (build the image elsewhere, then swap), and it belongs with the deploy pipeline, alongside the day my build filled the disk.

Being honest about the boundary matters: unless-stopped turns "down until I wake up" into "down for a few seconds". That is a huge win, and it is not the same as high availability. On one VPS, there is still exactly one of everything.

The takeaway

restart: no is the right default on a laptop and a decision to have no recovery on a server. If your production compose file has it, a crash at 3am is an outage until you notice, and a host reboot is a full outage until you SSH in.

Set unless-stopped on every long-running service. Give every service a healthcheck, not just the database. Use condition: service_healthy in depends_on so things start in an order that actually works. And remember the part that trips people up: in Compose, a healthcheck only labels the problem. Something else has to act on it, whether that is an autoheal sidecar or an alert that reaches you.

Originally published on jguillaumesio.com

Shipping a SaaS on one VPS with Docker Compose: an honest audit

jguillaumesio — Sun, 09 Aug 2026 09:29:37 +0000

The entire product runs on a single VPS. One machine, one docker-compose.yml, a containerised nginx in front, and a make deploy I run over SSH. It serves real, paying customers, and it has never been down for more than a few minutes. It is also, by any serious production standard, held together with tape.

I built this SaaS end to end for a small agency: a marketplace that connects brands with creators. Solo. The infrastructure reflects that. Every choice optimised for "one person ships this next week," not "this survives a datacentre fire." This article is the honest audit: what the setup is, why it was the right call at launch, and a full map of everything that can take it down.

It is also the start of a series. Each single point of failure below becomes its own article, built around the real incident that forced me to fix it. This is part 0: the state of the system before any of that hardening.

Why one VPS and Docker Compose was the right call

I want to defend this setup before I tear it apart, because the instinct to reach for Kubernetes on day one is how solo projects die. You spend your runway operating a cluster instead of shipping the features that decide whether the product lives.

One VPS and one compose file buys you the thing that actually matters early: a single mental model. Everything lives in one file. One machine you can SSH into and reason about end to end. And Docker Compose is not a toy: it gives you named networks, service dependencies, healthchecks, and log rotation out of the box. For a product finding its first customers, that is genuinely enough.

The mistake is not starting here. The mistake is staying here without knowing exactly where the sharp edges are. So here they are.

The actual architecture

Six services, one network, nginx as the only public entrypoint:

# docker-compose.yml (trimmed and anonymised)
services:
  nginx:      # reverse proxy + TLS, the only thing exposed to the internet
    image: nginx:1.24
    ports: ['80:80', '443:443']
    restart: no
  api:        # Bun + Express, Postgres + Redis/BullMQ
    build: { context: ., dockerfile: apps/api/Dockerfile }
    ports: ['8000:8000']
    restart: no
  db:         # Postgres 15, data in a local bind mount
    image: postgres:15
    ports: ['127.0.0.1:5432:5432']
    volumes: ['./postgres/data:/var/lib/postgresql/data']
    restart: no
  redis:
    image: redis:7
  storage:    # MinIO, S3-compatible object storage
    ports: ['9000:9000', '9001:9001']   # 9001 is the admin console
    restart: no
  dashboard:  # React + Vite, built to static files served by nginx
    build: { context: ., dockerfile: apps/dashboard/Dockerfile }

The dashboard is a React single-page app served as static files. The API is a Bun and Express monorepo talking to Postgres, Redis, and a BullMQ queue. MinIO holds uploads. Persistent data (Postgres, MinIO) sits in bind mounts on the host filesystem.

Read that compose file with a critical eye and the problems are already visible. Two are worth flagging now because they are the good habit and the bad habit sitting side by side. Postgres binds to 127.0.0.1:5432, so it is not reachable from the internet: that is the right reflex. The MinIO admin console on 9001 is published to every interface: that one is a hole, and it even carries a TODO in the real file. More on that in the security article.

How a deploy works today

There is no pipeline. A deploy is me, on the server, typing:

# on the production server, over SSH
git pull
make deploy SERVICE=api
# which runs, roughly:
#   docker compose build api           # builds the image ON the prod box
#   docker compose up -d --force-recreate api

Two things are wrong with this, and both get their own article. The image is built on the same machine that is serving live traffic, so the build competes with customers for CPU, RAM, and disk. And --force-recreate means the container stops and restarts, so every deploy is a few seconds of downtime with nothing catching the requests in between.

It works. It is also exactly the kind of "works until it doesn't" that this series is about.

The map: every single point of failure

None of these are hypothetical. Each one bit me, and each one is the hook for its own article.

The weak spot	Why it hurts	Covered in
Build runs on the prod box	The build filled the disk and took production down	Article 1: the disk that killed prod
Docker and the host firewall disagree	Docker writes its own iptables rules, and a port I thought was blocked was open to the internet	Article 2: Docker bypasses your firewall
Tests exist but never run	Real regressions shipped while the unit tests sat unused in the repo	Article 3: tests nobody ran
Nothing protects `main`	A Friday `git push` deployed straight to prod, untested, with downtime	Article 4: ban commits on main
`restart: no` on every service	A crash at night was never restarted, including the database	Article 5: when Docker watches prod die
No status view, logs only over SSH	During an incident I was grepping JSON in a terminal to find out what broke	Article 6: a status dashboard and centralised logs
Errors die in a log file	Users reported 500s by email before I ever saw them	Article 7: real error tracking
Constant attacks, a certificate that silently expired	Scans and brute-force all day, plus a renewal cron that ran a broken path	Article 8: CrowdSec, and Cloudflare as the next layer
Bind mounts, no backup	A `pgdata` directory got wiped, with no volume and no backup behind it	Article 9: backups that actually restore
One machine, and a client request	The client wanted preprod wired to production data	Article 10: when the client wants prod data in preprod

Where this series goes

The thread through all ten is the same: harden a solo-built SaaS without reaching for Kubernetes and without blowing a small agency's budget. Everything stays on a VPS you can afford. The goal is not a perfect platform. It is a production you can sleep through.

Next up is the one with the best story and the sharpest lesson: the day a docker compose build filled the disk and took the whole thing down.

Originally published on jguillaumesio.com

Docker bypassed my firewall: the database port was open to the internet

jguillaumesio — Sun, 09 Aug 2026 09:27:40 +0000

I had a firewall. I had explicitly blocked the Postgres port on the host. And Postgres was still reachable from the internet.

I found out the way you never want to: the database was taking connection attempts from IP addresses that had no business talking to it. My firewall rules said port 5432 was closed. A scan from a machine outside the server said it was wide open. Both were telling the truth, and the gap between them is one of the most common production traps in the Docker world.

This is part 2 of a series on hardening a solo-built SaaS in production. The setup and the full list of weak spots are in the pillar article. This one is about the day I learned that Docker does not care about the firewall I thought I had.

Why the firewall did nothing

Here is the mental model most people have, and why it is wrong. You picture traffic hitting the host, going through your INPUT chain, and getting dropped by your rule. Tools like ufw reinforce this: you type ufw deny 5432, it says the rule is active, and you move on.

But a published Docker port does not travel through INPUT. When you publish a port, Docker rewrites the firewall for you. It adds a DNAT rule in the nat table that redirects incoming traffic to the container, and it accepts that traffic through the FORWARD chain, because from the kernel's point of view the packet is being routed to another network (the container's), not delivered to the host itself.

Your ufw deny and your handwritten INPUT ... DROP rules govern the INPUT chain. Docker's published ports are handled in FORWARD. The two never meet. So the firewall was doing exactly what I told it: dropping traffic to the host on 5432. Docker was quietly forwarding traffic to the container on 5432, through a completely different path, and it wins because it runs first.

The convenience that makes Docker pleasant (publish a port and it just works) is the exact mechanism that punched a hole through my firewall without a word.

The one binding that was already right

There is a good habit in the compose file, and it is worth calling out because it is the cheapest fix of all. The database publishes its port like this:

db:
  ports:
    - '127.0.0.1:5432:5432'   # loopback only, not the whole internet

That 127.0.0.1: prefix matters enormously. Without it, - '5432:5432' tells Docker to publish on 0.0.0.0, every interface, including the public one. With it, Docker's DNAT rule only matches traffic that arrived on loopback, so nothing from outside the machine can reach it. This is the binding that closed the hole.

The honest part: the same file still publishes other services on all interfaces.

api:
  ports: ['8000:8000']          # 0.0.0.0, exposed
storage:
  ports: ['9000:9000', '9001:9001']   # object storage AND its admin console, exposed

So the database is fixed, but the API on 8000 and the MinIO admin console on 9001 are still published to the entire internet. Which leads to the better question.

The real fix: internal services need no published ports at all

Publishing a port is for traffic that comes from outside the machine. Look at what actually needs that: nothing except nginx. The reverse proxy already reaches every other service by name over the Docker network:

# nginx talks to the containers directly, over the internal network
proxy_pass http://api:8000;
proxy_pass http://storage:9000;

nginx does not use the host-published 8000 or 9000. It uses Docker's internal DNS and the shared network. Which means the ports: entries on api, storage, and the dashboard are not just risky, they are pointless. Nothing outside the compose network uses them.

So the fix for most services is to delete the port publishing entirely. The only container that should publish to the host is nginx, on 80 and 443. Everything else talks over the internal network, invisible from outside:

nginx:
  ports: ['80:80', '443:443']   # the only public surface
api:
  # no ports: block at all, nginx reaches it as http://api:8000 internally
db:
  ports: ['127.0.0.1:5432:5432'] # loopback only, for local psql/backups

If you genuinely need a port reachable from outside but restricted (say, Postgres from one office IP), do not reach for ufw. Put the rule in the DOCKER-USER chain, the one iptables chain Docker guarantees it evaluates before its own rules and never flushes:

# only this source may reach forwarded container traffic on 5432
iptables -I DOCKER-USER -p tcp --dport 5432 ! -s 203.0.113.10 -j DROP

Audit from outside, never trust `iptables -L`

The lesson underneath all of this: your firewall config is a claim, not a fact. iptables -L and ufw status show you what you intended, through the chain you were looking at. They will happily show 5432 as denied on INPUT while it is wide open through FORWARD.

The only source of truth is a scan from a machine that is not the server:

# from your laptop, not from the box
nmap -Pn -p 5432,8000,9000,9001 your.server.ip

If anything other than 80 and 443 comes back as open, you have a service exposed that you probably did not mean to expose. That single command would have told me about the database months before an attacker did.

The takeaway

Docker manages the firewall for you, and it does not consult the one you wrote. Publishing a port silently adds rules in FORWARD and nat that your INPUT and ufw rules never touch, so "the port is blocked" and "the port is open" can both be true at once. Bind anything you must publish to 127.0.0.1, publish nothing that nginx can proxy internally, use DOCKER-USER for real restrictions, and verify with an external scan rather than a local list.

Closing the ports is only half of perimeter security. The other half is what to do about the traffic that is still allowed to hit nginx: the constant scans and brute-force, and the day my TLS certificate silently failed to renew. That is a later article in this series.

Originally published on jguillaumesio.com

The docker build that filled the disk and took down production

jguillaumesio — Sun, 09 Aug 2026 09:26:20 +0000

Production went down, and I had not shipped any code. No traffic spike, no bad migration, nothing in the application logs. The API just stopped answering, and so did every container on the box.

The cause was mundane and entirely self-inflicted: the disk was full. And the thing that filled it was my own deploy process, building Docker images on the exact machine that serves customer traffic.

This is part 1 of a series on hardening a solo-built SaaS in production. The setup and the full list of weak spots are in the pillar article. This one is about the first weak spot that bit me: building where you serve.

What "deploy" meant that day

There is no pipeline. A deploy is me, over SSH, running make deploy. Stripped down, the target does this:

# Makefile, roughly
docker compose build api                      # build ON the prod box
docker compose build --no-cache dashboard     # and this one with no cache
docker compose up -d --force-recreate --no-build
docker image prune -f                          # (added later, see below)

Read that with fresh eyes and the problem is right there. Every deploy builds images on the production host. The build pulls base layers, compiles the app, and writes new image layers, all onto the same disk that Postgres, MinIO uploads, and the container logs are living on. The build is a heavy, bursty batch job sharing a disk with a latency-sensitive service that must never stop.

But the resource contention was not what took prod down that day. Something quieter did.

Why the disk filled, and what cleared it

Here is the mechanism I had not thought about. docker compose build does not update an image in place. Each build produces a brand new image and moves the service's tag to it. The previous image loses its tag and becomes a dangling image, still on disk, still taking space, tagged <none> and referenced by nothing.

Deploy after deploy, those orphaned <none> images piled up. The dashboard, built with --no-cache, wrote a fresh full set every single time. Nothing was cleaning them, so they grew until the disk hit 100 percent and the kernel started refusing writes. Postgres cannot write, so the API errors, so everything falls over.

The thing that saved me was exactly the command people underestimate:

docker image prune -f    # removes dangling <none> images

docker image prune -f removes dangling images, and dangling <none> images were precisely what had filled the disk. It reclaimed the space and brought production back in minutes. That is the whole reason that line is in my deploy today: I added the docker image prune -f at the end of make deploy after this incident, so the images the build orphans get cleared on every deploy instead of piling up.

Diagnosing it: two commands

When a Linux box misbehaves in a way the app logs cannot explain, check the disk before anything else:

df -h                    # is the filesystem full? almost always the answer

If Docker is the culprit, this is the command that shows you where the space went:

docker system df         # summary: images, containers, volumes, build cache
docker system df -v      # per-item breakdown, marks reclaimable space

The -v view is the one that matters. It lists images individually and splits "active" from "reclaimable". The reclaimable images, the dangling <none> ones, were the space my deploys had been leaking. Seeing that stack of untagged images is what told me the deploy, not the data, had filled the disk.

One dangerous reflex to avoid

If the build cache has also grown, you can reclaim more, but be precise, because the aggressive commands can delete more than you mean to:

docker image prune -af      # all unused images, tagged or not
docker builder prune -f      # the build cache

What I did not run is docker system prune --volumes. That flag also removes volumes. On this setup the data lives in bind mounts rather than named volumes, so it would not have been caught, but the reflex is dangerous: on a box that does use named volumes, that one flag deletes your database. Never reach for --volumes on a production machine you have not audited. (That the data was in bind mounts with no backup is its own problem, and its own article later in this series.)

The one good habit already in place

Credit where it is due: the log side of this was already handled. Every service in the compose file caps its logs:

logging:
  driver: json-file
  options: { 'max-size': '10m', 'max-file': '3' }

Without that, Docker's default json-file logs grow unbounded and are a classic way to fill a disk on their own. So the logs were not the problem. The orphaned images were. It is worth saying because "the disk is full" has several possible causes, and capping logs removes one of them cleanly. If your compose file does not do this, add it now.

The real fix: stop building where you serve

Auto-pruning after every deploy stops the disk from filling again, and it is the right immediate fix. But it patches one symptom of a deeper problem: I am still building images on the machine that serves customers. That has costs the prune does not touch. The build competes with live traffic for CPU, RAM, and disk while it runs. A deploy that fails before it reaches the prune line still leaves its garbage behind. And --force-recreate still means a few seconds of downtime on every deploy.

The clean answer is to take the build off the box entirely: build the image in CI, push it to a registry, and have the server only ever pull a finished image. The prod machine goes back to doing one job, running containers, and never sees a build again. That is a bigger change (it comes with banning direct pushes to main and an actual deploy pipeline), so it gets its own article later in this series.

The takeaway

The outage was my deploy process leaving orphaned <none> images on the machine it deployed to, one per build, until the disk was full. docker image prune -f clears exactly those, and wiring it into every deploy is the five-minute fix everyone building with Compose should have. But the deeper lesson is that a production box should not be building images at all.

Check docker system df -v right now. If you see a stack of dangling <none> images, your deploys are leaking disk, and a prune (ideally automatic) is one command away. The clean version of the fix, build in CI and pull on the server, is coming up in this series.

Originally published on jguillaumesio.com

CPU utilization lies: autoscaling a single-threaded service

jguillaumesio — Sun, 09 Aug 2026 09:24:26 +0000

The service was slow. Not down, just slow: p95 latency climbing well past where users notice, requests piling up, the kind of degradation that generates support tickets instead of alerts. And the autoscaler, the whole point of which is to add capacity when a service is under strain, sat there doing nothing.

The metric it was watching said everything was fine. Average CPU utilization on the tasks was hovering around 30 percent, nowhere near the scale-out threshold. The dashboard was calm. The users were not. Both were right, and the gap between them is one of the most common autoscaling traps on a container platform.

This is the first article in a series on running a multi-tenant SaaS on AWS at team scale. It is about a metric that lies, quietly, by design.

Why 30 percent CPU meant 100 percent busy

The service was a single-threaded application. A Node.js API, in this case, but the same is true of any process that does its real work on one thread: a classic Python or Ruby worker, most single-process runtimes.

A single-threaded process can, by definition, saturate exactly one CPU core. The task it was running on had four vCPUs. So the arithmetic that matters is brutally simple:

one core fully pegged / four vCPUs on the task = ~25% task-average CPU

At full saturation, the busiest that process can ever make the task look is about 25 percent. Add a little async I/O overhead spread across the runtime and you land around 30 percent. That is not a service with headroom. That is a service redlining on the only core it can use, while three cores sit idle and drag the average down to a number that reads as "barely working."

The autoscaling policy was tracking average CPU across the task's cores. For a workload that can only ever use one of them, that average is not a measure of load. It is a measure of load divided by four.

The metric was answering a different question

This is the real lesson, and it is not specific to AWS or ECS. Average CPU utilization answers "how much of the machine's total compute is in use." Autoscaling needs the answer to a different question: "is this service keeping up with its work."

For a multi-threaded service that spreads across all its cores, those two questions have nearly the same answer, which is why CPU is the default scaling metric and why it usually works. For a single-threaded service, they diverge completely. The service can be fully saturated and failing its latency targets while the CPU metric, honestly and correctly, reports 30 percent.

Scaling on the wrong metric is worse than not scaling, because it comes with a dashboard that actively reassures you. Everything looks healthy right up until a human notices it is not.

Three ways to fix it

There is a quick lever and a couple of durable ones.

Size the task to one vCPU. If the process can only use one core, stop giving it four. On a one-vCPU task, "one core saturated" is 100 percent CPU, and the autoscaler can finally see the truth. This is the cheapest fix and it doubles as a cost cleanup: you were paying for three idle cores per task. Standardising the fleet to one vCPU per service is not a downgrade here, it is making the reported metric mean something again.

Scale on a signal that reflects the actual constraint. The workload is latency-bound, so scale on latency or on request pressure, not CPU. On this stack that means target-tracking on the load balancer's target response time, or on request count per target, rather than CPU utilization. A custom metric works too: for an event-loop runtime, event-loop lag is a direct measure of "this process is behind." Scale on the thing that hurts, not on a proxy that averages it away.

Run one worker per core if you must use big tasks. If a task really needs four vCPUs, then run four workers on it (cluster mode, one process per core, a process manager, or four smaller tasks instead of one big one). Now all four cores can be busy, and average CPU becomes an honest metric again. The rule underneath: CPU utilization is only meaningful when your app can actually use all the cores it is measured against.

The immediate mitigation, while you decide, is to scale out by hand: add tasks and watch latency. If more tasks fix the latency, you have just confirmed the service was capacity-starved all along, and you can lower the scaling threshold (or switch metrics) with confidence.

The takeaway

Autoscaling is only as good as the metric it watches, and the default metric quietly assumes your app uses every core it is given. A single-threaded service breaks that assumption: it maxes out one core, reports a fraction of the task's total CPU, and sails under a CPU-based scale-out threshold while its latency falls apart.

Match the metric to how the app actually consumes resources. Size single-threaded tasks so one core is the whole task and CPU tells the truth, or scale on latency and request pressure instead. And treat a calm dashboard next to unhappy users as a bug in your monitoring, not a coincidence. The metric that says everything is fine is the first thing to distrust.

Originally published on jguillaumesio.com

A practical playbook for running a remote dev team

jguillaumesio — Thu, 09 Jul 2026 16:00:34 +0000

We are eight people, fully remote: one lead, four developers, two product (one of them a designer), and one QA. We build a multi-tenant product: one shared codebase, plus features specific to individual tenants.

For a while it did not work. Bugs reached production regularly. We picked up work ad-hoc. Nobody had a clear picture of what was shipping next. And one QA cannot cover four developers on a codebase where a change for one client can quietly break another.

Then a new lead arrived. He did not rewrite our process from a book. He introduced five ceremonies, one at a time, each one aimed at a specific thing that was hurting us. This is the playbook: what each ceremony is, exactly how we run it, and what changed.

One honest caveat up front: we do not run fixed-length sprints. When I say "sprint review" below, read it as "release review." We run regular ceremonies over a continuous flow of work, not textbook Scrum. I think that is fine, and I will come back to it at the end.

Ship fewer bugs: put quality in the ticket, not just in QA

The worst bugs we shipped were per-tenant: a change for one client broke another. You cannot hire your way out of that fast enough. The only thing that scales is moving quality earlier, before a line of code is written. The industry name is "shift left," and the numbers are blunt: ambiguous or incomplete requirements are blamed for an estimated 50 to 60% of production defects, and a bug caught at the requirements stage costs a fraction of the same bug caught in production.

Requirements-phase defects (ambiguous acceptance criteria, missing edge cases, contradictory business rules) account for an estimated 50 to 60% of production defects, and teams that shift QA left reduce defect costs dramatically compared to teams that catch bugs in production.

Source: synthesised from IBM's Systems Sciences Institute data and Capgemini's World Quality Report

So we changed what a ticket must contain before anyone picks it up.

Every ticket gets acceptance criteria before it is estimated. Not a vague title. A concrete list of what must be true for the ticket to be done.

If a ticket is unclear, it gets an investigation first. We do not estimate something nobody understands. A short spike answers the open questions, and only then do we size it. This alone killed a class of "we did not realise it also touched X" bugs.

The multi-tenant rule: every ticket names the blast radius. Because our worst failures were cross-tenant, each ticket answers one question explicitly: does this touch shared code or a single tenant, and if shared, which tenants need checking? That one line turns an invisible risk into a checklist item.

Here is a template we paste into every non-trivial ticket:

## Acceptance criteria
- [ ] Given <state>, when <action>, then <result>
- [ ] Negative case: given <bad input>, then <handled how>

## Blast radius
- Touches: [ ] shared core   [ ] single tenant
- If shared, tenants to verify: <list>

## How to test
- <steps QA can follow without asking us>

Here is the part we have not fully solved, and I would rather be honest about it. Today the ticket creator writes the acceptance criteria, and we developers write the "how to test" notes. That works until it does not: sometimes those notes are thin, the ticket gets marked ready for release, and it should not have been. The root cause is not who typed the notes. It is that the author is effectively self-certifying. The person who wrote the code is the last person who should be the gate that says "this is releasable."

Where I think this goes next is a lighter version of what the industry calls the "Three Amigos": product, QA, and a developer shaping the criteria together, product owning the "what," QA owning "what could break," dev owning "what it touches." We do not run that ceremony yet. But even a lightweight Definition of Ready would help, and the release gate should be someone other than the author, QA when possible, a peer when QA is swamped.

Here is the checklist we are converging on, tick it against your own next ticket:

Acceptance criteria written and understood before estimation
Blast radius named: shared core or single tenant, and which tenants to verify
"How to test" steps a non-author can follow without asking questions
Release gate is someone other than the author

The regression net: the same cases, tested before every release

Acceptance criteria stop new bugs. They do nothing for the old feature that silently breaks when you touch shared code. For a multi-tenant product that is the most dangerous failure of all, and it needs its own guard: a fixed set of checks that runs before every release, not per ticket.

There are two acceptable versions of this, and you should be climbing from the second toward the first:

Automated end-to-end tests covering the critical paths: login, the core workflow, billing, and the per-tenant features that diverge from the shared core. Green suite or no release.
At minimum, a written list of must-pass scenarios, one set per important client, that QA runs before every release. Nothing ships until that list is green.

The multi-tenant rule applies here too: the list must include at least one scenario for every tenant whose behaviour differs from the shared core, because that is exactly where a "harmless" change detonates. And the list is a ratchet: the moment a regression slips through, its scenario gets added, so the same bug can never reach production twice.

This is the difference between "we tested the thing we changed" and "we proved we did not break the things we did not touch." Only the second one keeps a multi-tenant product stable as it grows.

What changed anyway: more tickets are genuinely ready for release, because more eyes hit them before they get there. The remaining gap is the self-certification one above, and naming it is the first step to closing it.

The retro: a Miro board, and votes that force focus

At the end of each release we open a Miro board with four columns:

What is good
What we learned
What is not going well
What we would want

Everyone adds notes, privately, at the same time. Writing before talking is the whole trick: it avoids the loudest voice setting the agenda and it avoids groupthink. Then everyone gets a small number of votes (we use three) to spend on the topics they most want to discuss. We only discuss the top-voted items. The rest are logged, not lost.

The retro is where every other ceremony on this page actually got tuned. It is also the ceremony most likely to become theatre. The single most common way retros fail is not the format, it is the follow-through:

The reason most retrospectives fail is not the meeting itself, it is what happens (or does not happen) afterwards. Action items get discussed, then forgotten by the next sprint.

Source: Easy Agile, Why Retrospectives Fail

This is exactly where we are weakest, so I will not pretend otherwise. We discuss the top items and we track them loosely, but we do not assign a named owner or turn each one into a Jira ticket. So some drift, and the same complaint resurfaces a few retros later. The fix we know we need is boring: two or three action items maximum, each with an owner and a ticket, revisited at the start of the next retro. An action with no owner did not really happen. We are not there yet, and writing it down here is partly to shame ourselves into it.

Planning poker: the size is disposable

We size tickets with planITpoker, but with one deliberate change: we do not vote in numbers. We vote in t-shirt sizes, XS, S, M, L, XL. Numbers invite false precision (is it really a 5 and not an 8?) and, worse, they invite someone to add them up into a velocity number and wave it around. Sizes resist both. "This is a Large" starts a conversation. "This is 8 points" starts a spreadsheet.

Everyone votes at the same time, hidden, then we reveal together. The blind reveal is the point: it kills anchoring, so a junior does not just echo the senior's card.

A wide spread, S next to XL, is not a failure of the estimate, that is the estimate doing its job. When the sizes disagree, someone knows something the others do not, a hidden dependency, an edge case, a tenant that behaves differently. The conversation that follows is the actual deliverable.

I want to be fair to the critics here, because they are the reason we dropped numbers in the first place. There is a loud, credible backlash against story points:

I don't like story points because they give a false sense of precision, and the moment management starts treating them as a productivity metric, they are weaponised against the very developers who produce them.

Source: paraphrasing Tomasz Lakomy, Why I don't like story-point-driven estimates, and Scott Logic, Story points are wasting time

They are right about the failure mode. Points become poison the instant a manager turns them into a burndown scoreboard. T-shirt sizes are our defence: there is nothing to sum, nothing to chart, nothing to weaponise. The size is disposable, the discussion is the deliverable. We use the vote to trigger a conversation, and then we mostly forget the letter.

The daily: coordination, not a status report

Fifteen minutes, same time every day, everyone on camera. Here is what it is for, and what it is not.

It is not a status report to the lead. The dailies people rightly hate are the ones where each person recites yesterday's tasks to a manager who writes them down. If that is all your daily is, it should have been an async written update, a Slack message. The critics are right about that. Keep the synchronous time for the blockers. So that is what each person actually covers: what is blocking me, and what I am about to touch, especially if it is shared code. Not a recital of yesterday's tickets.

Being fully remote, we felt the pull toward async written standups, and for pure status they would be better. But we kept the live daily for one reason it does well: coordination and surfacing blockers in real time. It is where "I am about to touch the shared billing code" meets "wait, so am I." That collision does not happen in a threaded update. So the rule is: keep it short, no solving problems in the meeting, take the deep dives to a breakout right after.

Less ad-hoc drift is one of the payoffs, not the biggest but a real one: because we sync every morning, fewer people quietly start random work, and everyone has a rough idea of what is being picked up. The bigger win is still the collisions we avoid by saying out loud what we are about to touch.

The sprint review: shared visibility, not a demo

Call it a release review in our case, since we do not run fixed sprints. The trap everyone falls into is treating this as a demo: a parade of every task, performed for an audience. Ours has no external audience at all, no clients, just the developers, product, and QA, which is exactly why we can optimise it for shared understanding instead of performance. Scrum's own guidance pushes back on the demo trap too:

The Sprint Review is not a demo. It is a working session to inspect the outcome and figure out what to do next. If it is a one-way presentation, you are doing it wrong.

Source: Scrum.org, Sprint Review: Much More Than Just A Demo

For us the real value is not the demo at all, it is shared visibility. We walk the whole internal team through what is going into the next release and what is queued behind it. Three things fall out of that:

Everyone learns the whole product. Because we discuss many tickets, including ones we will not personally pick up, more people understand more of the system. New joiners ramp faster.
Fewer PRs stuck in the review queue. When everyone already has context on a change, reviews are quick instead of archaeological.
We catch duplicates and collisions. Discussing tickets together is when someone says "that is the same as the thing I did last month" or "those two will conflict."

We do keep a separate slot for real demos: a Friday session you sign up for to show something you shipped. Honestly, it is our weakest ceremony. It tends to run long, not many people sign up, and half the time the interesting work never gets shown. The visibility review above is what earns its keep. The demo is still looking for its format.

None of this is clever. The reason shared visibility beats a polished demo is that the demo optimises for looking productive to outsiders, while the review optimises for the team actually knowing what is happening. The first is theatre. The second is the point.

The actual lesson

None of these five ceremonies is novel. You have heard of all of them. The thing worth copying is not the ceremonies, it is that each one was introduced to fix a specific, named pain:

Bugs in prod → acceptance criteria and blast radius in the ticket.
Ad-hoc chaos → the daily as coordination.
Estimates nobody trusted → blind planning poker in t-shirt sizes, with the size thrown away.
No shared picture → the review as visibility, not demo.
Everything drifting → the retro, with owned action items.

We do not do textbook Scrum. No fixed sprints, no formal Three Amigos, and one honest gap where authors still self-certify their own releases. That is fine. The goal was never to run the ceremonies correctly. It was to stop shipping bugs and start knowing what the team is doing. Pick the pain you actually have, and add the one ceremony that targets it. Then run the retro, and let it tell you what to add next.

Originally published on jguillaumesio.com

The cart timer expired while they were paying. Now what?

jguillaumesio — Fri, 03 Jul 2026 12:02:39 +0000

Here is a bug report I have received in some form at three different companies. The customer opens the payment widget, enters their card, and hits pay. While the bank is doing its 3-D Secure dance, the cart's countdown timer hits zero. The frontend, doing exactly what it was told, tears down the session. The payment succeeds a second later against a cart that no longer exists. Now there is money in the account and no order attached to it.

Every engineer who built that timer built it correctly. The countdown protects a limited resource: a concert seat, a hotel night, the last unit in stock. Holding it forever would let one abandoned tab starve every other customer. So you set a timer, and when it fires you release the hold. Clean, defensible, and completely blind to the fact that a real person had already committed their money.

That blindness is the actual subject here. The countdown-expiry bug is small. The habit that produces it is not.

The timer answers a question nobody asked at that moment

Step back and ask what the countdown is for. It exists to resolve contention: two people want the same seat, and the hold decides who gets first refusal. That is the only job. It is a fairness mechanism between customers who have not yet paid.

The moment a customer enters the payment flow, the question the timer answers stops being relevant. Contention is over. This person is not browsing, not hesitating, not sitting on a tab they forgot about. They are actively handing you money for the exact thing the hold was protecting. Firing the timer now does not serve any other customer, because no other customer can be served: the seat is about to be sold. It only serves the abstraction.

This is the tell. An engineer thinking in systems sees a timer that reached zero and a rule that says "release on zero". An engineer thinking about the product sees a person who did everything right and is about to be punished for the bank's latency. Same event, two completely different readings, and only one of them keeps the customer.

So, do you let them finish?

Yes. Almost always, yes. If the payment authorizes, honor it.

The reasoning is not sentimental. A successful authorization is the strongest possible signal of intent, far stronger than the "still holding" state the timer was guarding. Rejecting a payment you already accepted creates the worst outcome for everyone: you now have to refund, the customer sees a charge-then-refund cycle that reads as "this company is sketchy", and you have burned trust to enforce a rule that protected nobody. Forcing that refund by letting a timer kill a live payment is a self-inflicted wound.

The one case where you genuinely cannot honor it is true oversell: while this customer was in 3-D Secure, the last unit actually went to someone who completed faster. That is a real conflict, not a timer artifact. Handle it as an inventory failure with an immediate, automatic reversal and an honest message, not as a "your time ran out" error. The customer's experience should be "we were one second too slow and here is your money back instantly", never "you were too slow".

The two follow-up questions

Once you accept that a live payment should win, two design questions follow, and they are where most implementations quietly go wrong.

Do you re-lock the offer right after they paid? No. This one trips people up because the instinct is to keep the state machine consistent: the hold expired, so on the way out we should re-acquire it. But re-locking something the customer already bought is nonsense. They own it now. Payment is the terminal state, not another step that needs its slot reserved. If your code path re-locks after capture, you have a state machine that does not know the transaction is over, and that will produce its own class of bugs (double-holds, phantom availability) down the line.

Do you run a separate timer for the final step? Yes, and this is the actual fix. The browsing countdown and the payment window are two different clocks measuring two different things. The cart timer manages contention while the customer decides. The moment they commit to paying, you switch to a payment grace window: a separate, more generous timer whose only job is to give the authorization time to resolve. Bank redirects, 3-D Secure, wallet confirmations, slow networks: a 30-second cart hold that felt urgent while browsing is absurd once someone is staring at their banking app's confirmation screen. Give the final step its own clock, sized for how long a real payment actually takes.

Make expiry server-authoritative

The bug at the top of this article only reaches production when the timer lives in the browser and the browser is allowed to decide the session is dead. A frontend countdown is a display. It should never be the thing that releases inventory.

Put the authority on the server. The hold has an expiry timestamp the server owns. The frontend counts down for the human, but the decision to release is a server-side check that runs when it actually matters: when someone else tries to grab the seat, or when the payment webhook arrives. When the capture webhook lands, the server compares it against the hold and the inventory, and decides. A visual timer hitting zero is not an event that should mutate anything. It is a hint to the user, nothing more.

That single architectural move, holds expire on the server, the frontend only displays, eliminates the entire category. The countdown can hit zero and reset to "processing your payment" without ever touching the underlying reservation, because the reservation was never the frontend's to release.

The habit behind the bug

None of the above is hard. The reason it ships broken so often is not technical difficulty, it is that the timer was built as a pure systems problem. Resource, contention, TTL, release. Every line of that is correct and the sum of it fails a customer, because the model never included the customer's experience of time. To the system, zero is zero. To the person, they were three-quarters of the way through paying.

This is the engineer-versus-product gap, and it does not show up in the happy path. It shows up in exactly these seams: what happens when two clocks disagree, when the network is slow, when the user does the right thing at the wrong instant. Those edge cases are where the product actually lives, and they are the first thing a systems-only mindset drops, because from inside the system they look like correctly handled states, not like a human being told "too late" for something they already did.

The fix is not "care more". It is to make one question part of the definition of done for anything user-facing: what does the person on the other end experience when this fires? Ask it of the timer and you invent the grace window on your own. Skip it and you ship a state machine that is right about everything except the one thing it was for.

The rule I follow now

Any timer, lock, or expiry that a user can be actively working against needs two things: a server that owns the real deadline, and an answer to "what happens to someone mid-action when this fires". A countdown that can cancel a payment already in flight has neither. Give the final step its own clock, keep the authority on the server, and let a committed customer finish. The abstraction does not need protecting. The person does.

Originally published on jguillaumesio.com.

Multiple accounts in Claude Code: the complete setup

jguillaumesio — Thu, 25 Jun 2026 19:54:15 +0000

You have a personal Claude Code subscription and a work one. Or you freelance for two clients who each provide their own API key. Or you want a sandboxed account for experiments that won't pollute your main config.

Whatever the reason, Claude Code doesn't ship with a built-in "switch account" command. But the architecture makes it straightforward: everything lives in a single config directory, and you can redirect it.

How Claude Code stores state

Claude Code keeps all its state in ~/.claude by default. That includes:

Authentication tokens
Project-level settings (.claude/ inside each repo)
Conversation history
Memory files
MCP server configs

The key insight: the CLAUDE_CONFIG_DIR environment variable overrides where Claude Code looks for all of this. Point it somewhere else, and you get a completely independent instance.

Basic setup: shell aliases

Create one config directory per account:

mkdir -p ~/.claude-personal
mkdir -p ~/.claude-work

Find your Claude binary path:

which claude
# e.g. /Users/you/.nvm/versions/node/v22.15.0/bin/claude

Add aliases to your ~/.zshrc (or ~/.bashrc):

alias claude-personal='CLAUDE_CONFIG_DIR=~/.claude-personal claude'
alias claude-work='CLAUDE_CONFIG_DIR=~/.claude-work claude'

Reload your shell:

source ~/.zshrc

Then authenticate each one separately:

claude-personal   # run /login inside the session
claude-work       # run /login inside the session

Each alias now opens Claude Code with its own auth, memory, and settings.

What the alias approach misses

The alias trick works, but it has gaps that will bite you in production use.

Problem 1: NVM version upgrades break absolute paths

If you hardcode the binary path in your alias (as many guides suggest), upgrading Node via NVM silently breaks it. Use just claude instead of the absolute path, and let $PATH resolve it:

# fragile
alias claude-work='CLAUDE_CONFIG_DIR=~/.claude-work /Users/you/.nvm/versions/node/v22.15.0/bin/claude'

# resilient
alias claude-work='CLAUDE_CONFIG_DIR=~/.claude-work claude'

Problem 2: project-level config leaks between accounts

Claude Code creates a .claude/ directory inside your project repo. That directory stores project settings, CLAUDE.md, and settings.json. These are shared across all your aliases because they live in the repo, not in the config dir.

This means your work account and personal account see the same project-level instructions. That's usually fine, but if you need different MCP servers or permissions per account per project, you'll need to handle it differently (see the wrapper script section below).

Problem 3: hooks and MCP servers are per-config-dir

If you've configured custom hooks or MCP servers in ~/.claude/settings.json, those won't exist in your new config directories. You'll need to copy or symlink the parts you want shared:

# share hooks across accounts
ln -s ~/.claude/settings.json ~/.claude-work/settings.json

Or, if you want different hooks per account, copy and customize:

cp ~/.claude/settings.json ~/.claude-work/settings.json

Problem 4: memory doesn't transfer

Each config directory has its own memory system. Your personal account won't remember what your work account learned. If you use memory-heavy workflows, that isolation is sometimes a feature, sometimes a problem.

Better approach: a wrapper script

Instead of simple aliases, a small wrapper gives you account switching with validation:

#!/usr/bin/env bash
# Save as ~/bin/claude-switch and chmod +x

ACCOUNT="${1:?Usage: claude-switch <account-name> [claude args...]}"
shift

CONFIG_DIR="$HOME/.claude-$ACCOUNT"

if [ ! -d "$CONFIG_DIR" ]; then
  echo "Account '$ACCOUNT' not found. Available accounts:"
  ls -d ~/.claude-* 2>/dev/null | sed 's|.*/.claude-||'
  exit 1
fi

CLAUDE_CONFIG_DIR="$CONFIG_DIR" exec claude "$@"

Usage:

claude-switch work
claude-switch personal --resume
claude-switch work -p "fix the login bug"

This passes all arguments through, so flags like --resume, --print, and -p all work.

Per-project account defaults

If a specific repo should always use a specific account, you can set it in a .envrc file (if you use direnv):

# /path/to/work-project/.envrc
export CLAUDE_CONFIG_DIR="$HOME/.claude-work"

Now every time you cd into that project and run claude, it automatically uses the work account. No alias needed.

Without direnv, you can add it to the project's shell history or a local .env file that your shell sources.

API key accounts vs OAuth accounts

There are two authentication modes in Claude Code, and they interact differently with multi-account setups:

OAuth (default): You run /login and authenticate through Anthropic's web flow. The token is stored in the config directory. This is what most people use with Claude Pro/Max subscriptions.

API key: You set ANTHROPIC_API_KEY as an environment variable. This bypasses the config directory's auth entirely.

For API key setups, you don't even need separate config directories for auth. You can just switch the key:

alias claude-client-a='ANTHROPIC_API_KEY=$CLIENT_A_KEY claude'
alias claude-client-b='ANTHROPIC_API_KEY=$CLIENT_B_KEY claude'

But you'll still want separate config directories if you need isolated memory, hooks, or MCP servers.

Combining both: API key + config isolation

The most robust setup for freelancers or consultants:

# ~/.zshrc
export CLIENT_A_KEY="sk-ant-..."  # or source from a secrets manager

alias claude-client-a='ANTHROPIC_API_KEY=$CLIENT_A_KEY CLAUDE_CONFIG_DIR=~/.claude-client-a claude'
alias claude-client-b='ANTHROPIC_API_KEY=$CLIENT_B_KEY CLAUDE_CONFIG_DIR=~/.claude-client-b claude'

Each client gets:

Their own API key (billing goes to the right place)
Their own memory (client context stays separate)
Their own MCP servers (different clients, different tools)
Their own hooks (different code review standards, different workflows)

Running accounts simultaneously

You can run multiple Claude Code instances in parallel, each in its own terminal tab. The config directories are independent, so there's no locking or conflict.

# Terminal 1
claude-work

# Terminal 2
claude-personal

Both sessions run concurrently without interference. This is useful when you're waiting on a long task in one account and want to work on something else in another.

Quick reference

What you want	What to set
Different auth	`CLAUDE_CONFIG_DIR`
Different API key	`ANTHROPIC_API_KEY`
Different memory	`CLAUDE_CONFIG_DIR`
Different MCP servers	`CLAUDE_CONFIG_DIR` + custom `settings.json`
Per-project default	`.envrc` with `CLAUDE_CONFIG_DIR`
All of the above	Combined alias with both env vars

Going further: multiple Claude Desktop instances

Everything above covers Claude Code (the CLI). If you also use the Claude desktop app and want two instances running side by side with different accounts, the approach is different: you need to duplicate the app itself.

On macOS, Parallels Toolbox can create an "app shortcut" that acts as a second copy of Claude. Each copy maintains its own login session, so you can run your work account in one window and your personal account in another, without logging in and out. This walkthrough shows the full setup.

The process: open Parallels Toolbox, create an app shortcut pointing to Claude, give it a distinct name (like "Claude Work"), approve it in macOS security settings, and log in with your second account. Both instances live in your dock and run independently.

This pairs well with the CLI multi-account setup: use CLAUDE_CONFIG_DIR aliases for terminal work, and Parallels app shortcuts for the desktop GUI.

What I actually use

Two config directories: ~/.claude-personal for my own projects, ~/.claude-work for client work. Direnv handles the switching per project, so I just type claude and it picks the right account. I symlink settings.json from my personal config to the work one because I want the same hooks everywhere, but memory stays separate.

The total setup took five minutes. The part that took longest was realizing I needed to re-run /login in each config directory after creating it.

Originally published on jguillaumesio.com

Cloudflare under the hood: how it works and how attackers try to get around it

jguillaumesio — Mon, 22 Jun 2026 10:21:35 +0000

1. What Cloudflare actually is

Cloudflare is not a reverse proxy running on one server somewhere. It is a globally distributed edge network with over 300 points of presence (PoPs). When you put your domain behind Cloudflare, you are routing all traffic through that network before it ever reaches your server

The mechanism is anycast routing. Cloudflare announces the same IP address from every PoP simultaneously. When a user sends a request to your site, BGP routing automatically directs it to the closest PoP, not to your origin server. From there, Cloudflare decides what to do with it

User in Tokyo
   |
   | (anycast routes to nearest PoP)
   v
Cloudflare Tokyo PoP
   |-- cached? → serve from edge, origin never touched
   |-- blocked? → return 403, origin never touched
   |-- challenge? → run Turnstile, origin never touched
   |-- clean? → forward to origin, return response
   v
Your origin server

TLS termination happens at the edge PoP, not at your origin. Cloudflare holds the certificate, decrypts the request, inspects it, then re-encrypts it for the leg to your origin (assuming SSL between Cloudflare and origin is enabled, which it should be)

This is why Cloudflare can inspect HTTPS traffic for WAF rules without a man-in-the-middle attack: you are explicitly delegating that decryption to them

2. The layers between a request and your server

A request arriving at a Cloudflare PoP passes through several decision layers in order:

DDoS mitigation runs first. Volumetric floods are absorbed at the network layer. HTTP floods are identified by rate, pattern, and reputation

IP reputation and geofencing checks the source IP against Cloudflare's threat database. IPs from known botnets, Tor exit nodes, or datacenter ranges are scored

WAF inspects the HTTP layer: headers, path, query params, body. Cloudflare maintains a managed ruleset covering OWASP Top 10 plus known CVEs

Bot management (Turnstile is the visible part) assigns each request a bot score from 1 to 99. Score 1 is almost certainly a bot. Score 99 is almost certainly human

Cache is the last layer before origin. If the response is cacheable and a fresh copy exists at the PoP, Cloudflare serves it without touching your server

3. How Turnstile works

Turnstile is Cloudflare's CAPTCHA replacement. Unlike reCAPTCHA v2, it has no image challenge: the goal is to verify a visitor is human without making them solve anything visible

1. The widget loads a JS challenge from Cloudflare's edge. The script is different per request, not a static file you can analyze once

2. The script collects passive signals:

Timing: how long did each JS operation take? Headless browsers running at full CPU speed have suspiciously uniform timing.
Interaction: did the mouse move before the form was submitted? Did keystrokes have natural delays?
Browser fingerprint: canvas rendering, WebGL renderer, installed fonts, audio context output.
Environment: is navigator.webdriver exposed? Are dev tools open?

3. Cloudflare runs those signals through a model trained on billions of requests and issues a signed token if the request looks human

4. Your backend verifies the token against Cloudflare's siteverify API:

POST https://challenges.cloudflare.com/turnstile/v0/siteverify
{
  "secret": "your-secret-key",
  "response": "token-from-widget"
}

If your backend does not make this call, the protection is entirely client-side and trivially bypassed by skipping the form submission step

4. Finding the origin server behind Cloudflare

If an attacker finds your origin IP, they can bypass Cloudflare entirely by sending requests directly to that IP. Your WAF, DDoS protection, and Turnstile all disappear

Here are the techniques commonly used, in order of how often they succeed

SSL certificate history

Before you put a domain behind Cloudflare, it had a certificate issued directly to the origin. Certificate transparency logs are public and record every certificate ever issued:

https://crt.sh/?q=example.com

If the origin IP appeared in a certificate before Cloudflare was enabled, it is in the log forever

DNS history

Before Cloudflare, your A record pointed directly to your origin. Those records are archived by SecurityTrails, DNSDumpster, and ViewDNS.info, often with timestamps showing exactly when you switched

Subdomains not behind Cloudflare

Many teams proxy www and the apex but leave other subdomains with a grey cloud (not proxied) by accident:

ftp.example.com: legacy, often points to origin
dev.example.com, staging.example.com: forgotten
api.example.com: sometimes bypasses the proxy for latency reasons

A subdomain enumeration pass reveals which subdomains resolve to a non-Cloudflare IP

MX records

Mail servers cannot be proxied through Cloudflare. Your MX record points directly to a mail server, often on the same IP block as your web server:

dig MX example.com        # → mail.example.com
dig A mail.example.com    # → 203.0.113.42

SPF records

SPF records list every IP authorized to send email on your behalf. They often include your origin server or hosting provider's IP range:

dig TXT example.com
# v=spf1 ip4:203.0.113.0/24 include:sendgrid.net ~all

Shodan + certificate fingerprint

If your origin uses a Cloudflare origin certificate, its fingerprint is the same regardless of how it is accessed. Shodan and Censys index TLS certificates across the entire IPv4 space: search for your cert fingerprint to find the raw IP

5. Bypassing Turnstile

Solving services

2captcha, Anti-Captcha, and CapSolver use human workers who run a real browser session and return the token. This works but is slow (seconds per token) and costs money per solve. Practical at low volume, expensive at scale

Headless browser spoofing

Playwright and Puppeteer combined with stealth plugins patch the detectable properties:

navigator.webdriver set to undefined
Spoofed canvas fingerprint
Realistic mouse movement and keystroke timing
Full Chrome user agent

A well-configured headless browser can pass Turnstile at a reasonable rate. Cloudflare's model is continuously updated, but it is an ongoing arms race

What actually stops most bots

The visible Turnstile widget is not the main defense. Cloudflare's bot score from network-level signals (IP reputation, ASN, request rate, TLS fingerprint) catches far more traffic than the JS challenge does. A request from AWS Lambda with a clean User-Agent still has a datacenter ASN: that alone raises the bot score before any JS runs

Turnstile alone, validated client-side only, is weak. The combination of network scoring plus behavioral analysis is what makes the system effective

6. How to actually protect your origin

Use Cloudflare Tunnel. This is the only approach that fully hides your origin IP. cloudflared opens an outbound connection from your server to Cloudflare's network. No open inbound ports, no IP to find.

cloudflared tunnel create my-tunnel
cloudflared tunnel route dns my-tunnel example.com
cloudflared tunnel run my-tunnel

If you cannot use Tunnel, firewall your origin to Cloudflare IPs only. Cloudflare publishes its full IP range at cloudflare.com/ips-v4. Allow only those ranges on 80 and 443. Drop everything else

Proxy every subdomain. Audit your DNS records. Every subdomain that should be proxied must have the orange cloud enabled. Grey-cloud records pointing to your origin are a bypass by design

Keep mail on a separate IP. Your mail server should not share an IP or IP block with your web server

Validate Turnstile server-side, always. The token must be verified by your backend on every form submission

Check your certificate history now. Run your domain through crt.sh and SecurityTrails. If your old origin IP is visible, either move to a new IP (and use Tunnel going forward) or rely entirely on the firewall approach

Originally published on jguillaumesio.com

DEV Community: jguillaumesio

My fine-tuned model scored 100%... The benchmark was lying

Is LoRA real fine-tuning?

The task and the hardware

The part nobody writes about: the dataset

Attempt one, and why it lied

Attempt two: real data, and a licensing trap

The shortcut that would have faked the score again

Four defects in the public data

The Mistral gotcha that costs you an afternoon

Training

Results

400 held-out rows

30 hand-written lines the model has never seen

It is also cheaper to run

When you should not do this

The lesson that generalises

Reproduce it

GDPR, PII and data masking: how to stop leaking personal data everywhere

The distinction that actually matters

Where to mask: static vs dynamic

Masking techniques, and their pitfalls

Do not log PII in the first place

A practical approach

GDPR data retention and deletion: a practical guide for developers

Duty one: storage limitation (stop keeping data forever)

Enforcing it: automated purging

Soft delete is not deletion

Duty two: the right to erasure

The part everyone forgets: backups

Do not forget your logs

A short checklist

Docker never restarted my database: the price of restart: no

The line, on all seven services

The part that surprises people: I had healthchecks

Two mechanisms, two different failure modes

Choosing a restart policy

Making "unhealthy" actually do something

What this does not fix

The takeaway

Shipping a SaaS on one VPS with Docker Compose: an honest audit

Why one VPS and Docker Compose was the right call

The actual architecture

How a deploy works today

The map: every single point of failure

Where this series goes

Docker bypassed my firewall: the database port was open to the internet

Why the firewall did nothing

The one binding that was already right

The real fix: internal services need no published ports at all

Audit from outside, never trust iptables -L

The takeaway

The docker build that filled the disk and took down production

What "deploy" meant that day

Why the disk filled, and what cleared it

Diagnosing it: two commands

One dangerous reflex to avoid

The one good habit already in place

The real fix: stop building where you serve

The takeaway

CPU utilization lies: autoscaling a single-threaded service

Why 30 percent CPU meant 100 percent busy

The metric was answering a different question

Three ways to fix it

The takeaway

A practical playbook for running a remote dev team

Ship fewer bugs: put quality in the ticket, not just in QA

The regression net: the same cases, tested before every release

The retro: a Miro board, and votes that force focus

Planning poker: the size is disposable

The daily: coordination, not a status report

The sprint review: shared visibility, not a demo

The actual lesson

The cart timer expired while they were paying. Now what?

The timer answers a question nobody asked at that moment

So, do you let them finish?

The two follow-up questions

Make expiry server-authoritative

The habit behind the bug

The rule I follow now

Audit from outside, never trust `iptables -L`