DEV Community: Srdjan Popovic

Eleven Terabytes and No Raster Database

Srdjan Popovic — Wed, 19 Aug 2026 08:53:56 +0000

DailyMeteo is about 11 TB of GeoTIFFs — 1 km daily temperature and precipitation for all land on Earth, back to 1961. There is no raster database anywhere in the system. No PostGIS raster tables, no tile server in the read path for point queries, no object store. A Django app reads files off a disk.

That sounds like something we haven't gotten around to fixing. It isn't. Here's the reasoning, and the places where it bit us.

Two machines, one job each

The archive is produced on one machine and served from another.

The production machine is a big box — a terabyte of RAM, most of it in use — that does nothing but run the interpolation. Space-time kriging over a continent is not a workload you want sharing a server with anything user-facing, because it will happily consume every core for six hours and then a request times out for reasons no one will connect to weather.

The serving machine runs the API, the frontend, the database, GeoServer, the workers, and the reverse proxy, all as a Docker Swarm stack.

Between them, a rsync that runs every 30 minutes. Two details in it are load-bearing:

It pulls, it doesn't push. The serving machine reaches into the production machine and takes what's missing. Nothing runs on the production box on our behalf except a remote rsync under nice -n 19 ionice -c3, with a bandwidth cap. If the transfer is slow, the interpolation doesn't care. A push would have put the scheduling decision on the machine whose whole job is to not be interrupted.

It transfers incomplete dates. It used to only take dates where all 24 rasters existed. The effect was that when one zone failed, the perfectly good rasters for the other five stayed on the production box and the API reported the date as empty. Missing rasters are missing either way; this way, the ones that exist are at least reachable.

Why the filename is the index

A point query needs to answer: for this coordinate, this variable, and these 400 dates, which files do I open?

With files on a disk and a naming convention, that's string formatting:

{var}_day_{YYYYMMDD}_equi7{_early|_late|}.tif

Work out which continental zone the point falls in, build 400 paths, open the ones that exist. No index to keep in sync, no ingest step, no migration when a year of backfill lands, no second system that can disagree with the disk about what exists. The pipeline finishes writing a file and it is served, with no further action.

What that buys is worth being explicit about, because it's easy to reach for a database out of habit. There is no state anywhere that can drift from reality. A file is either on the disk or it isn't. When we needed to know exactly what we could serve, the answer was os.scandir over 24 directories — about a second for 580,000 entries — and it was the truth, not a cached projection of it.

What it costs:

No query planner. "Every date where the July mean exceeded X anywhere in Europe" is not a question this shape can answer. It answers "give me these pixels from these files" very well and nothing else at all.
Directories with 24,000 files. Fine on ext4, and scandir doesn't care, but ls in a terminal will make you wait and any tool that stats every entry becomes the bottleneck.
The convention is the schema, and it's enforced by nothing. One file written with a different suffix is a silent data bug, not a constraint violation. This is exactly how we ended up serving duplicate values for 3,624 days — a variant appeared that the read path's ranking didn't know about.

Given the access pattern — read a handful of pixels from an arbitrary set of files, never scan or join — I'd make the same call again. The mitigation for the last point isn't a database. It's that every read path goes through one function that knows the convention, and that function is the only place the convention exists.

The file format does the work a database would

The reason reading raw files is fast enough is Cloud Optimized GeoTIFF.

Each raster is Int16, LZW-compressed, internally tiled in 512×512 blocks, with five levels of overviews baked in. A European tile is 8229 × 5588 pixels and 10.2 MB on disk.

The tiling is the point. To read one pixel you read one 512×512 block, not 10 MB. To draw a zoomed-out map you read an overview level that's already there, instead of downsampling the full raster. The layout means a reader can seek to the bytes it needs, and everything downstream — the point query, GeoServer's mosaics, the polygon clip — gets that for free.

Int16 instead of a float is a factor-of-two on 11 TB, which is worth roughly five and a half terabytes of disk. Temperature stored in tenths of a degree loses nothing anyone can measure.

What a request actually does

A point query is Django REST Framework, and the interesting parts are all about what not to do.

Requests are authenticated with an API key, priced in credits, and cached in Redis — a keyed lookup of an immutable historical value is the ideal cache entry, so the TTL is a week. Bulk exports don't happen in the request at all; they're Celery tasks that write a file and email a link, because "clip 20 years of daily rasters to this polygon" and "answer within an HTTP timeout" are incompatible requirements.

The polygon pricing had a nice bug in it, and it's the kind that only shows up at the edges. Cost is computed from pixels times days. Pixels came from the mask of the clipped raster — except the mask counts pixels inside the polygon, and a polygon over the ocean is full of nodata pixels that are inside it and contain nothing. Draw a box over the Atlantic and you'd be charged for a full raster's worth of nothing. The fix is one line — exclude nodata as well as masked — but the shape of the mistake is general: "how much data is here" and "how much of this rectangle is here" are different questions, and the second one is easier to compute, so it's the one you accidentally write.

Running R in someone else's browser

The part of the system I find most interesting isn't on the server at all.

There's an R environment on the site — a console, an editor, plots, tables — and it doesn't run R on our machines. It runs webR, R compiled to WebAssembly, inside the user's browser tab. Their code fetches from our API, computes locally, and renders locally. We never see what they ran.

That was a deliberate call. The alternative is an RStudio-shaped service where users execute arbitrary R on our infrastructure, and that is a sandbox problem, a resource-limits problem, a queueing problem and a security problem, forever. Moving it into the browser makes all four somebody else's — specifically, the browser's, which is very good at exactly this.

The bill comes as about 45 MB of WebAssembly and R packages, downloaded before anything runs. We warm the runtime up in the background as soon as the page loads, so by the time someone has typed a question the worker is usually alive. "Usually" is doing real work in that sentence, and the first visit on a slow connection is not a great experience.

The second cost is subtler. webR's default communication channel uses SharedArrayBuffer, and browsers only expose that to pages that are cross-origin isolated — which means serving two specific headers:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

The catch is that require-corp also blocks every cross-origin subresource that doesn't explicitly opt in. Turn it on site-wide and the map's vector tiles stop loading and half the images on the marketing pages disappear. So it's scoped to the R pages only, which means the R runtime lives on an island with different security semantics from the rest of the app, and every asset it needs has to be reachable from that island.

It's worth the trouble for one reason: SharedArrayBuffer is the only channel that supports interrupting a running computation. Without it, a runaway while (TRUE) can only be stopped by destroying the worker and booting a fresh 45 MB one. With it, there's a stop button that actually stops.

Two failures worth stealing

Both of these cost more time than they should have, and both look like someone else's bug until you find them.

nginx doesn't know what .mjs is. The bundled mime.types maps .js and stops there. Every ES module served straight off disk — including both worker entry points, the map's and R's — went out as application/octet-stream, and the browser refused them: "Strict MIME type checking is enforced for module scripts." Every other asset was fine, so the app looked healthy while two of its major features were dead.

The fix is a line in the Dockerfile. The part worth copying is the second line:

RUN sed -i 's|\(application/javascript[[:space:]]\+\)js;|\1js mjs;|' /etc/nginx/mime.types \
 && grep -q 'js mjs;' /etc/nginx/mime.types

If a future base image formats that entry differently, the sed silently does nothing and we're back to a broken worker discovered by a user. The grep turns that into a failed build. A patch applied by pattern-matching someone else's file should always be followed by an assertion that it took.

A URL nothing would accept. The assistant generates R against a helper with this signature:

get_data(var, agg_level, time_scale, from = NULL, to = NULL, time = NULL, lat, lon)

time takes a comma-separated list of dates, as an alternative to from/to. Asked for six years of daily data, the model enumerated every date into time. That's 2,192 dates and a 24,222-character URL.

webR failed with problem writing module_download template in internet module, which reads like a webR bug and sent me looking in the wrong place. It isn't. Our own API returns 414 above roughly 8,190 characters — Apache's default LimitRequestLine — so the same generated code, copied into RStudio, would have failed too:

  700 dates ( 7,810 chars) → 400   (request accepted)
1,000 dates (11,110 chars) → 414 URI Too Long
2,192 dates (24,222 chars) → 414

The tempting fix is to teach the model not to do that. We fixed the helper instead. Before building any URL, get_data now normalises time: a contiguous run collapses into a single from/to request, and a list with gaps is split into batches that keep every URL under 2,000 characters. The failing case went from 24,222 characters to 165, in one request, returning exactly the same 2,192 rows as from/to would.

The signature didn't change — it's the contract the model was trained against, and it's written into its system prompt. Only the body did.

That distinction is the whole point. When a language model generates code against your helpers, the helpers are the place to be defensive, because they're the part you control and the part that doesn't need retraining. Prompt changes are a request. Helper changes are a guarantee.

The blind spot

The last thing this architecture taught me is that it degrades quietly, everywhere.

A missing raster isn't an error; the date is just thinner. A failed source isn't an error; the day is produced with five zones instead of six. A transfer that skips an incomplete date isn't an error; the API simply has nothing for it. Every one of those is a reasonable local decision, and stacked together they mean the system can be substantially broken while every component reports success.

The number that finally made it visible was the difference between two dates: the most recent day with any raster, and the most recent day with all 24. The first was five days back, which is normal and looked fine. The second was six weeks back.

Nothing was alerting, because nothing was subtracting.

DailyMeteo is at dailymeteo.com. The previous post covers what the data is and why every day is computed three times.

We Publish the Data Before It's Correct. On Purpose.

Srdjan Popovic — Wed, 19 Aug 2026 08:52:13 +0000

Someone asks for yesterday's maximum temperature in Belgrade. We don't have it. We won't have it for another four days, and when we do have it, the number will be wrong — not badly wrong, but wrong enough that we'll replace it a month later, and replace it again a year after that.

That's not a bug report. That's the design.

I want to explain why, because it turns out the three-pass thing is the most consequential decision in the whole system, and every other piece — the API, the storage layout, the way a chart gets drawn — bends around it.

What the data actually is

DailyMeteo is a gridded daily climate archive. Four variables — maximum, minimum and mean temperature, and precipitation — at 1 km resolution, for all land on Earth, every day since 1 January 1961.

Some numbers, because the shape of the problem is mostly numbers:


Resolution	1 km, daily
Period	1961-01-01 → present
Variables	`tmax`, `tmin`, `tmean`, `prcp`
Continental zones	6, in the Equi7 projection
Rasters per day	24 (6 zones × 4 variables)
One European tile	8229 × 5588 px, 10.2 MB
Archive	~11 TB

Twenty-four rasters per day, roughly 24,000 days. That's around 580,000 GeoTIFFs, and it's why "just put it in a database" stopped being an option early.

The projection deserves a note. Equi7 splits the land surface into seven continental zones, each with its own equidistant projection, so that a kilometre is actually a kilometre wherever you are. A single global grid can't do that — it either stretches at the poles or bunches at the equator. The cost is that "the world" is six separate rasters that don't share a coordinate system, and every query has to work out which zone the point falls in before it can read a pixel.

How one day gets made

Weather stations don't cover the world. They cover the parts of the world that historically built weather stations, which is a very uneven map. Turning some thousands of point observations into a continuous 1 km surface is the actual work.

The pipeline pulls station observations from several archives — OGIMET, GSOD, GHCN-D, ECA&D, MeteoManz — because none of them alone is complete, and each has a different idea of what "yesterday" means. On a normal day that's something like 18,000 stations queried for a single variable.

Then, per day and per variable:

Fit a trend. A linear model on three covariates: elevation from a DEM, topographic wetness index, and a geometric temperature trend — a deterministic function of latitude and day of year that carries most of the seasonal signal before any interpolation happens.
Take the residuals. Observed minus trend, at each station.
Krige the residuals in space and time. Not just "what do the neighbouring stations say today" but "what did this neighbourhood say yesterday and the day before", which matters enormously when a station is missing for a day.
Add the two rasters back together. Trend surface plus interpolated residual surface.

Precipitation gets a second pass, because rain isn't like temperature. Temperature always has a value; rain is mostly zero. So it's modelled twice — first whether it rained at all, then how much, given that it did. Interpolating rainfall directly gives you a light drizzle over an entire continent, which is both wrong and hard to notice.

None of this is my invention. The method comes from Kilibarda et al. (2014), where Milan Kilibarda and co-authors established spatio-temporal regression kriging for global daily temperature at 1 km, and the interpolation runs through the meteo R package developed by Milan Kilibarda and Aleksandar Sekulić. What's ours is the part that turns it into something that runs every night and answers HTTP requests. It is not fast. A single day, all zones, all variables, takes hours on a machine with a terabyte of RAM.

Why the same day is computed three times

Here's the tension. Station data arrives late, and it arrives incrementally.

The archive that gives you a station's observation within a day or two is not the archive that eventually gives you the quality-controlled version. Some networks publish a preliminary value and revise it. Some publish nothing for weeks and then backfill a month at once. If you wait until the data is final, "today's weather" is a year old.

So we don't wait. Every day gets computed three times:

early — about four days behind, from whatever stations have reported. Runs nightly.
late — about four months behind, once the monthly archives have settled. Runs monthly.
final — the following year, on the fully quality-controlled record. Runs yearly.

Each pass overwrites nothing. The three versions sit side by side on disk, distinguished by a suffix in the filename:

tmax_day_20200715_equi7_early.tif
tmax_day_20200715_equi7_late.tif
tmax_day_20200715_equi7.tif        # final, no suffix

The API ranks them — final beats late beats early — and serves the best one that exists for the date you asked about. Right now that means anything before 2024 comes back final, and the last few weeks come back early. The transition is invisible in the response, which is either elegant or dishonest depending on your mood; I'll come back to that.

The ratio is heavily in favour of finished data. In the European maximum-temperature series: 23,010 final rasters, 851 late, 585 early. Ninety-five percent of the archive is settled. It's the leading edge that churns.

What it costs

Three variants of every file, ranked at read time, is a rule that lives in exactly one function. Rules like that get broken by accident.

We added the late stage after early and final were already running. Somewhere in the code that assembles a time series for a point query, the ranking knew about two variants and the directory now contained three. The unknown file didn't lose the ranking — it wasn't in the ranking at all, so it came through as a separate, additional row.

The result was a chart with two values for the same date. Not a crash, not an error, not a log line. Two dots where there should be one, on 3,624 days' worth of data, sitting there for however long it took someone to look closely at a chart.

The fix is three lines. The lesson isn't about the three lines. It's that "prefer the best available version" is a domain rule, and if it's implemented as an incidental sort somewhere in a view, it will drift the moment the domain gains a version. It belongs in one named function that every read path goes through, and adding a variant should be a change to that function and nothing else.

What you can actually do with it

The data is the product, but nobody wants an 11 TB tarball. What's on top of it:

Point queries. Click a location, get a series. The API takes a latitude, longitude, variable, and either a range or a list of dates, and returns timestamps and values. Daily, monthly or annual, aggregated or as long-term means over the two standard climate normals — 1961–1990 and 1991–2020.

GET /meteo/v2/pq/?var=tmax&agg_level=agg&time_scale=day
    &from=2020-01-01&to=2020-12-31&lat=44.81&lon=20.46&api_key=…

Areas. Draw a polygon, get the aggregate over it, or export the clipped rasters. Large exports are priced by area and period and delivered by email when they're ready, because a polygon over Asia for a decade is not a request you answer inside an HTTP timeout.

Long-term means and anomalies. The monthly and annual aggregates and the two climate normals are precomputed — roughly 750 GB of them — so "how does this July compare to the 1991–2020 average" is a lookup, not a computation over 30 years of daily rasters.

R in the browser. There's an R environment on the site that runs entirely client-side, compiled to WebAssembly. You write R, it fetches from the API and plots, and your code never touches our servers. There's an assistant next to it that turns a question in English into R code against the same helpers. That one has enough engineering in it to deserve its own post, which is the next one.

Embeddable charts, because half the time what someone actually wants is a temperature curve on their own page.

The part I'd rather be honest about

Two things about this design are genuinely awkward and I don't have clean answers.

The seam is invisible. A response doesn't tell you which pass produced each value. Ask for a series spanning 2023 to now and you get final data and early data in one array, with no marker. For a chart, fine. For a paper, not fine. Exposing the processing level per value is the obvious fix and it's the kind of obvious fix that stays on the list because nothing visibly breaks without it.

"Available" and "complete" aren't the same thing, and we conflated them. A date is only fully covered when all 24 rasters exist. When one source fails — and sources fail; a station archive returned zero rows out of 18,000 for several days recently — the pipeline still produces the zones it can. So a date can have 22 of 24 rasters: present, queryable, and quietly missing two continents.

We were tracking "latest date with any raster". By that measure everything looked five days behind, which is normal. By the measure that matters — latest date with all 24 — we were six weeks behind and nobody had noticed, because no counter anywhere was counting that. Now there's an internal page that shows both numbers side by side, and the gap between them is the number I look at first.

That's the recurring shape of this whole system, honestly. The hard part was never the kriging. It's that a pipeline which degrades gracefully also fails quietly, and you have to go out of your way to build the thing that tells you it did.

The interpolation method and the meteo R package behind it are the work of Milan Kilibarda and Aleksandar Sekulić at the University of Belgrade, Faculty of Civil Engineering.

DailyMeteo is at dailymeteo.com. Next post: how 11 TB of rasters get served without a raster database, and what happened when we tried to run R inside a browser tab.

Our Documentation Was Lying. The Model Believed It.

Srdjan Popovic — Wed, 19 Aug 2026 08:26:58 +0000

There is a variable called slp — sea-level pressure. Our API documentation lists it as available. Our error messages list it among the valid options. Our fine-tuned model, asked about air pressure over Belgrade, will happily write you fifteen lines of R to fetch it.

The request comes back 400.

It has been coming back 400 the whole time.

What this is

We run dailymeteo.com — a daily meteorological archive for Europe, gridded at 1 km, running from 1961 to roughly five days ago. There's a chat endpoint where you ask a question in plain language and get back R code that queries the archive and answers it. Behind that sits a fine-tuned GPT model, trained on a few hundred question-and-code pairs.

Last week I set out to retrain it on a better dataset. I expected to spend the day on hyperparameters. I spent it finding out that the model had been taught things that were not true.

The training data was built from the documentation

This is the part worth stopping on, because I suspect it is extremely common.

When you build a fine-tuning set for "write code against our API", the natural move is to sit down with the API documentation and write examples from it. That is what had happened. Each example carried a system prompt describing what the API does, a question, and the R code that answers it.

The trouble is that documentation is a claim about a system, not the system. And nobody had checked the claim in a while.

So before touching anything, I did the boring thing: I called the service and wrote down what actually came back.

Three of its claims were wrong.

Claim 1: `slp` is an available variable

The data exists. There are 21,916 daily rasters sitting on disk, covering 1961 to 2020, Europe only.

The API will not serve any of them. The allow-list in the view is:

VARS = ['tmax', 'tmin', 'tmean', 'prcp']

and the check var not in VARS guards every entry point. On top of that, the continent mapping points at a newer data folder that never received the pressure rasters at all. Two separate reasons for the same 400.

The best detail: the error message you get back still lists slp among the available variables. The message is older than the list.

The model had been taught to ask for it. Two examples in the training set did exactly that. So on any question about pressure, the model produced confident, well-formed, non-functional code.

Claim 2: "data from 1960 to 2024"

Both ends wrong.

1960 returns "There's no data for date range." The archive starts in 1961. And the far end isn't a year at all — the archive is kept near-real-time. Daily data run to about five days ago, monthly to the previous month, annual to the last complete year. Writing a fixed end year into a system prompt guarantees it will be wrong within twelve months, silently.

While measuring this I found something genuinely useful: asking for a range outside the archive is not an error. The API quietly clips.

ann  1901 → 1970   returns  1961 → 1970
ann  1961 → 2035   returns  1961 → 2025
mon  1961-01 → 2030-12   returns through 2026-07

That changes what "correct" code looks like. Generated code doesn't need to know where the archive ends — it can ask wide and read the actual extent back out of the response. That's a pattern that never goes stale. Several of our training examples had been hardcoding an end year instead, which ages badly and quietly.

Claim 3: the timestamp format

The docs said the returned timestamp is formatted like the input date. For aggregated data, true. For long-term means, not remotely:

ann   "1961-1990"          "1991-2020"
mon   "05.1961-1990"       "05.1991-2020"
day   "25.07.1961-1990"    "25.07.1991-2020"

It carries the climate period, not a date. Any code doing substr(timestamp, 1, 4) to pull a year out gets nonsense. Some of ours did.

And the thing that wasn't documented at all

Long-term means — the most semantically awkward corner of the API — had no description whatsoever. The rule, once measured, is simple: the year inside the date selects which climate period you get. Pass 1961, get 1961–1990. Pass 1991, get 1991–2020. Omit the year, get both.

Nobody had written that down. So across 23 calls in the training set, long-term means were invoked seven different ways, two of them mutually contradictory. The model wasn't learning a convention. It was learning that there isn't one.

The subtler poison: examples whose answer the question doesn't determine

This one I didn't expect, and it's the one I'd most like other people to check for.

Consider this pair from the training set:

Question: plotting temperatures during autumn (September 1 to November 30) in Prague

Code: from = "1999-09-01", to = "1999-11-30"

Why 1999? No reason. The question doesn't say. Whoever wrote the example picked a year.

There were 28 examples like this. And here is why they matter beyond tidiness: the model cannot possibly predict the answer from the question. No amount of training reduces the error on that example, because the target contains information the input doesn't.

You can see it in the metrics. In the previous training run, the loss spikes that survived all the way to the final epoch — steps 223, 238, 249, 253, 260, still spiking at 0.46–0.66 while everything around them sat at 0.13 — were these. They're not a hyperparameter problem. They're irreducible.

Worse, what the model does learn from them is the behaviour: invent a year, say nothing. In production that's a model quietly answering a different question than the one asked.

The fix cost nothing and didn't change a single choice the code makes. We just made it say so:

cat("No period was specified in the question - using 1961 to 2020.\n\n")

Same year. Now the answer is determined by the question, plus a disclosure the model can actually learn.

What we changed

Nine steps, but the shape is simple:

Measure the API. Every boundary verified by calling the service. The result is one Python file that every other script imports — a single source of truth that is not the documentation.
Audit against it. A script that walks each example and flags what doesn't work, what wastes model capacity, and what's cosmetic. First run: 13 examples the API rejects, 28 inventing periods, and a training set split 135-to-62 between two different code formatting styles for identical tasks.
Fix what's unambiguously wrong. The 13 rejects, and 23 silent period choices turned into disclosed ones.
Normalize style. Half the set broke calls across multiple lines, half kept them on one — same task, double the tokens. That split was costing model capacity on a question with no informational value.
Fill the gaps. There was not a single example covering questions the API can't answer — wind, humidity, forecasts. The model had been improvising. Now it answers "I don't have that, here's what I do have" — still as runnable R, because the contract is that every answer executes.
Derive the explanations from the code, not by hand, so they stay in sync when the code changes.
Split off a validation set. 29 examples the model never sees.
Preflight. Refuse to upload if anything above regressed.
Train, then choose.

The previous run had no validation set

This is the single change that mattered most, and it's the cheapest one.

The earlier job ran three epochs with no validation file. Its training loss fell nicely, from 0.38 to 0.16, and everyone was happy.

Training loss falls whether the model is learning or memorising. Without a held-out set, those two are indistinguishable. You are looking at a number that goes down in both the good case and the bad case, and concluding things about it.

With validation, this run:

Epoch	Training loss	Validation loss
1	0.368	0.285
2	0.133	0.131
3	0.071	0.140
4	0.038	0.152

Training loss keeps falling all the way to 0.038 — near-perfect reproduction of what it was shown. Validation bottoms out in epoch 2 and turns back up. Textbook. Invisible without the held-out set.

One incidental finding: the platform's auto hyperparameter selection picked a learning-rate multiplier of 0.51 on the previous run and 2.0 on this one. The only meaningful difference was that our examples got longer. If you rely on auto, know that it can quadruple your learning rate because you added a paragraph to your system prompt.

Then loss and behaviour disagreed

Validation loss says take epoch 2. So I built a second evaluation that asks a different question: does the generated code actually work? Each checkpoint answers the 29 held-out questions, and we measure whether R parses it, whether the API arguments are valid, whether it invents packages that aren't installed, and whether it discloses the period it chose.

Checkpoint	Validation loss	Failures out of 29
Epoch 2	0.131	4
Epoch 3	0.140	1
Epoch 4	0.152	0

Exactly inverted.

The explanation is that validation loss measures token-level similarity to a reference answer. It does not measure whether code runs. As training continues, the model drifts away from the reference answer's exact wording — which raises the loss — while the code stays correct and the behaviours we deliberately added keep consolidating.

Epoch 2 has the best number and cannot do the thing we trained it to do: on three of four questions with no stated period, it silently picks a year.

We shipped epoch 4. Had we trusted the loss, we'd have shipped the one that doesn't work.

And then I got the measurement wrong too

Worth including because it's the mistake I'd most likely repeat.

The behavioural evaluation gave epoch 4 a perfect score on disclosure: 4 out of 4. I reported 100%.

It asked each question once.

Re-running the same question against the deployed model several times:

temperature = 1.0    discloses 3/4
temperature = 0      discloses 1/3

The real figure is somewhere around 50–75%, not 100%. The behaviour is genuinely learned and genuinely better than the 25% the previous model managed — but it is not reliable, and a single sample per question cannot tell you that. One sample measures what a model did once. It says nothing about what it does.

(temperature = 0 also isn't deterministic without a seed. Three calls, three different answers.)

What I'd tell anyone fine-tuning against their own API

Your model inherits your documentation's lies. If the training data was written from the docs, every stale claim in them is now a learned behaviour. Call the service and write down what comes back. It took an afternoon and found three errors in a prompt that had been in production for months.

Check whether the question determines the answer. Any example where the target contains information absent from the input is teaching the model to make things up. It also shows up as loss spikes that never come down, which is a cheap way to find them.

Ship a validation split, even a tiny one. Twenty-nine examples were enough to reveal an overfitting turn that was completely invisible for three epochs previously.

Measure whether the output works, not whether it matches. Loss is a proxy. Parse rate, valid arguments, no hallucinated dependencies — those are the thing itself. When the two disagree, the proxy is not the one to trust.

Sample more than once. A percentage from one draw per question is not a measurement, it's an anecdote with a decimal point.

The model was maybe a fifth of the work. The rest was going back and asking the system what it actually does — which, in hindsight, is what anyone should have done before writing the documentation the model learned from.