I Took First Place Without Training a Model

Builder Journal · Mars Environmental Dynamics Analyzer (MEDA) Virtual Sensor Recovery

My first submission to this competition scored 61.04 and put me at the top of the leaderboard. It contained no machine learning. No gradient boosting, no neural network, no ensemble. It was a curve fit to a coordinate and a Fourier series, and the most important line of work behind it was a paper I read before I opened an editor.

That should tell you something is unusual about the problem.

The competition in one breath

Perseverance carries an environmental station called MEDA. It reads air pressure, temperature, wind, humidity, radiation and dust from the floor of Jezero Crater. Some of the pressure readings are missing, and the competition is to reconstruct them from everything else the rover recorded. Scored on mean squared error.

Then you look at the data split, and the problem changes shape entirely.

Training data covers sols 1 through 100. Test data covers sols 201 through 300. Sols 101 through 200 do not exist in either file.

train:  sol 1 .......... 100                              (pressure rising)
gap:              101 ......... 200                       (nothing)
test:                              201 .......... 300     (pressure falling)

There is no overlap. None. You are not filling gaps inside a season you have seen. You are predicting a season you have never observed, from a hundred sols away, across a hole in the record.

That makes this time series extrapolation in its least forgiving form. Not imputation, where the answer is bracketed by data on both sides. Extrapolation, where every prediction is outside the range the model was fit on and nothing in your training set can tell you when you have gone wrong.

Almost every instinct I have as a practitioner is wrong here, and it took me a while to work out why.

The 100 Pascal mistake I nearly made

Before writing anything I went reading, because I knew nothing about Martian weather. That turned out to be the highest-value hour of the entire project, and not because of anything clever.

Mars has a real seasonal pressure cycle, driven by carbon dioxide freezing onto the poles in winter and coming back in summer. At Jezero, daily mean surface pressure climbs to a peak of about 761 Pa around sol 105, then falls to a minimum near 625 Pa around sol 306.

Read that against the split again.

The training window is the entire rise. The test window is the entire fall. A model fit on sols 1 through 100 sees pressure going up and, asked about sol 250, will confidently continue it upward. The truth is that pressure is coming down, hard, by more than a hundred pascals.

Not a subtle bias. A sign error on the largest signal in the data, and one that no amount of tuning would have surfaced, because there is no labelled data anywhere on the falling limb to argue with you.

I found that in Sánchez-Lavega et al. 2022 before I had written a line of modeling code. If I had started with the modeling instead, I would have built something sophisticated on top of a foundation that was pointed in the wrong direction, and the leaderboard would have been the first thing to tell me.

Why gradient boosting could not save me

The obvious move on tabular data is to throw LightGBM at it. I could not, and the reason is worth being precise about.

A decision tree partitions the input space using thresholds it saw in training. Ask a tree about sol = 250 when it has only ever seen sol ≤ 100 and it does not extrapolate. It falls into its outermost leaf and returns that leaf's value. Flat. Every test row gets essentially the same seasonal contribution, which is the one thing you cannot afford when the seasonal term is moving 100 Pa across your prediction window.

So the trend cannot come from a tree. It has to come from something that understands what it is looking at.

There was one more trap in the column list. The dataset ships a field called SOLAR_LONGITUDE_ANGLE, which sounds exactly like the seasonal coordinate I needed. It is not. It is solar azimuth, and it swings from −180 to 180 in both train and test. Reading the schema instead of the column name saved me from building the whole model on a decoy.

A coordinate that knows where it is

The fix is to stop asking about sol and start asking about Ls.

sol is a counter. Days since landing. It has no physical meaning, so a curve fit in sol is a line running off a cliff the moment you leave the training range.

Ls, areocentric solar longitude, is where Mars actually is in its orbit, from 0 to 360 degrees. The pressure-versus-Ls relationship is roughly the same from one Mars year to the next, because it is driven by orbital geometry rather than by how long the rover has been sitting there. A curve fit in Ls extrapolates by physics. The same curve fit in sol extrapolates by wishful thinking.

Converting one to the other is the Mars24 algorithm from Allison and McEwen: sol to UTC, UTC to Julian Date, then a trigonometric series.

jd = LANDING_JD + sol * 88775.244 / 86400.0   # a sol is 88775.244 seconds
# ...Mars24 series: mean anomaly, equation of center...
Ls = (alpha_fms + nu_minus_M) % 360.0

Then I did the thing I would tell anyone to do when they implement physics from a paper: I pinned it against known values. My implementation gives Ls = 5.64° at landing against a published 5.2°, and 13.12° at sol 15 against a paper's 13°. A 0.12° match on an independent checkpoint is what let me trust every downstream number.

If you write physics and never check it against a value someone else published, you have written something that runs. That is not the same as something that is right.

Splitting the signal into three jobs

With a coordinate that extrapolates, the model became a decomposition rather than a predictor.

PRESSURE = B(Ls) + D(time_of_day) + R

B is the seasonal baseline, moving over months, driven by orbital position. D is the daily cycle, the thermal tides as the sun heats the atmosphere and it sloshes. R is everything left over: weather, dust, whatever the planet is doing that week.

The point of splitting it is that the three parts extrapolate completely differently. B and D are physics. Give them the right coordinate and they will tell you about a season they have never seen. R is messy and local and stubborn, and it is the only part that deserves a machine learning model at all.

The decomposition does not make the problem easier so much as it makes the problem honest. It puts the extrapolating work on the parts that can extrapolate, and shrinks the ML down to something stationary and small.

Using a published paper as training data

That still leaves the awkward question. B has to describe a curve turning over at sol 105 and falling through the test window, and I have no observations past sol 100 to fit it with.

So I borrowed them. The published climatology gives the peak (761 Pa) and the minimum (625 Pa). I injected those as pseudo-observations with enormous weight, so the fit could not talk itself out of them.

# anchors weighted 1e6 against 1.0 for real observations:
# the fit is not permitted to ignore them
w = np.concatenate([np.ones(len(real)), np.full(len(anchors), 1e6)])
coef = np.polyfit(ls_all, pressure_all, degree, w=w)

This is the difference between extrapolating a guess and extrapolating something a research group already measured. My data ends at sol 100. The literature does not. Treating published values as high-confidence observations is how the model knows about a season it has no rows for.

Cross-validation that does not flatter you

Random k-fold would have been a disaster here. Shuffle the rows and a fold trains on sol 90 while validating on sol 40, which means the model gets to peek at the future and hands you back a fantasy.

So the folds walk forward instead. Always train on earlier sols, always validate on later ones.

fold 1: train [1..70]  -> validate [71..80]
fold 2: train [1..80]  -> validate [81..90]
fold 3: train [1..90]  -> validate [91..100]

That is more honest. It is still not honest enough, and understanding why is the whole point of this entry.

It is worth saying that this is the good case. At least here the training labels exist and forward folds are possible. On a hyperspectral tracking challenge I hit the version where the leaderboard is the only thing that can score you at all, because the test set ships unlabeled and you have to rebuild the official scorer yourself before you can measure anything locally.

Two bugs that were really the same bug

Before the first submission scored, two things went wrong, and both of them were about guard rails built from the wrong range.

I clipped predictions to a physically sensible band of 710 to 775 Pa. That band came from the training data, where pressure never once goes below 710. The test season legitimately falls toward 625. So my floor caught nearly every test prediction and flattened the entire declining season into a near-constant line. The guard rail became a wall. Lowering the floor to 600 fixed it.

Then the seasonal polynomial. I had it at degree 4, and with only three anchors that fourth degree of freedom had nothing to constrain it, so the curve bent back upward inside the test range. Its minimum landed at Ls 143.5 while the test runs out to 146.6, meaning the model predicted the season starting to recover right where it is still falling. Dropping to degree 3 made it monotone across the window.

Here is the part I keep coming back to. The degree-4 model had the better cross-validation score. It fit the rising limb slightly more snugly, and CV only ever looks at the rising limb. If I had selected on CV, I would have shipped the model that was physically wrong on the only sols that count.

Both bugs are one bug wearing different clothes: a decision made from the range the training data happened to occupy, applied to a range where the answer lives somewhere else.

The number that mattered more than the score

The submission scored 61.04 and took the top of the leaderboard, which felt good for roughly the length of time it took me to look at my own validation number.

If you have no intuition for what an MSE of 61 means on this data, the leaderboard supplies one. The sample_submission.csv that ships with the competition scores 31,830. Second place, on six entries over the preceding month, sits at 86.14. The physics decomposition landed at 61.04 on the first attempt, five minutes after I uploaded it.

I want to be careful about how much credit that deserves. Beating a sample submission by a factor of five hundred is not a sign of brilliance, it is a sign that the sample submission is a constant. And the gap to second place says more about how early the competition was than about how good my model was.

Forward cross-validation said 20.39. The leaderboard said 61.04.

Three times worse. And here is what makes that gap interesting rather than alarming: it is not noise, and it is not a bug. It is exactly the quantity my cross-validation is structurally incapable of measuring.

CV validates on sols up to 100, all of them on the rising limb, all of them in a regime where B is fit to real observations. The leaderboard validates on sols 201 to 300, entirely on the falling limb, entirely in the region where B is extrapolating off borrowed anchors. The two numbers are not measuring the same thing and were never going to agree.

So the gap has a name. Roughly 40 MSE of it is seasonal extrapolation error, and it is invisible from inside my own validation. That told me precisely where the next work was: not the ML I had not built yet, not the diurnal harmonics, but the seasonal baseline.

A validation gap that large is not a failure of the validation. It is the most specific piece of information you have.

I have written this same sentence in three different competitions now. On a wellbore-geology problem it took me a month to notice that my validation score was lying to me because I was pooling an error metric the wrong way. On a hyperspectral tracking challenge, my own test set lied to me through selection bias I had built by hand. The specific mechanism is different every time. The shape is identical: a number I trusted was answering a question I had not asked.

The next layer proved me right and wrong at once

I went into the following session aiming squarely at the seasonal term, because that is where the diagnostic pointed. Better climatology curve, PCHIP interpolation across 23 digitized knots instead of a polynomial guessing at the shape, the whole treatment.

Leaderboard went 61.04 to 34.28. Down 44 percent.

Then I checked which change had done it, and it was not the seasonal one.

The v1 and v2 test predictions differ by about 2.6 Pa on the seasonal level. Almost nothing. The 44 percent came from the diurnal model, where fitting five harmonics on a clean within-sol anomaly caught semidiurnal and terdiurnal tides that my original three-harmonic fit had been smearing over.

I had the right diagnosis and hit a different target. Without actually comparing the two prediction sets I would have credited the seasonal refinement, written that down as confirmed, and planned the next layer on a belief that was wrong.

That is a small thing that happens constantly and almost never gets caught. You predict A, you change A and B together, the score improves, and you file it under "A worked." The only defense is to make the comparison boring and explicit: what did the predictions actually do, term by term.

What I would take from this

The three things I would carry into any problem shaped like this one:

Reading the domain literature is modeling work, not preparation for it. An hour with a 2022 paper caught a sign error on the dominant signal. No hyperparameter search finds that, because the data that would reveal it does not exist in your files.

Choose the coordinate before you choose the model. Everything downstream got easier the moment the seasonal axis became orbital position instead of a counter. The model class barely mattered by comparison.

Know what slice of reality your validation covers. Mine covered the rising limb of a season while I was being scored on the falling limb. It was not lying to me. It was answering a different question than the one I was asking, and it took a three-times gap for me to notice I had been reading its answer as though it applied.

That last one is the recurring character in this whole log. Every competition I enter eventually produces a version of it, and the failure is never in the code. On ARC-AGI it turned out the bug was in my beliefs rather than anywhere I could put a breakpoint. Here it was a polynomial degree chosen by a metric that could not see the part of the season that mattered.

The rest of this series is mostly the story of that last one refusing to stay learned.

Frequently asked questions

Why can't gradient boosting extrapolate a time series?
A decision tree splits the input space on thresholds it saw during training. Given a value beyond that range it falls into its outermost leaf and returns that leaf's constant, so predictions flatten instead of continuing the trend. This is fine when test data overlaps training data, and fatal when it does not. The trend has to come from a model with a physical or functional form that carries beyond the observed range.

What is Ls, and why use it instead of a day counter?
Ls is areocentric solar longitude, the position of Mars in its orbit from 0 to 360 degrees. A day counter like sol has no physical meaning, so a curve fit against it has nothing to say about values it has not seen. Seasonal pressure repeats against Ls from one Mars year to the next, so a curve fit against Ls extrapolates on physics rather than on trend continuation.

Why did my cross-validation score not match the leaderboard?
Usually because the two are scoring different regimes. Here, forward cross-validation only ever validated on sols 1 to 100, the rising limb of the seasonal cycle, while the leaderboard scored sols 201 to 300, entirely the falling limb. Cross-validation was measuring in-distribution fit quality, and the leaderboard was measuring extrapolation error. A large, stable gap between the two is diagnostic information about which component is failing, not noise.

How do you model a season you have no training data for?
Anchor it to published measurements. Peer-reviewed climatology supplied the seasonal peak and minimum for Jezero Crater, which were injected into the fit as heavily weighted pseudo-observations so the fitted curve was forced to follow the known shape into the unobserved window.

What is the difference between interpolation and extrapolation here?
Interpolation predicts inside the range of the training data, where the answer is bracketed by observations on both sides. Extrapolation predicts outside it. In this competition the training and test windows do not overlap at all, so every prediction is extrapolation and no amount of held-out validation on the training window can measure it.

Standing: this submission held rank 1 of 9 on the public leaderboard. Later entries cover what happened when I started trusting that leaderboard a little too much.

More in this series

• From Moon Craters to Martian Dust · why I entered this competition in the first place.
• The Validation Score That Lied to Me for a Month · the same lesson on a wellbore-geology problem.
• The Builder Journal · the live log across every competition I'm in.
• Every entry from this competition · the full MEDA Virtual Sensor Recovery thread.

This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.