Builder Journal · ROGII Wellbore Geology Prediction
This is a Builder Journal entry written from inside a machine-learning competition I am still competing in, prize money on the table and the clock running toward a deadline. It is not a tidy after-the-fact writeup where I already know how it ends. It runs long and it shows code, because it is about the least glamorous decision in the whole sport, the one that quietly decides everything else: how you grade your own work when nobody will tell you your real score. You are reading where I was, not where I am, with the parts that are still my edge kept dark.
Imagine driving down a highway at night with the windshield painted over. You cannot see the road at all. The only thing you have is a single needle on the dashboard, and you have decided to trust it with your life, because it is the one instrument you own. Now imagine the needle is wrong. Not shattered, not pinned at zero, not flickering in a way that would warn you. Just quietly, confidently off by a little. You would steer straight into the ditch and feel good about every correction you made on the way down.
That was my first month in this competition. The needle I built to grade myself was lying, and it lied with a perfectly straight face.
The competition is ROGII Wellbore Geology Prediction, and the real job is a lot like that blacked-out windshield. An oil well does not go straight down. Past a certain point it turns and runs sideways, horizontally, for a mile or more through rock nobody will ever lay eyes on. Somewhere in that rock is a specific layer the drillers are trying to stay inside, and the whole operation lives or dies on knowing how deep that layer sits at every step of the way. You do not get to look. What you get is a gamma-ray log, a single readout of how naturally radioactive the rock is as the bit grinds past it, because different layers glow at different levels. You also get a typewell, a nearby well someone already drilled straight down and measured properly, as a reference signature. The task is to line up the faint radioactive whisper coming off your horizontal bit against that reference, and from that alone, call the depth of the layer foot by foot.
They grade you on how far off you are, in feet, averaged across every foot of every well, and there is fifty thousand dollars for the team that gets closest. The catch, the one that sets up everything that went wrong for me, is that the wells they grade you on are hidden. You never see the answers. You submit, a while later a single number comes back, and you are rationed to a handful of submissions a day. So you cannot just keep guessing at the real scoreboard until something sticks. You have to build your own scoreboard at home, one you can check as often as you want, and then you have to pray it tells you the truth.
Mine did not.
Two numbers that were never the same number
Here is how the lie showed up. I would make a change I believed in, score it on my home scoreboard, and watch my number improve. A tenth of a foot here, a quarter there. Real, measurable progress, according to me. Then I would spend one of my precious submissions to check it against the actual leaderboard, and the leaderboard would shrug. Sometimes it did not move. Sometimes it got worse. The two scoreboards did not just disagree about the size of my progress. They disagreed about whether I was making any.
For a while I did what everyone does. I blamed the leaderboard for being noisy, blamed the hidden wells for being weird, blamed variance. That is the comfortable story, the one where you are right and the world is just being unfair about it. It took me longer than I would like to admit to accept the uncomfortable version instead: my scoreboard and their scoreboard were not measuring the same thing. Not approximately the same thing with some noise on top. Two entirely different statistics, both wearing the same label, "error in feet."
The leaderboard pools everything. Every single foot of every single well goes into one giant bucket, gets its error squared, and the whole bucket is averaged into one number. A well that runs long throws more feet into the bucket, so it counts for more. Nothing hides in there. If one well falls apart badly enough, it drags the whole score down no matter how many other wells behaved themselves.
I was doing something that felt more reasonable and was quietly poisonous. I scored each well on its own, then took the median across all my wells. It sounds fair. Every well gets an equal vote, big or small. But a median has a specific and dangerous habit: it throws away the extremes. It reports the well sitting in the middle of the pack and forgets the disasters out at the edges entirely. So a run where most of my wells were fine and a couple were catastrophically wrong looked, on my scoreboard, like a good day. On the leaderboard, those couple of catastrophes were most of the score.
Here are the two graders side by side. This is the entire bug, in two functions.
import numpy as np
def pooled_rmse(pred, truth):
"""The leaderboard's metric. Every row's squared error goes into one
bucket, pooled across all wells, then rooted. A long well contributes
more rows, so it weighs more. Nothing gets to hide."""
err = np.asarray(pred) - np.asarray(truth)
return np.sqrt(np.mean(err ** 2))
def per_well_median_rmse(pred, truth, well_id):
"""What I was accidentally grading myself on. Score each well on its
own, then take the median across wells. Every well counts the same,
and the median quietly discards the very best and the very worst."""
pred, truth, well_id = map(np.asarray, (pred, truth, well_id))
scores = []
for w in np.unique(well_id):
m = well_id == w
scores.append(np.sqrt(np.mean((pred[m] - truth[m]) ** 2)))
return np.median(scores)
They can be miles apart. Take nine wells where eight of them track the layer within a couple of feet, and one long well loses the layer completely on its back half and drifts thirty-five feet away. Watch what each grader says about the exact same predictions.
rng = np.random.default_rng(0)
preds, truths, wells = [], [], []
# Eight wells behave: a couple of feet of noise around the truth.
for w in range(8):
truths.append(np.zeros(50))
preds.append(rng.normal(0, 2, 50))
wells.append(np.full(50, w))
# One long well loses the rock on its back half and drifts 35 ft off.
truths.append(np.zeros(400))
preds.append(np.concatenate([rng.normal(0, 2, 200), rng.normal(35, 5, 200)]))
wells.append(np.full(400, 8))
pred = np.concatenate(preds)
truth = np.concatenate(truths)
well_id = np.concatenate(wells)
print(round(per_well_median_rmse(pred, truth, well_id), 1)) # 2.0 -> "you're fine"
print(round(pooled_rmse(pred, truth), 1)) # 17.8 -> "you're not"
My scoreboard said two feet. Theirs said almost eighteen. Same predictions, same truth, two completely different verdicts, and the gap between them was one well I was allowed to ignore under my metric and not allowed to ignore under theirs. For a month I had been happily lowering the number that ignored my worst wells, while the number that counted them sat there and did not budge. I was not optimizing my model. I was optimizing my own blind spot.
The one-submission experiment that broke it open
Fixing the statistic was only half of it. The other half was worse, and I only found it because I got suspicious enough to run a stupid little experiment.
I had been grading myself against the handful of wells I could actually see the answers for. A small set of example wells ships with the data so you have something to develop against. Reasonable. Except I had drifted into treating that small set as if it stood in for the real, hidden test. To find out whether it did, I built the dumbest possible model: a lookup table. It memorized the answers for the wells I could see and did nothing intelligent whatsoever. On any well it recognized, it was flawless. On anything else, it just fell back to holding the last known depth flat.
# Memorize the answers for the wells I can see. Guess nothing.
lookup = {well: answers[well] for well in visible_wells}
def predict(well, rows):
if well in lookup:
return lookup[well] # perfect, for wells I've already seen
return hold_last(rows) # otherwise, carry the last known depth flat
On my home scoreboard, this thing scored a perfect zero. Zero feet of error. Of course it did. It was holding the answer key to every well it was being graded on. Then I spent a submission and sent it to the real leaderboard, fully expecting either a suspiciously great score or an obvious flare of leakage.
It scored 15.883. To put that number in context, holding the last known depth flat and doing nothing else at all also scores 15.883.
Holding the last depth flat sounds like giving up, and in a way it is. But it is a shockingly hard baseline to beat, because its error is boring and even. It never makes a confident wrong turn. Every clever model that opens its mouth risks saying something dumber than silence, and plenty of them do. So when my perfect-memory model tied the exact sound of silence, it was not a small failure. It was the loudest possible signal that the model and the test had never once been in the same room. The leaderboard was scoring wells my lookup had never seen in its life. The wells I could see were a tiny development sample. The real test lived somewhere else entirely, a couple hundred hidden wells, and no amount of studying my few visible wells was ever going to tell me how I would do on them.
That one experiment cost me a single submission and it reordered the whole competition in my head. My home scoreboard had been wrong in two ways at once. It was computing the wrong statistic, and it was computing that wrong statistic on the wrong wells. I had been grading a practice exam with the wrong rubric and the wrong answer key, and then acting shocked that the grades never predicted the real test.
Building a scoreboard that finally told the truth
The fix, once I could actually see both halves of the problem, was not clever, and that is usually a good sign. I stopped grading myself on the few visible wells and started grading on the seven hundred and change fully labeled wells that come in the training set, the ones with real answers all the way down. I split them so that no single well ever showed up in both what I trained on and what I scored on, because a well leaking across that line is just the memorization trick again wearing a nicer suit. And I scored every split with the pooled number the leaderboard actually uses, paying deliberate attention to the deep tail, the long wells that fall apart, because those were the ones quietly deciding my real fate.
Then I did the thing that finally let me trust the instrument. I gathered up a few cases where I knew both my home score and my leaderboard score, and I checked whether one predicted the other. It did. The relationship was almost boringly clean: my leaderboard score came out very close to one and a quarter times my home score, plus about three quarters of a foot, and every point I had landed on that line within a sixth of a foot. For the first time, I could make a change, score it at home, and know before I ever spent a submission roughly what the leaderboard was going to say back. The blacked-out windshield finally had a working instrument behind it.
I want to be honest about the edge of that victory, because this is where a lot of tellings stop and pretend the lesson came out tidy. A trustworthy home scoreboard is only trustworthy inside the neighborhood you calibrated it in. Later in the competition I looked at a completely different family of approaches, a different kind of model built on different features, and it snapped my clean little line in half. Those models scored worse than mine at home and better than mine on the real leaderboard. My instrument, so reliable within my own family of ideas, had no honest way to compare across families. The calibration I had earned was real, and it was also local. When I made a bet that jumped to a fundamentally different kind of model, I still had to pay a submission and ask the real scoreboard directly, because my home version had no earned opinion about a neighborhood it had never visited.
The signal so far
None of this is the part of a competition anyone frames and hangs on a wall. There is no medal for the week you spent proving your own ruler was bent. But it is the foundation the entire rest of the work stands on, and I would rather show it to you with the concrete still wet than hand you a clean final number in a couple of months and pretend I always knew which way was up.
The lesson underneath is not really about wells or gamma rays. It is that the most expensive mistakes in this kind of work are almost never wrong models. They are right models measured with the wrong ruler. A model you can measure honestly will improve on its own, because you can finally tell your good ideas from your bad ones. A model you cannot measure honestly will wander for a month while you congratulate yourself the whole way, because you have handed the wheel to an instrument that lies. Before I tuned a single thing that mattered, the highest-leverage hour I spent was the one where I stopped trusting my own scoreboard and made it earn the trust back. The whole game, it turned out, was making sure the needle told the truth before I ever started to steer.
Where I stand right now: past that first month, sitting above the do-nothing baseline that quietly beats a surprising number of clever ideas, with a home scoreboard I can finally believe and a long climb still ahead of me toward the top of the board. Everything before this was learning to see. Everything after it is the actual drive.
More in this series
- The Builder Journal · the live log across every competition I'm in.
- Every entry from this competition · the full ROGII Wellbore Geology Prediction thread.
This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.
Top comments (0)