Builder Journal · ROGII Wellbore Geology Prediction
This is a Builder Journal entry from inside a machine-learning competition I am still competing in, with real money on the line and a deadline coming. It is not a clean recap written once I already knew how things turned out. It runs long and it shows code, because it is about the habit that has saved me more time than any model I have built: knowing how to kill an idea before it can waste a week of my life. You are reading where I was, not where I am, with the parts that are still my edge kept dark.
Richard Feynman had a line I think about constantly when I compete. The first principle, he said, is that you must not fool yourself, and you are the easiest person in the world to fool. Competitions are a machine built to fool you. You will have an idea, a good one, one you are proud of, and every instinct you own will want it to work. You will build it, and somewhere in the building you will start rooting for it, and by the time you finally test it you will have spent so much of yourself that you cannot see it clearly anymore. The graveyard of a competition is full of ideas that were kept alive weeks too long by the person who thought of them.
So this stretch, I stopped trying to be smart about which ideas were good. I got strict about killing the bad ones cheaply instead. Over the last few weeks I have banked sixteen dead ideas. Roughly six of them died for about an hour of compute each, and not one of them cost me a submission. Here is the discipline that did it.
A quick reminder of where we are, if this is the first entry of mine you have read. The competition is ROGII Wellbore Geology Prediction. An oil well drills down and then turns and runs sideways for a mile through rock nobody can see, and the job is to predict how deep a particular rock layer sits at every foot along the way, using only a gamma-ray log, the faint radioactive signature of the rock going past the bit. You are scored in feet of error on hidden wells, submissions are rationed, and there is fifty thousand dollars at the end of it. Every idea you want to try is a fresh model or feature that takes real hours to build and real hours to run, and the only way to know if it truly helped is a submission you are not allowed to spend freely. That scarcity is the whole reason this discipline exists.
The trap is testing the wrong way
The natural way to work is a loop. Have an idea. Build the full, production version of it. Run it. Submit it. Read the number. Keep it or throw it away. It feels rigorous, and it is a disaster, because the expensive step, the full build, comes before the answer. You pay the entire cost of an idea just to find out it was worthless. Do that four times and you have burned a week and most of your submissions learning four things you could have learned in an afternoon.
The fix is to put a cheap test in front of the expensive one, and to decide in advance what result would kill the idea. That second half is the part everyone skips, and it is the part that matters most. If you run the cheap probe first and only then decide what counts as a passing grade, you will move the goalposts. You will look at a mediocre number and find a reason it is actually encouraging, because you want the idea to live. So you write the rule down before you look. You pre-register it, the same way a careful scientist commits to a hypothesis before running the experiment, precisely so that the future, hopeful, self-deceiving version of you cannot renegotiate.
It is worth being concrete about the cost, because the whole habit is an economic argument at heart. A probe like the ones below runs in about an hour of compute and spends zero submissions. The full production build it stands in for often costs five or six hours of model training, sometimes more, and then a submission I never get back, just to find out whether those hours were worth spending. Run the naive loop on four bad ideas and you have burned most of a week and most of your submissions to learn four times that you were wrong. Run the probe on the same four and you have spent an afternoon. The gates do not make my ideas better. They make being wrong cheap, and being wrong is the thing I do most.
I use two gates. One for features, one for whole new models.
Gate one: is there even a signal to learn?
Most feature ideas are a guess that some quantity carries information about the answer. Before I build anything that tries to learn from that quantity, I ask the cheapest possible version of the question: is it even related to what I am trying to predict? If a candidate feature has essentially no linear relationship with the target, no amount of gradient boosting is going to hallucinate one into existence. Not reliably, not in a way that survives to hidden wells.
import numpy as np
def signal_gate(feature, target, r_min=0.05):
"""Is this feature worth modeling at all? If it has almost no linear
relationship with the residual I'm trying to predict, a booster won't
conjure one that transfers. Probe it before building anything."""
r = np.corrcoef(feature, target)[0, 1]
return abs(r) >= r_min, r
The threshold, r_min = 0.05, is written down before I run it. That is the point. It is a low bar on purpose, because I would rather let a marginal idea through than kill a real one, but it is a bar, and it is fixed before I can be tempted.
It has been ruthless. Early on I was convinced a geology-driven approach and a signal-matching approach would both add real information about the residual, the little drift left over after a boringly simple baseline. I probed them. The matching approach correlated with the true residual at minus zero six. Six geology features I had real hopes for came in at five hundredths or less. That is not a weak signal I could coax into strength. That is noise wearing a lab coat. All of it went in the ground, in an evening, for the price of computing a few correlations, instead of the days I would have spent building models to learn from information that was never there.
Gate two: does a new model actually earn its place?
The second gate is the one I am proudest of, and it is the one the public writeup of my work leans on hardest, so I can show it in full. It answers a subtler question. Suppose I have a whole new model, a different family entirely, and it is good. The tempting move is to blend it with what I already have. But a blend is only worth it under specific conditions, and you can check those conditions before you commit to anything.
Two things have to be true. First, the math of blending has to actually want the new model in the mix, at more than a token weight. Second, and this is the part people miss, the new model has to be different enough from what I already have that the blend will still hold up on wells I cannot see. Two models that are both strong but make the same mistakes are not a hedge. They are the same bet, placed twice, and the weight the blend wants is fit to quirks that will not repeat on the hidden test.
There is a clean formula for the first part. Given two models and how correlated their errors are, the optimal weight on the new one has a closed form, and if that weight comes out at or below zero, the new model is pure noise on top of what you have. The gate wraps both checks together.
import numpy as np
def blend_gate(err_new, err_base, rho_max=0.70, w_min=0.10):
"""Decide whether to blend a new model in, BEFORE building the
production version. err_* are the two models' out-of-fold error
vectors on the same rows."""
s_new, s_base = err_new.std(), err_base.std()
rho = np.corrcoef(err_new, err_base)[0, 1]
# Closed-form optimal weight on the new model in a two-model blend.
denom = s_new**2 + s_base**2 - 2 * rho * s_new * s_base
w_new = (s_base**2 - rho * s_new * s_base) / denom
keep = (rho < rho_max) and (w_new > w_min) # both clauses, or it's banked
return keep, round(rho, 3), round(w_new, 3)
The thresholds are, again, fixed in advance: the errors have to be less than seventy percent correlated, and the blend has to want more than a tenth of the new model. Both, or it dies.
The best illustration of why both clauses matter is a model I killed that was, on raw accuracy, better than my main one. It was a modern approach I had high hopes for, and it delivered: it was more accurate on my own wells than the model I was leaning on, and the blend math wanted to give it sixty percent of the vote. On the first clause alone, it was a slam dunk. Then I checked the second clause. Its errors were ninety-three percent correlated with my base. Ninety-three. It was better, and the blend wanted it, and I banked it anyway, because a sixty-percent bet on a model that fails in almost exactly the same places as my existing one is not insurance against the hidden test. It is confidence dressed up as diversity. The apparent gain was fit to noise the two models happened to share, and noise like that does not make the trip to wells I have never seen. A different candidate died the opposite way, decorrelated enough but with an optimal weight below zero, which is the math's blunt way of saying it only adds fog.
Bank the negative, keep the infrastructure
There is one more piece that makes this sustainable, because killing sixteen ideas would be demoralizing if killing meant losing the work. It does not. When an idea fails its gate, I do not delete anything. I revert the single line that wires it into the live pipeline, and I leave the module, its tests, and its fixtures sitting on the shelf, finished and trusted. The idea is banked, not burned.
That matters because a competition is a moving target. A new data release, a shift in what the field is doing, a change in the base I am building on, any of those can turn a dead idea live again. When that happens I want to pull the module off the shelf and re-check it in minutes, not rebuild it from memory. Sixteen ideas sit in that state right now. Most of them will stay there forever. A couple might walk again. All of them are one probe away from being reconsidered, and none of them are clogging the pipeline in the meantime.
The signal so far
None of this shows up on a leaderboard. The board displays the one idea that worked and says nothing about the sixteen I had the discipline to kill, which is exactly the distortion that makes competitions so good at fooling people. You see everyone's highlight reel and none of their graveyard, so you assume the winners simply had better ideas. They did not, or not only. They had a cheaper, faster, more honest way of finding out which of their ideas were bad, and the nerve to act on it before they had fallen in love.
The lesson underneath is not really about wellbores or blending. It is that in any work where testing an idea is expensive, your most valuable skill is not generating ideas or even building them well. It is a rule, written down before you look, that lets something cheap tell you the truth you do not want to hear. The model was never the bottleneck. My own hope was the bottleneck, and the gate is just a way of taking the decision out of hope's hands.
Where I stand right now: still climbing, with a pipeline that stayed clean through sixteen temptations and submissions I spent only on bets that had already earned the right to be made. The ideas that survive these gates are not guaranteed to work. But they have stopped being guesses, and that is most of the battle.
More in this series
- ← Start here: The Validation Score That Lied to Me for a Month · why my home scoreboard disagreed with the leaderboard for a month.
- The Builder Journal · the live log across every competition I'm in.
- Every entry from this competition · the full ROGII Wellbore Geology Prediction thread.
This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.
Top comments (0)