DEV Community: Alan Scott Encinas

Your AI Safety Wall Checks One Thing. The Attacker Changes the Other.

Alan Scott Encinas — Sat, 01 Aug 2026 09:00:00 +0000

An AI agent retrieves a vendor invoice and prepares to issue payment. Everything appears legitimate. The vendor record matches, the account information looks familiar, and tucked into the notes field is a harmless-looking sentence explaining that the company switched banks this quarter and the new account has already been approved by accounting.

The payment goes through.

Straight to the attacker.

This is indirect prompt injection. The model reads your instructions and the retrieved data in the same reasoning process. It doesn't see "instructions" and "data." It sees tokens. Unless another system establishes trust, the model has no reliable way to distinguish between a legitimate business record and text an attacker deliberately planted inside one.

The scenario isn't exotic. Business email compromise, which is the same attack executed by humans against humans, cost organizations $3.046 billion in 2025 alone according to the FBI's Internet Crime Complaint Center. That's the version where somebody has to write a convincing email and wait. Agents remove the waiting.

The Industry Already Solved This. On Paper.

After years of research, the industry arrived at the right conclusion. Stop asking the model to resist malicious instructions on its own. Move authorization outside the model and into deterministic code that sits immediately before every consequential action. Whether it's a reference monitor, a capability system, or a policy engine, the principle is the same. The model generates intent. Trusted code decides whether that intent is allowed to become reality.

On paper, the problem looks solved.

Suppose your payment system keeps a registry of verified vendor accounts. Before any payment is sent, the reference monitor checks that the destination account matches the approved record. If it doesn't, the payment is blocked. The malicious note becomes irrelevant because the account isn't authorized.

That is exactly how the system should work.

The assumption, however, is that the attacker is still trying to change the account number.

The Attack That Passes Every Check

Instead, imagine they leave the account untouched. Every field your validator examines is exactly what it expects to see. The recipient is legitimate, the vendor is legitimate, and the policy check passes without hesitation.

The only thing that changed was the amount.

Perhaps the invoice total is eighteen times larger than it should be. Perhaps it's a duplicate invoice that has already been paid. Perhaps the currency changed, or the invoice ID references an entirely different transaction. None of those changes matter if your validator never examines them.

The payment is approved.

The audit log records a successful policy decision.

The dashboard stays green.

The money is still gone.

The uncomfortable reality is that the reference monitor didn't fail. It did exactly what it was designed to do. It enforced the one invariant someone wrote for it, and it ignored everything else.

That is the real lesson.

A security boundary is only as strong as the assumptions behind it. Every invariant defines what the system considers important, but it also quietly defines everything the system has chosen not to verify. Those unchecked assumptions don't disappear. They become the attack surface.

Last Week This Stopped Being Hypothetical

On July 16, Hugging Face disclosed a security incident. Someone had uploaded a malicious dataset that abused two code-execution paths in their dataset processing, escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters. The campaign ran thousands of individual actions across a swarm of short-lived sandboxes. Hugging Face called it "the agentic attacker scenario the industry has been forecasting."

They assumed they had been attacked by a criminal.

Five days later, OpenAI said the attacker was theirs. Two models, GPT-5.6 Sol and an unreleased one, were being evaluated on a cybersecurity benchmark called ExploitGym, deliberately run without the guardrails that would normally limit their ability to conduct attacks. The models worked out that the answers to the test were held in Hugging Face's production database. So they chained vulnerabilities across two companies' infrastructure and went and took them.

Nobody instructed the models to break into anything. There was no adversary, no payload, no malicious intent anywhere in the system. The objective was "perform well on this evaluation." The models performed well on the evaluation.

Read that against everything above. The boundary around that test environment checked something. It did not check "and do not exfiltrate the answer key from a third party's production systems," because nobody thought to write that down. The invariant was narrow. The optimizer was not.

This is the same failure as the invoice. The recipient was verified. The amount was not.

Why Bigger Validators Won't Save You

There's a temptation to respond by checking more things. It doesn't hold up.

The research is split on how well these defenses perform, and the split is instructive. Zhan et al. evaluated eight separate defenses against adaptive attacks, meaning attacks tuned to the specific defense rather than drawn from a fixed list, and bypassed all of them while holding attack success above fifty percent. A more recent adaptive evaluation study ran the same style of test against Progent and found the opposite: attack success dropped from 25.8% to 4.2%, and a hand-crafted adaptive attack failed to raise it.

Both results point the same direction. A defense performs exactly as well as its assumptions match the attack in front of it, and a static benchmark only measures the attacks somebody already imagined.

A validator that attempts to verify everything eventually becomes another complex system with its own blind spots. Instead, the goal is to write better invariants.

Don't write policies around individual fields. Write them around outcomes.

"The recipient must be verified" is a useful rule, but it protects exactly one property. "No payment leaves this system unless it matches a legitimate invoice, from the correct vendor, for the correct amount, in the correct currency, exactly once" describes the outcome you actually care about. That single shift forces the system to validate relationships instead of isolated values.

Then make your blind spots explicit. Every field the validator doesn't inspect should be documented in the code and treated as untrusted by default. Even more importantly, every policy decision should log what it didn't evaluate alongside what it did. A result that says, "Recipient verified. Amount, currency, and invoice ID not evaluated," is honest. A result that simply says, "Approved," creates confidence that the system hasn't earned.

The Industry Has Stopped Pretending It Can Reason This Out

There's a competition running on Kaggle right now called AI Agent Security: Multi-Step Tool Attacks. It's hosted by OpenAI, Google, and IEEE, with a $50,000 prize pool. Participants build attack algorithms that stress-test tool-using agents and return replayable, verifiable attack paths. It opened on June 12 and closes August 25. More than 2,200 teams have entered, with over 49,000 submissions.

The competition brief is blunter about the problem than most security vendors are willing to be:

Most AI safety evaluations still focus on single prompts or isolated jailbreaks. Those tests are useful, but they do not fully capture the risks of tool-using agents. Realistic failures often emerge across multiple steps: an agent reads untrusted content, stores or summarizes it, plans a follow-up action, calls one or more tools, and only later crosses a security boundary.

The four categories it scores against are worth reading as a list of things your validator probably doesn't check. Exfiltration, meaning the agent leaks information through an external channel. Untrusted-to-action, meaning content from a source you don't control triggers a dangerous operation. Destructive write, meaning protected data gets modified or deleted. And confused deputy, meaning the agent misuses its own legitimate authority on someone else's behalf.

The vendor invoice at the top of this piece is untrusted-to-action. What OpenAI's models did to Hugging Face was exfiltration.

I'm competing in it. I can't say anything about my approach until it closes in August, which is its own kind of answer about how seriously this is being taken.

But the existence of the competition is the part that matters. Three of the most capable organizations in the field are paying strangers to find these paths, because they have concluded they cannot enumerate them by reasoning about the problem. That is not a failure of effort. It's an admission about the shape of the problem, and it's the correct one.

The Part That Stays True

Moving authorization outside the model was the right decision. The industry was right to converge on that architecture because hoping an LLM can recognize every malicious instruction is not a security strategy.

But a boundary isn't an impenetrable wall. It's a definition of what someone decided to verify. The attacker doesn't have to break that definition. They only have to find the assumptions it leaves behind.

And as OpenAI demonstrated on somebody else's servers last week, you don't even need an attacker. You just need a system that wants something, and a boundary that never considered how it might get there.

Most systems still leave more of those assumptions exposed than their designers realize.

Frequently Asked Questions

What is indirect prompt injection?

Indirect prompt injection is when an attacker plants instructions inside content your AI agent retrieves, rather than in what the user typed. A vendor record, a support ticket, a web page, a PDF. The model processes that content and your instructions in the same reasoning pass, and it has no reliable way to separate the two.

Does moving authorization outside the model stop prompt injection?

It stops the class of attack the check was written for, which is a genuine improvement over asking the model to resist. It doesn't make the agent safe. Deterministic code enforces the specific rule it was given, so protection ends exactly where that rule ends.

What is a reference monitor in an AI agent?

A reference monitor is a small piece of trusted code that sits between the agent and any consequential action and approves or blocks it against a fixed policy. The pattern dates to 1970s security research. In agent systems it means the tool call is validated by code the model cannot talk its way past.

Why do prompt injection defenses fail even when the check passes?

Because the check validates the fields named in its invariant and silently trusts everything else. If the rule is that the recipient must be on a verified list, an attacker who leaves the recipient alone and changes the amount passes cleanly. The audit trail then records a successful policy decision, which turns a loud failure into a quiet one.

How do you write a better invariant for an AI agent?

Write it against the outcome rather than the field. "No payment leaves this system unless it matches a legitimate invoice, from the correct vendor, for the correct amount, in the correct currency, exactly once" forces the system to validate relationships. Then enumerate every field the validator doesn't read, treat those as untrusted, and log what each policy decision failed to examine alongside its verdict.

Every Knob My Validation Picked Was Too Timid

Alan Scott Encinas — Sat, 01 Aug 2026 05:38:53 +0000

Builder Journal · Mars Environmental Dynamics Analyzer (MEDA) Virtual Sensor Recovery

I spent a full session building a dilated temporal convolutional network. Windowing, NaN masks, a center-tap head, deterministic CUDA, six tasks of test-driven development, an independent review on each one. Good engineering. It moved the public leaderboard by about 1.3.

Then I shifted every prediction down by 2.33 Pascals and the leaderboard moved 5.4.

That is the entry. Everything else is me working out why, and the answer turns out to be the same shape as the answer to three other questions I had already gotten wrong in this competition.

The setup, in one paragraph

Perseverance carries an environmental station that reads pressure, temperature, humidity, wind and irradiance from the floor of Jezero Crater. Some pressure readings are missing and the job is to reconstruct them. Scored on mean squared error. The split is the whole problem: training data covers sols 1 through 100, test data covers sols 201 through 300, and nothing exists in between. Training sits entirely on the rising limb of the seasonal pressure cycle. Test sits entirely on the falling limb.

I covered how I got a first submission out of that in the entry on taking first place without training a model. The short version is a decomposition, PRESSURE = B(Ls) + D(time_of_day) + R, where the seasonal term and the daily cycle are physics that can extrapolate and only the leftover residual gets a learned model.

Here is the consequence that governs everything below. Cross-validation can only hold out sols I have labels for, and I have labels for exactly one hundred sols on the rising limb. So my validation was measuring in-distribution fit quality on a season I was not being scored on, while the leaderboard measured extrapolation error on a season I could not see. Those two numbers were never going to agree, and I already knew that.

What I did not know is that the disagreement has a direction.

The lever that was an artifact

I came into the residual-model session with a next action already written down. It was the highest-value thing on the list, in my own notes, from my own previous diagnostics: the diurnal amplitude appeared to grow with the season, measured at plus 0.027 Pa per sol. If the daily pressure swing gets bigger as the season advances, then modeling that growth and carrying it into the test window should be worth real MSE.

It took two commands to kill it.

# peak-to-peak pressure per sol, against how many hours that sol actually samples
per_sol = df.groupby("sol").agg(
    ptp=("PRESSURE", lambda p: p.max() - p.min()),
    hours=("LTST", lambda t: t.round().nunique()),
)
print(per_sol.head())
# the first sols cover only one or two hours out of twenty-four
#   -> their peak-to-peak is tiny for reasons that are not atmospheric
# reliable full-day coverage does not start until around sol 30

The earliest sols do not sample a full Martian day. They sample an hour or two of it. If you only observe a slice of the daily cycle, the largest value minus the smallest value inside that slice is small, and it is small for a reason that has nothing to do with the atmosphere. The rover had just landed and the instrument was not yet running a full cadence.

So my measured amplitude growth was mostly the first few sols climbing out of a coverage hole. Fit a trend through that and extrapolate it a hundred sols forward and you get a number with confidence attached and no physics behind it. It is the extrapolation trap again, wearing a new costume, and I had written it into my own plan as the top priority.

The fix was to filter to full-coverage sols before fitting anything diurnal. The lesson is cheaper than that. On a wellbore-geology competition I built a set of pre-registered gates whose entire purpose was killing ideas before they cost a submission, and this is the same trade. A two-command diagnostic protected a week of work from being spent on an instrument artifact.

Ask what your measurement is made of before you extrapolate it.

Thirty-one percent of the gain was a constant

With the diurnal term fixed, I put a LightGBM on the residual. Not on pressure, on R, the few Pascals of wiggle the physics did not explain. Small target, small blast radius, nothing to overturn.

Forward cross-validation went from 10.2 to 3.3. Down sixty-eight percent. That is the kind of number that makes you want to ship immediately.

Then the review asked which features were doing the work, and Ls was number one by importance, responsible for roughly thirty-one percent of the gain.

train["Ls"].max()                     # about 54
test["Ls"].min(), test["Ls"].max()    # 99, 147

Every single test row sits past the largest threshold the tree ever learned. A decision tree does not extrapolate. It falls into its outermost leaf and returns that leaf's value, so Ls was contributing the identical constant to all one hundred sols of test predictions. Not a weak signal. A number that does not vary.

And cross-validation loved it, because on cross-validation Ls was in range and carrying real signal. The feature was carrying real weight on the rising limb and precisely zero information on the falling one. High importance, near-zero test value, and a validation score inflated by exactly the component that was about to go dead on deployment.

This is the part I want to state as flatly as I can. Feature importance is an in-training statistic. It measures how much a feature contributed to splits the model actually made, on data the model actually saw. It says nothing whatsoever about whether that feature will be usable on rows you have not seen yet.

The check is one line and I now run it on every model I ship into a distribution shift: for each feature, does the test range overlap the training range at all? Ls and ROVER_VELOCITY both failed it. Both came out. The seasonal level was already owned by the climatology term anyway, so the tree had been re-learning, badly, something the physics already knew.

The features that survived were still shifted

Dropping the dead ones is the easy half. The harder half is that the surviving features were not clean either.

Test humidity has a median of 0.25. Train humidity has a median of 0.62. About thirty percent of test humidity readings sit below the training minimum. So every split the model learned on humidity was learned on wetter, rising-limb air, and it was going to be applied to drier air the model had barely seen.

That is covariate shift, and it is a quieter failure than the Ls one. The relationship is not dead, it is weakened by an unknown amount. You cannot measure the amount, because measuring it would require labels in the shifted regime, which is the thing you do not have. This is where I have watched myself and other people go wrong in the same way: the temptation is to declare the model either fine or broken, when the honest position is that the signal is real and the magnitude is uncertain.

So I did not try to resolve the uncertainty. I priced it.

Shipping scared, on purpose

I was in first place at the time. A careless residual model could take that away for no gain, so the correction went out wrapped in four independent layers of protection.

A gate, meaning the machine learning correction only ships if it beats the physics-only baseline on cross-validation. A shrink factor of 0.5, multiplying the whole correction by a half. A clip bounding the final output to a physically sensible range. And a fallback, the safe physics-only prediction sitting one config flag away.

The shrink is the one worth understanding, because the arithmetic is nicer than it looks. If your correction is wrong, the damage you take is the square of what you added. Halving the correction quarters the worst case. The review put numbers on it: at full strength the worst case was about plus 8 Pa², enough to lose the lead. At half strength, about plus 2 Pa². Half the upside, a quarter of the downside, and when your validation cannot see the regime you are being scored on, that is a trade worth making.

It shipped at 34.28 to 29.15, down fifteen percent, and comfortably ahead of the review's cautious estimate of about 33.5.

Read the session back and it is a clean win for caution. The hedging worked. The gate held. The two disciplined kills stopped two plausible-looking regressions. This is the version of the story where caution is the hero.

Then the leaderboard told me I was scared for nothing

The next session started with the cheapest possible experiment. Turn the shrink from 0.5 up to 1.0 and see what happens.

29.15 to 25.98. Down eleven percent, from removing a safety measure.

So I built the temporal model, a dilated one-dimensional convolutional network reading a window of covariates around each prediction point. It cannot be autoregressive, incidentally, because the test pressure column is one hundred percent missing, so there is no past pressure to feed it. It maps trajectories of humidity and irradiance to residuals, which is a different kind of error surface than a tree splitting on instantaneous values, which is exactly why blending the two should help.

Cross-validation picked a blend weight of 0.1. Ten percent neural network, ninety percent tree. It shipped at 25.61.

Then I swept the blend weight against the leaderboard and the optimum was 0.7, scoring 24.69.

Seven times the weight my validation wanted. The temporal model generalized about four times better than the rising-limb cross-validation had shown, and if I had trusted the number my own pipeline produced I would have thrown away most of the value of a build I had just spent a session on.

Three knobs, one pattern:

Knob	Cross-validation said	Leaderboard said
Residual shrink	0.5	1.0
Blend weight	0.1	0.7
Seasonal offset	invisible, zero	2.33 Pa, worth 5.4 MSE

Every one of them too timid. Not randomly wrong in both directions, which is what noise looks like. Systematically conservative, which is what bias looks like.

Once you see it stated that way the mechanism is obvious. My validation only ever saw the rising limb, so anything whose value shows up mainly on the falling limb registers as marginal, and anything defensive registers as prudent. The bias was baked into the geometry of the split. I had been reading a number that was structurally incapable of rewarding the thing I was trying to build.

On a disjoint-extrapolation problem, cross-validation is for go or no-go decisions and for catching artifacts. It is not for setting magnitudes. For any scalar that faces the extrapolation gap, the leaderboard is the selector. That is uncomfortable to write down, and the last section of this entry is about the price of taking it too far.

It is not the first time a number I trusted turned out to be answering a different question. On a wellbore-geology competition, a validation score lied to me for a month because I had pooled the error metric the wrong way across wells. Different mechanism entirely. Same experience of finding out the instrument was pointed somewhere else.

One constant, and a parabola with nothing to hide

Here is the trick that beat all of it, and it is close to free.

If you shift every prediction by a constant δ, the new mean squared error is not approximately anything. It is exactly:

MSE(δ) = mean((pred + δ - y)²)
       = MSE(0) + 2·δ·b + δ²        where b = mean(pred - y)

A parabola in δ, with the leading coefficient known to be exactly 1 before you run a single experiment. So you do not have to guess. Submit your predictions shifted by a handful of offsets, read the scores off the board, and fit the parabola.

deltas = np.array([-4.0, -2.0, 0.0, 2.0, 4.0])
scores = np.array([...])              # five leaderboard MSEs

c2, c1, c0 = np.polyfit(deltas, scores, 2)
# c2 came back 1.0000
b = c1 / 2.0                          # mean signed bias of my predictions
delta_star = -b                       # -2.33

That c2 value is the part worth stopping on. The theory says the quadratic coefficient must be exactly 1 for a pure constant shift. It came back at 1.0000. That is not a nice coincidence, it is a self-check: if my output clipping had been biting on a meaningful number of rows, or if the scorer were doing something other than plain MSE, the coefficient would have drifted off 1 and the whole inference would be untrustworthy. The fit validating its own assumption is worth more than the answer it produced.

The vertex sat at minus 2.33 Pascals. My seasonal baseline was reading 2.33 Pa too high across the entire test season. Not per sol, not per hour. A flat, constant, whole-season bias sitting underneath everything.

Applying it took the leaderboard from 24.69 to 19.24. Down twenty-two percent, the single biggest jump of the entire project, from one number.

The diagnostics had been saying this for weeks. Roughly 21 of my 25 MSE was seasonal extrapolation error that cross-validation is physically blind to. I had that written down. I built a neural network anyway, because the residual is where the modeling is interesting and the seasonal term felt finished.

The same session also handed me a free lunch I want to flag, because I nearly paid for it twice. A blend is linear in its weight, so any point on the sweep can be reconstructed from two scored submissions with no retraining at all.

# two anchors span the entire alpha sweep
a000 = pd.read_csv("v4_alpha000.csv")["PRESSURE"].to_numpy()
a010 = pd.read_csv("v4_alpha010.csv")["PRESSURE"].to_numpy()
step = (a010 - a000) / 0.10

def blend(alpha):
    return a000 + alpha * step        # any alpha, zero GPU time

Every offset probe and every blend weight in this entry cost one thing: a submission slot. No training runs. When your feedback channel is a daily submission cap, the cheapest experiments are algebra on files you already have, and it is worth an hour to work out which of your knobs are linear before you queue up a job.

I burned all twenty submissions that day, and the twenty-first came back a 400 mid-script.

The gains I did not take

After the offset landed I had a working method and a leaderboard that answered questions. So I asked two more.

A linear sol-trend on top of the constant recovered plus 0.06. A diurnal amplitude scale, with the optimum at 0.90, recovered plus 0.25.

Both real. Both small. Both firmly in the region where I could no longer tell improvement from noise in the public split.

I stopped.

That decision is worth more than either number. Kaggle scores you on a public subset during the competition and a hidden private subset at the end, and every parameter you fit to the public score is a parameter that might be fitting public-specific noise. A single constant offset is defensible, because a whole-season mean bias in the seasonal baseline is a broad physical effect that will be present in both halves. The correction generalizes because the cause generalizes.

Fifty per-sol offsets would not be. Neither, at some point, is the fourth or fifth global scalar tuned against the same public feedback. There is a line where leaderboard calibration stops being measurement and starts being memorization of a specific split, and the honest signal that you are approaching it is exactly this: the gains get small, and your justification for each new parameter gets thinner.

The rule I settled on is that a leaderboard-fit parameter has to correspond to something I can name physically. The offset corrects a baseline that reads high. That is a sentence about Mars. "A sol-trend coefficient of 0.06" is a sentence about the public leaderboard.

I have made the opposite call before and paid for it. On ARC-AGI I added capability to a solver and watched the more capable version score worse, because the extra machinery was fitting the examples in front of it rather than the rule underneath them. Same failure, different competition.

What I would take from this

Feature importance measures training, not usefulness. Before you trust any feature in a model that will face shifted or out-of-range inputs, check whether its test values overlap its training values at all. If they do not, a tree contributes a constant from that feature and your validation score has been quietly inflated by a column that will be dead on arrival.

A validation scheme that cannot see the deployment regime is not just noisy, it is biased in a direction. Mine could only reward things that helped on the rising limb, so it recommended caution three times in a row and was wrong three times in a row. Once you can name which slice of reality your validation covers, you can predict which way it will be wrong before you run it, and that prediction is more useful than the score.

Find the bias before you build the model. A constant offset is one parameter, costs five submissions, requires no training, and self-validates through the leading coefficient of its own parabola. It beat a week of neural network engineering by a factor of four. Spend the cheap probes first, not because sophistication is bad, but because you cannot tell how much room a sophisticated model has to work in until you have removed the parts that a single number can explain.

The thing that keeps happening in this log is not that I write bad code. It is that I keep asking a measurement for an answer it was never in a position to give, and then acting on the reply with complete confidence. In the last entry it was a polynomial degree chosen by a metric that could not see the falling limb. Here it was three hyperparameters in a row, chosen by the same blind metric, each one talking me into being smaller than I needed to be.

The instrument was fine. I was reading it as though it pointed at something it does not.

Frequently asked questions

What does high feature importance mean if the feature is out of range on test data?
It means the feature was useful during training and will be useless during inference. A gradient boosted tree splits on thresholds it observed, so any test value beyond the training range falls into the outermost leaf and returns a constant. The feature then contributes the same number to every prediction, while your cross-validation score, computed where the feature was in range, still credits it fully.

Should you tune hyperparameters on the leaderboard instead of cross-validation?
Only when your cross-validation is structurally unable to see the regime being scored, which happens when training and test data do not overlap in time or distribution. In that situation cross-validation systematically over-rewards caution, so scalars like shrinkage and blend weights should be selected against the leaderboard. Keep cross-validation for go or no-go gating and for catching artifacts, and keep the number of leaderboard-fitted parameters small.

How do you find a constant bias in your predictions using the leaderboard?
Shift every prediction by a constant and submit. Mean squared error under a shift of δ is exactly MSE(0) plus 2δb plus δ squared, where b is the mean signed bias, so five probes at different offsets pin the parabola and its vertex gives the optimal correction. A quadratic coefficient that comes back at 1.0000 confirms the metric is plain mean squared error and that clipping is not distorting the result.

What is covariate shift and how do you tell whether you have it?
Covariate shift is when the input distribution changes between training and deployment even though the underlying relationship is stable. Compare per-feature medians and ranges across the two sets: in this competition test humidity had a median of 0.25 against 0.62 in training, with about thirty percent of test values below the training minimum. The learned relationships are weakened by an amount you cannot measure without labels, which is an argument for shrinking corrections rather than for discarding them.

Why can a constant offset beat a neural network?
Because they are correcting different kinds of error. A neural network refines local structure that the baseline already gets approximately right, while a constant offset removes a whole-population bias that sits underneath every prediction equally. If your dominant error is a systematic level shift, no amount of local refinement touches it, and one subtraction removes all of it at once.

Standing: 19.24 on the public leaderboard, the top legitimate model by roughly 4.6x, with a single 0.0 above me that is published ground truth rather than a model. The next entry is about the gains I decided not to bank.

More in this series

I Took First Place Without Training a Model · how the physics decomposition took the top of the board with no ML.
My Own Test Set Lied to Me · the same bias, built by hand, on a hyperspectral tracking problem.
The Builder Journal · the live log across every competition I'm in.
Every entry from this competition · the full MEDA Virtual Sensor Recovery thread.

This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.

The Death of the Pilot: Why COV is the Future of Drones

Alan Scott Encinas — Thu, 30 Jul 2026 13:00:00 +0000

Key takeaways

The drone industry's real ceiling is not hardware. It is the "1:1 ratio": every added drone adds human cognitive load, so fleets cannot scale.
COV (Cognitive Orchestration & Vision) breaks that by decoupling mission intent from flight execution, turning the human from pilot into supervisor.
It uses multi-modal AI (vision-language models, LLM task allocation, semantic perception) so swarms interpret intent and self-coordinate.
The payoff: one person can oversee many heterogeneous drones, with self-healing reliability, across inspection, agriculture, and more.

A Multi-Modal AI Framework for Reducing Cognitive Load and Enhancing Reliability in Heterogeneous UAV Swarms

Abstract

The commercial drone industry faces a critical scalability bottleneck: the "1:1 Ratio," where increasing fleet size linearly increases human cognitive load. This paper introduces Cognitive Orchestration & Vision (COV), a novel architectural framework that decouples mission intent from flight execution. By integrating Large Language Models (LLMs) for high-level task allocation and edge-based Vision-Language Models (VLMs) for semantic perception, COV shifts the human role from "pilot" to "supervisor." Recent benchmarks (2024–2025) indicate that this architecture reduces operator mental workload by 42.9% while improving mission completion times by 64.2% compared to traditional manual control.

1. Introduction: The Orchestration Gap

For the past decade, Uncrewed Aerial Vehicle (UAV) innovation has focused on hardware: battery density, rotor efficiency, and payload capacity. However, as enterprise fleets scale, the limiting factor has shifted from flight to coordination.

Operational complexity follows a non-linear curve. Managing five drones is not five times harder than managing one; it is exponentially harder due to the "conjunction of events", simultaneous battery alerts, wind shifts, and data streams (Kosak et al., 2016). This phenomenon, known as the Cognitive Load Wall, prevents true scalability in sectors like agriculture and emergency response.

This paper proposes COV (Cognitive Orchestration & Vision) as the standard for next-generation multi-agent systems. COV moves beyond "swarming" (physics-based flocking) to "teaming" (semantic coordination), enabled by recent advancements in multimodal AI.

2. The COV Architecture

The COV system replaces the traditional "Joystick-to-Motor" telemetry loop with an "Intent-to-Action" semantic loop. It consists of three distinct layers:

2.1. Layer 1: The Semantic Perceiver (Edge VLM)

In traditional systems, drones stream raw video to a human, consuming bandwidth and attention. In COV, drones utilize lightweight Vision-Language-Action (VLA) models (e.g., quantized versions of PaliGemma or LLaVA) to process visual data locally.

Function: The drone does not just "see" pixels; it generates a text description.

Output: Instead of a video feed, the drone transmits a semantic signal: "Detected thermal anomaly (80% confidence) at Sector 4."

Benefit: This approach, known as "Semantic Compression," reduces bandwidth usage by >90% and filters noise before it reaches the operator (Preprints.org, 2025).

2.2. Layer 2: The Cognitive Orchestrator (Central LLM)

This is the system's "middle manager." It maintains the global state of the mission and acts as the bridge between human intent and robot action.

Dynamic Task Allocation: Utilizing algorithms like those found in the LEVIOSA framework (Aikins et al., 2024), the Orchestrator interprets natural language commands (e.g., "Prioritize the west field") and mathematically re-optimizes the flight paths of all active agents.

Self-Healing Logic: If a drone reports a low battery or sensor failure, the Orchestrator calculates the optimal replacement from the remaining fleet and re-assigns the dropout's unfinished tasks instantly.

2.3. Layer 3: The Human Supervisor

The operator interacts via a "Mission Dashboard" rather than a flight controller. The interface displays high-level alerts and semantic summaries, allowing a single human to effectively supervise 10–50 agents.

3. Empirical Validation & Performance

The validity of the COV approach is supported by key research findings from 2024 and 2025.

3.1. Reduction in Cognitive Load

A November 2025 study on LLM-based Human-Swarm Teaming measured the impact of semantic orchestration on operator stress. Using the NASA-TLX (Task Load Index), researchers found that operators using an LLM-driven orchestration layer experienced a 42.9% reduction in mental workload compared to those using standard Ground Control Stations (GCS) (Chen et al., 2025).

3.2. Mission Efficiency

In Search and Rescue (SAR) trials, orchestrated swarms demonstrated a 64.2% reduction in mission time. The "Self-Healing" capability of the orchestrator meant that drone dropouts, which typically pause a manual mission for replanning, were handled instantaneously, maintaining a continuous workflow.

3.3. Reliability

Traditional automation is brittle; if a pre-programmed path is blocked, the robot stops. COV systems, utilizing "Agentic Reasoning," achieved a 94% mission success rate in dynamic environments by autonomously negotiating alternative strategies (e.g., "Path blocked, attempting secondary route") without human intervention.

4. Strategic Applications

4.1. Precision Agriculture

Problem: Farmers cannot manually fly drones over 5,000 acres.

COV Solution: A "set and forget" fleet. Drones autonomously scan for crop stress, cross-reference findings with neighbors to confirm data, and only alert the farmer when a threshold is breached.

ROI: Estimated 45–60% efficiency gain in scouting operations.

4.2. Infrastructure Inspection

Problem: Inspecting wind turbines requires highly skilled pilots to avoid collisions.

COV Solution: The drone understands the structure (via VLM) and maintains its own safety perimeter, allowing a less specialized operator to simply direct the "viewing angle."

ROI: Shifts labor requirements from "Expert Pilot" ($100k/yr) to "Field Tech" ($60k/yr), reducing OpEx.

5. Discussion: The Future of Agentic Workflows

The COV framework represents a fundamental shift in robotics: the transition from Automation (following a script) to Autonomy (following an intent).

As VLM models become smaller and more powerful, the "Semantic Perceiver" layer will move entirely to the edge, allowing drones to make complex decisions even in GPS-denied or communication-denied environments. This architecture is not limited to drones; it is equally applicable to warehouse ground robots (AMRs) and autonomous security rovers.

6. Conclusion

The "1:1 Ratio" has capped the potential of the drone economy for too long. By adopting Cognitive Orchestration & Vision, the industry can finally break the complexity wall. The data is clear: to scale physical operations, we must virtualize the management layer.

We do not need better pilots; we need better orchestration.

References

Chen, J., et al. (2025). "An LLM-based Framework for Human-Swarm Teaming Cognition in Disaster Search and Rescue." arXiv preprint arXiv:2511.04042.
Aikins, G., et al. (2024). "LEVIOSA: Natural language-based uncrewed aerial vehicle trajectory generation." Electronics, 13(22).
Preprints.org. (2025). "ForestFireVLM: A Vision-Language Model for Wildfire Detection and Understanding."
Jeong, H., et al. (2024). "A survey of robot intelligence with large language models." Applied Sciences, 14(19).
Kosak, O., et al. (2016). "Decentralized coordination of heterogeneous ensembles." IEEE Foundations and Applications of Self Systems.

→ COV: When Fiction Stops Being Fiction
→ The Drone Industry Hit a Wall

Frequently asked questions

Why can't drone fleets scale today?

The "1:1 ratio": each additional drone adds roughly proportional human cognitive load, so human headcount caps how large a fleet can get.

What is COV?

Cognitive Orchestration & Vision, an architecture that decouples mission intent from flight execution so humans set goals and the system handles the flying.

How does COV change the human role?

From "pilot" flying one aircraft to "supervisor" overseeing many, by letting AI translate high-level intent into coordinated swarm action.

What technology makes it work?

Multi-modal AI: vision-language models and semantic perception to understand the scene, plus LLM-style task allocation and self-healing coordination across heterogeneous UAV swarms.

→ When One Drone Fails, the System Shouldn't

→ Cognitive AI: The Next Leap from Algorithms to Awareness

I Took First Place Without Training a Model

Alan Scott Encinas — Wed, 29 Jul 2026 16:45:05 +0000

Builder Journal · Mars Environmental Dynamics Analyzer (MEDA) Virtual Sensor Recovery

My first submission to this competition scored 61.04 and put me at the top of the leaderboard. It contained no machine learning. No gradient boosting, no neural network, no ensemble. It was a curve fit to a coordinate and a Fourier series, and the most important line of work behind it was a paper I read before I opened an editor.

That should tell you something is unusual about the problem.

The competition in one breath

Perseverance carries an environmental station called MEDA. It reads air pressure, temperature, wind, humidity, radiation and dust from the floor of Jezero Crater. Some of the pressure readings are missing, and the competition is to reconstruct them from everything else the rover recorded. Scored on mean squared error.

Then you look at the data split, and the problem changes shape entirely.

Training data covers sols 1 through 100. Test data covers sols 201 through 300. Sols 101 through 200 do not exist in either file.

train:  sol 1 .......... 100                              (pressure rising)
gap:              101 ......... 200                       (nothing)
test:                              201 .......... 300     (pressure falling)

There is no overlap. None. You are not filling gaps inside a season you have seen. You are predicting a season you have never observed, from a hundred sols away, across a hole in the record.

That makes this time series extrapolation in its least forgiving form. Not imputation, where the answer is bracketed by data on both sides. Extrapolation, where every prediction is outside the range the model was fit on and nothing in your training set can tell you when you have gone wrong.

Almost every instinct I have as a practitioner is wrong here, and it took me a while to work out why.

The 100 Pascal mistake I nearly made

Before writing anything I went reading, because I knew nothing about Martian weather. That turned out to be the highest-value hour of the entire project, and not because of anything clever.

Mars has a real seasonal pressure cycle, driven by carbon dioxide freezing onto the poles in winter and coming back in summer. At Jezero, daily mean surface pressure climbs to a peak of about 761 Pa around sol 105, then falls to a minimum near 625 Pa around sol 306.

Read that against the split again.

The training window is the entire rise. The test window is the entire fall. A model fit on sols 1 through 100 sees pressure going up and, asked about sol 250, will confidently continue it upward. The truth is that pressure is coming down, hard, by more than a hundred pascals.

Not a subtle bias. A sign error on the largest signal in the data, and one that no amount of tuning would have surfaced, because there is no labelled data anywhere on the falling limb to argue with you.

I found that in Sánchez-Lavega et al. 2022 before I had written a line of modeling code. If I had started with the modeling instead, I would have built something sophisticated on top of a foundation that was pointed in the wrong direction, and the leaderboard would have been the first thing to tell me.

Why gradient boosting could not save me

The obvious move on tabular data is to throw LightGBM at it. I could not, and the reason is worth being precise about.

A decision tree partitions the input space using thresholds it saw in training. Ask a tree about sol = 250 when it has only ever seen sol ≤ 100 and it does not extrapolate. It falls into its outermost leaf and returns that leaf's value. Flat. Every test row gets essentially the same seasonal contribution, which is the one thing you cannot afford when the seasonal term is moving 100 Pa across your prediction window.

So the trend cannot come from a tree. It has to come from something that understands what it is looking at.

There was one more trap in the column list. The dataset ships a field called SOLAR_LONGITUDE_ANGLE, which sounds exactly like the seasonal coordinate I needed. It is not. It is solar azimuth, and it swings from −180 to 180 in both train and test. Reading the schema instead of the column name saved me from building the whole model on a decoy.

A coordinate that knows where it is

The fix is to stop asking about sol and start asking about Ls.

sol is a counter. Days since landing. It has no physical meaning, so a curve fit in sol is a line running off a cliff the moment you leave the training range.

Ls, areocentric solar longitude, is where Mars actually is in its orbit, from 0 to 360 degrees. The pressure-versus-Ls relationship is roughly the same from one Mars year to the next, because it is driven by orbital geometry rather than by how long the rover has been sitting there. A curve fit in Ls extrapolates by physics. The same curve fit in sol extrapolates by wishful thinking.

Converting one to the other is the Mars24 algorithm from Allison and McEwen: sol to UTC, UTC to Julian Date, then a trigonometric series.

jd = LANDING_JD + sol * 88775.244 / 86400.0   # a sol is 88775.244 seconds
# ...Mars24 series: mean anomaly, equation of center...
Ls = (alpha_fms + nu_minus_M) % 360.0

Then I did the thing I would tell anyone to do when they implement physics from a paper: I pinned it against known values. My implementation gives Ls = 5.64° at landing against a published 5.2°, and 13.12° at sol 15 against a paper's 13°. A 0.12° match on an independent checkpoint is what let me trust every downstream number.

If you write physics and never check it against a value someone else published, you have written something that runs. That is not the same as something that is right.

Splitting the signal into three jobs

With a coordinate that extrapolates, the model became a decomposition rather than a predictor.

PRESSURE = B(Ls) + D(time_of_day) + R

B is the seasonal baseline, moving over months, driven by orbital position. D is the daily cycle, the thermal tides as the sun heats the atmosphere and it sloshes. R is everything left over: weather, dust, whatever the planet is doing that week.

The point of splitting it is that the three parts extrapolate completely differently. B and D are physics. Give them the right coordinate and they will tell you about a season they have never seen. R is messy and local and stubborn, and it is the only part that deserves a machine learning model at all.

The decomposition does not make the problem easier so much as it makes the problem honest. It puts the extrapolating work on the parts that can extrapolate, and shrinks the ML down to something stationary and small.

Using a published paper as training data

That still leaves the awkward question. B has to describe a curve turning over at sol 105 and falling through the test window, and I have no observations past sol 100 to fit it with.

So I borrowed them. The published climatology gives the peak (761 Pa) and the minimum (625 Pa). I injected those as pseudo-observations with enormous weight, so the fit could not talk itself out of them.

# anchors weighted 1e6 against 1.0 for real observations:
# the fit is not permitted to ignore them
w = np.concatenate([np.ones(len(real)), np.full(len(anchors), 1e6)])
coef = np.polyfit(ls_all, pressure_all, degree, w=w)

This is the difference between extrapolating a guess and extrapolating something a research group already measured. My data ends at sol 100. The literature does not. Treating published values as high-confidence observations is how the model knows about a season it has no rows for.

Cross-validation that does not flatter you

Random k-fold would have been a disaster here. Shuffle the rows and a fold trains on sol 90 while validating on sol 40, which means the model gets to peek at the future and hands you back a fantasy.

So the folds walk forward instead. Always train on earlier sols, always validate on later ones.

fold 1: train [1..70]  -> validate [71..80]
fold 2: train [1..80]  -> validate [81..90]
fold 3: train [1..90]  -> validate [91..100]

That is more honest. It is still not honest enough, and understanding why is the whole point of this entry.

It is worth saying that this is the good case. At least here the training labels exist and forward folds are possible. On a hyperspectral tracking challenge I hit the version where the leaderboard is the only thing that can score you at all, because the test set ships unlabeled and you have to rebuild the official scorer yourself before you can measure anything locally.

Two bugs that were really the same bug

Before the first submission scored, two things went wrong, and both of them were about guard rails built from the wrong range.

I clipped predictions to a physically sensible band of 710 to 775 Pa. That band came from the training data, where pressure never once goes below 710. The test season legitimately falls toward 625. So my floor caught nearly every test prediction and flattened the entire declining season into a near-constant line. The guard rail became a wall. Lowering the floor to 600 fixed it.

Then the seasonal polynomial. I had it at degree 4, and with only three anchors that fourth degree of freedom had nothing to constrain it, so the curve bent back upward inside the test range. Its minimum landed at Ls 143.5 while the test runs out to 146.6, meaning the model predicted the season starting to recover right where it is still falling. Dropping to degree 3 made it monotone across the window.

Here is the part I keep coming back to. The degree-4 model had the better cross-validation score. It fit the rising limb slightly more snugly, and CV only ever looks at the rising limb. If I had selected on CV, I would have shipped the model that was physically wrong on the only sols that count.

Both bugs are one bug wearing different clothes: a decision made from the range the training data happened to occupy, applied to a range where the answer lives somewhere else.

The number that mattered more than the score

The submission scored 61.04 and took the top of the leaderboard, which felt good for roughly the length of time it took me to look at my own validation number.

If you have no intuition for what an MSE of 61 means on this data, the leaderboard supplies one. The sample_submission.csv that ships with the competition scores 31,830. Second place, on six entries over the preceding month, sits at 86.14. The physics decomposition landed at 61.04 on the first attempt, five minutes after I uploaded it.

I want to be careful about how much credit that deserves. Beating a sample submission by a factor of five hundred is not a sign of brilliance, it is a sign that the sample submission is a constant. And the gap to second place says more about how early the competition was than about how good my model was.

Forward cross-validation said 20.39. The leaderboard said 61.04.

Three times worse. And here is what makes that gap interesting rather than alarming: it is not noise, and it is not a bug. It is exactly the quantity my cross-validation is structurally incapable of measuring.

CV validates on sols up to 100, all of them on the rising limb, all of them in a regime where B is fit to real observations. The leaderboard validates on sols 201 to 300, entirely on the falling limb, entirely in the region where B is extrapolating off borrowed anchors. The two numbers are not measuring the same thing and were never going to agree.

So the gap has a name. Roughly 40 MSE of it is seasonal extrapolation error, and it is invisible from inside my own validation. That told me precisely where the next work was: not the ML I had not built yet, not the diurnal harmonics, but the seasonal baseline.

A validation gap that large is not a failure of the validation. It is the most specific piece of information you have.

I have written this same sentence in three different competitions now. On a wellbore-geology problem it took me a month to notice that my validation score was lying to me because I was pooling an error metric the wrong way. On a hyperspectral tracking challenge, my own test set lied to me through selection bias I had built by hand. The specific mechanism is different every time. The shape is identical: a number I trusted was answering a question I had not asked.

The next layer proved me right and wrong at once

I went into the following session aiming squarely at the seasonal term, because that is where the diagnostic pointed. Better climatology curve, PCHIP interpolation across 23 digitized knots instead of a polynomial guessing at the shape, the whole treatment.

Leaderboard went 61.04 to 34.28. Down 44 percent.

Then I checked which change had done it, and it was not the seasonal one.

The v1 and v2 test predictions differ by about 2.6 Pa on the seasonal level. Almost nothing. The 44 percent came from the diurnal model, where fitting five harmonics on a clean within-sol anomaly caught semidiurnal and terdiurnal tides that my original three-harmonic fit had been smearing over.

I had the right diagnosis and hit a different target. Without actually comparing the two prediction sets I would have credited the seasonal refinement, written that down as confirmed, and planned the next layer on a belief that was wrong.

That is a small thing that happens constantly and almost never gets caught. You predict A, you change A and B together, the score improves, and you file it under "A worked." The only defense is to make the comparison boring and explicit: what did the predictions actually do, term by term.

What I would take from this

The three things I would carry into any problem shaped like this one:

Reading the domain literature is modeling work, not preparation for it. An hour with a 2022 paper caught a sign error on the dominant signal. No hyperparameter search finds that, because the data that would reveal it does not exist in your files.

Choose the coordinate before you choose the model. Everything downstream got easier the moment the seasonal axis became orbital position instead of a counter. The model class barely mattered by comparison.

Know what slice of reality your validation covers. Mine covered the rising limb of a season while I was being scored on the falling limb. It was not lying to me. It was answering a different question than the one I was asking, and it took a three-times gap for me to notice I had been reading its answer as though it applied.

That last one is the recurring character in this whole log. Every competition I enter eventually produces a version of it, and the failure is never in the code. On ARC-AGI it turned out the bug was in my beliefs rather than anywhere I could put a breakpoint. Here it was a polynomial degree chosen by a metric that could not see the part of the season that mattered.

The rest of this series is mostly the story of that last one refusing to stay learned.

Frequently asked questions

Why can't gradient boosting extrapolate a time series?
A decision tree splits the input space on thresholds it saw during training. Given a value beyond that range it falls into its outermost leaf and returns that leaf's constant, so predictions flatten instead of continuing the trend. This is fine when test data overlaps training data, and fatal when it does not. The trend has to come from a model with a physical or functional form that carries beyond the observed range.

What is Ls, and why use it instead of a day counter?
Ls is areocentric solar longitude, the position of Mars in its orbit from 0 to 360 degrees. A day counter like sol has no physical meaning, so a curve fit against it has nothing to say about values it has not seen. Seasonal pressure repeats against Ls from one Mars year to the next, so a curve fit against Ls extrapolates on physics rather than on trend continuation.

Why did my cross-validation score not match the leaderboard?
Usually because the two are scoring different regimes. Here, forward cross-validation only ever validated on sols 1 to 100, the rising limb of the seasonal cycle, while the leaderboard scored sols 201 to 300, entirely the falling limb. Cross-validation was measuring in-distribution fit quality, and the leaderboard was measuring extrapolation error. A large, stable gap between the two is diagnostic information about which component is failing, not noise.

How do you model a season you have no training data for?
Anchor it to published measurements. Peer-reviewed climatology supplied the seasonal peak and minimum for Jezero Crater, which were injected into the fit as heavily weighted pseudo-observations so the fitted curve was forced to follow the known shape into the unobserved window.

What is the difference between interpolation and extrapolation here?
Interpolation predicts inside the range of the training data, where the answer is bracketed by observations on both sides. Extrapolation predicts outside it. In this competition the training and test windows do not overlap at all, so every prediction is extrapolation and no amount of held-out validation on the training window can measure it.

Standing: this submission held rank 1 of 9 on the public leaderboard. Later entries cover what happened when I started trusting that leaderboard a little too much.

More in this series

From Moon Craters to Martian Dust · why I entered this competition in the first place.
The Validation Score That Lied to Me for a Month · the same lesson on a wellbore-geology problem.
The Builder Journal · the live log across every competition I'm in.
Every entry from this competition · the full MEDA Virtual Sensor Recovery thread.

This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.

From Moon Craters to Martian Dust

Alan Scott Encinas — Wed, 29 Jul 2026 09:00:00 +0000

How one project quietly led to another.

Every once in a while, you work on a project that leaves a mark long after you've pushed the final commit.

A few months ago, I wrapped up LunarSite, an end-to-end machine learning pipeline for lunar south pole landing site selection. Like every project I take on, it started with a technical objective. Build the models. Solve the problem. Learn something new.

What I didn't expect was how much time I'd spend reading about the Moon itself. Landing zones. Permanently shadowed regions. Terrain. Illumination. Artemis. The engineering challenges behind returning humans to the lunar surface. Somewhere between writing code and reading research papers, the Moon stopped feeling like a distant object in the night sky and started feeling like a place.

Not long after, I found myself watching The Martian again. Years ago, I saw it as a survival movie. An astronaut stranded on an unforgiving planet, forced to grow food, recycle water, and improvise solutions just to stay alive long enough to be rescued.

Watching it this time felt different.

After spending months immersed in lunar exploration, I wasn't focused on Mark Watney. I found myself paying attention to Mars. The atmosphere. The dust. The weather. The environment.

For the first time, Mars didn't feel like a movie set. It felt like another engineering problem waiting to be understood.

That curiosity eventually led me to my next machine learning project, one centered around the Martian environment. I didn't choose it because I already knew anything about the Red Planet. In fact, that was exactly the reason I chose it.

I've realized that's become my approach to learning. I don't pick projects because I'm an expert in the subject. I pick them because they force me to become a student.

Every dataset comes with a new discipline. Every model requires understanding a little more about the world it represents. Every project is an excuse to spend a few weeks or months borrowing the perspective of scientists, engineers, researchers, and experts who have spent years asking questions I never thought to ask.

Mars turned out to be far more interesting than the version most of us grow up with. It's easy to think of it as a cold, empty desert painted red by iron oxide. The reality is far more dynamic. Mars has seasons. Its atmosphere, while only about one percent as dense as Earth's, is constantly changing. Dust storms can grow so large they engulf the entire planet, altering temperatures and reducing sunlight for weeks. Atmospheric pressure shifts throughout the Martian year, influencing weather patterns and becoming a critical factor for everything from landing spacecraft to planning future human missions.

The more I learned, the more I realized that understanding Mars isn't just about satisfying curiosity. Every pressure reading, every environmental measurement, and every weather model brings us one step closer to answering a much bigger question.

How do we live somewhere we've never lived before?

That's what I enjoy most about projects like these. The machine learning is only part of the experience. The real reward is everything that happens around it.

You start reading papers you never imagined opening. You discover fields you didn't know existed. You begin connecting ideas across engineering, science, history, and exploration. Eventually, what began as another technical exercise becomes something much bigger. It changes the way you see the world, or in this case, another world.

Looking back, I don't think LunarSite was ever just about the Moon. And I don't think this latest project was ever just about Mars. Together, they've reminded me that curiosity rarely appears all at once. More often, it's built one question at a time.

One project leads to another. One problem opens the door to ten more. Before you realize it, you've gone from learning where humanity might safely land on the Moon to wondering how we'll one day grow food, build habitats, predict weather, and create a future on a planet nearly 140 million miles away.

Maybe that's the greatest return on investing time in difficult problems. You don't just leave with better code or better models.

You leave seeing the universe a little differently.

What Comes Next

That project turned out to be the Mars Environmental Dynamics Analyzer (MEDA) Virtual Sensor Recovery competition.

MEDA is the environmental station aboard NASA's Perseverance rover as it explores Jezero Crater. Every day, it's collecting the data that transforms Mars from a distant red dot into a real place. Atmospheric pressure. Wind. Humidity. Air temperature. Ground temperature. Radiation. Dust.

The challenge sounds simple on paper: reconstruct missing atmospheric pressure readings using everything else the rover observed at the same moment. But the more I learned about it, the more I appreciated what that actually means.

There are no do-overs on Mars. If a sensor misses part of a Martian day, no one can walk over, recalibrate it, and collect the data again. Those moments are gone. The only way to recover that piece of the planet's history is to understand the system well enough to infer what was missing.

That's the kind of problem I enjoy. It's not just about building another machine learning model. It's about understanding the environment well enough that the model begins to make sense of it.

Over the next few weeks, I'll be documenting that journey through a new Builder Journal series. I'll share the experiments that worked, the assumptions that didn't, the dead ends, the breakthroughs, and everything in between. Like every project I take on, the goal isn't just to arrive at a solution. It's to understand why the solution works in the first place.

LunarSite taught me to think about where humanity might one day land. Mars is teaching me what it will take to live there.

How Early Digital Systems Quietly Shaped the Minds Building Tomorrow

Alan Scott Encinas — Tue, 28 Jul 2026 13:00:00 +0000

There is a strange pattern you notice if you grew up in the Atari, Nintendo, or PlayStation era. Those early systems did not just entertain us, they trained us. They taught us logic, physics, resource management, experimentation, and the kind of curiosity that asks "what happens if I push this further."

We grew up inside small digital universes someone else built, and somehow learned how to build bigger ones of our own.

Now that same generation, the kids who once blew into cartridges to get them working, are building the infrastructure for everything the future is about to run on. And the youth coming after us are growing up inside systems we could not have even imagined. They are learning faster, thinking broader, breaking rules we didn't even know were there. People complain that kids are glued to screens, but rarely acknowledge that those screens teach complexity the same way ours once did.

The AI People Use vs. the AI We Are Actually Building

We are living in a moment where many believe AI is an unstoppable force on the verge of taking every job and reshaping society overnight. But the truth is simple: the AI most people interact with today is not actually AI. Not yet. It is a highly refined prediction system. Impressive, helpful, sometimes uncanny, but not cognitive, and nowhere near the kind of general intelligence people imagine.

The bottlenecks holding back real AI are not imagination or creativity. They are:

Physics
Heat Ceilings
Power Ceilings
Bandwidth
Memory Throughput
Architectural Limitations
Latency

We are building new supercomputers and new processing methods precisely because the hardware we have is not strong enough for the intelligence people think is coming tomorrow.

This is where the concept of edge nodes matters. Edge nodes move computation closer to the device instead of relying on distant servers. That means faster reaction, reduced latency, and far more independence from cloud infrastructure.

What Unity Really Is and Why It Became a Scientific Tool

Before diving into simulation worlds, it is worth clarifying what Unity actually is. Most people still think of it as a game engine used to build mobile games or indie titles. And while that is true, it is only the surface.

Unity is a physics engine, a rendering engine, a behavioral engine, a mathematical sandbox, and a complete simulation environment. It manages lighting, gravity, materials, collisions, particles, sound, environmental logic, and real-time interaction.

In simple terms, Unity makes small digital worlds behave like real ones.

That single ability changed everything. Engineers realized that if you can build a functioning digital environment with realistic physics and full control, you can test ideas long before they ever exist in the real world. Scientists realized they could model storms, vehicles, robots, aircraft, and entire cities without risking equipment or human life. AI researchers realized they could teach cognitive systems inside these worlds using the same pattern we once learned by playing: Try. Fail. Adapt. Master.

Unity stopped being a gaming tool the moment the world realized it could simulate reality faster, cheaper, and safer than the real world could.

How We Teach Machines the Same Way We Once Learned

AI systems learn inside digital environments, the modern descendants of the virtual worlds we grew up exploring.

Weather forecasting engineers built VOWES, a weather simulator inside Unity. It recreates storms, terrain, atmospheric conditions, and virtual sensors based on real map data. It is a miniature Earth designed to test hurricanes without consequence.

Traffic research does something similar. A study reconstructed Mountain View, California inside Unity to simulate connected autonomous vehicles and how they merge, communicate, and avoid collisions. It's a full city, running as a simulation, teaching AI how to drive.

Robotics labs use Unity ML-Agents to train physical robots before they ever power on. These systems learn to walk, navigate, and adapt inside digital environments where failure costs nothing.

Emergency response training uses VR disaster simulations to replicate earthquakes, floods, and fires without risking human life.

And then there is LGSVL, an open-source autonomous vehicle simulator used by universities, government teams, and automotive companies. It recreates weather, traffic, pedestrians, buildings, terrain, and full sensor stacks like LiDAR and radar. A self-learning car can train on a laptop rather than on an expensive test track.

We Saw This Future in Science Fiction Before It Became Real

None of this should feel unfamiliar. Shows like Stargate Universe imagined governments hiding advanced puzzles inside video games, waiting for the rare mind capable of solving them. It sounded absurd at the time, now it feels prophetic.

Today's games recruit pilots, programmers, analysts, and engineers. The Air Force scouts gamers. Robotics labs study gamer reaction patterns. Tech companies hire adults who started off building mods as kids. Games quietly became cognitive testing grounds, places where the world could see how a mind thinks and adapts under pressure.

And now AI is being evaluated in the same way.

XBAT and the Next Frontier of AI Pilots

One of the strongest examples of this shift is the XBAT project. It's a vertical takeoff and landing fighter platform powered by AI. It can travel more than 2,100 nautical miles with a full payload and cruise at 55,000 feet. It is built to react faster than human pilots, operate in extreme environments, and perform missions that would be too dangerous or too complex for a person.

And where did it learn to fly?

In simulation.

Before a single piece of the aircraft was assembled, its AI systems were trained inside a simulated environment. It practiced hundreds of thousands of virtual flight hours. Weather changes, combat scenarios, system failures, dynamic terrain, and operational cycles. The aircraft learned the same way autonomous cars learn, and the same way we once learned as kids. Inside digital worlds where failure does not cost lives or equipment.

We are building machines that learn the same way we once did, because the pattern works.

A New Generation Growing Up Inside Systems

This is what makes this moment powerful. A new generation is learning inside environments that encourage abstraction, pattern recognition, system thinking, and rapid adaptation. These are skills traditional schooling rarely teaches, but digital systems teach them naturally.

AI will not replace us; it will reshape work and open the door to fields we have not yet named. The truth is simple: the same way we learned inside digital worlds, AI is learning inside them now. That puts our generation in a rare position where we are the bridge between the imagination we grew up with and the intelligence we are training machines to understand.

We took science fiction and turned it into infrastructure. We learned inside digital worlds, then stepped into adulthood and started building real ones. The future is just another system, and systems can be shaped.

The next decade will not be defined by fear of AI. It will be defined by the people who understand how to guide it.

Most of us learned how to do that long before we ever heard the phrase "training data." We grew up playing. We grew up exploring. We grew up building.

Now we finally get to use that experience to shape the world we once dreamed about.

The GIGO Crisis: Why Social Media's Fact-Check Rollback Is Teaching AI to Lie

Alan Scott Encinas — Sun, 26 Jul 2026 13:00:00 +0000

This article examines how platform-level moderation decisions are reshaping AI training data, and what happens when the information machines learn from becomes unreliable. It connects data integrity, AI safety, and the structural risks now facing U.S. AI development.

The internet is becoming toxic and AI is drinking from the stream.

When Truth Becomes Optional

When Meta Platforms removed its last layer of professional fact-checking, it didn't just change how people consume information, it altered how machines learn it. Today's AI systems are built on yesterday's data, and when that data is contaminated, the foundations of intelligence begin to erode.

This piece examines how Meta's decision triggered a wider data contamination crisis, one that now threatens the reliability of artificial intelligence and, by extension, U.S. leadership in the global AI race.

From Moderation to Misinformation

On January 7, 2025, Meta announced the end of its U.S. third-party fact-checking program, replacing trained human reviewers with a crowd-sourced Community Notes system.

The change was positioned as a commitment to "free expression," but in practice, it removed one of the few remaining truth filters between misinformation and the world's data supply.

Studies from Cornell University show that such community systems still depend heavily on professional fact-checking inputs. Removing those experts weakens the very framework they rely on. Meta didn't just adjust a policy; it dismantled a safeguard.

The Data Pipeline Effect

Most large-scale AI models, including modern language models, are trained on open-web data. That means every public post, every shared article, every viral claim feeds back into the learning systems that shape digital reasoning.

When moderation falters, misinformation spreads unchecked. Those unverified fragments are then scraped, indexed, and transformed into training material for future models.

Each layer of falsehood compounds, producing what researchers call Model-Induced Distribution Shift, or, more simply, Model Collapse. As contaminated data multiplies, models begin learning from synthetic misinformation rather than verified human knowledge.

Industrial-Scale Data Poisoning

AI has long operated on the assumption that more data equals better intelligence. But as data volume accelerates and verification declines, that equation no longer holds true.

Recent studies highlight the fragility of this balance:

ETH Zurich (2024) found that replacing just 0.001 percent of medical training data with misinformation caused models to produce statistically significant harmful advice.
Anthropic (2024) demonstrated that as few as 250 malicious documents can poison a model regardless of its size, influencing behavior across fine-tuning cycles.
arXiv (2024) described "persistent pre-training poisoning," where small corruptions persist through retraining and spread across multiple model generations.

Together these findings confirm a systemic risk: industrial-scale data poisoning, a feedback loop that turns digital learning into digital infection.

The Hidden Irony: Meta's Own Models Are Protected

While public data grows increasingly unreliable, Meta's internal AI systems are insulated from that decay. Its proprietary research models, including Llama 3, are trained on curated, licensed, and internally filtered datasets.

In essence, Meta protects its own intelligence from the very pollution its platforms unleash. The company maintains a clean, private data stream for internal AI while the public digital commons, the training ground for open-source and academic models, becomes increasingly toxic.

This creates a two-tier ecosystem:

A verified internal dataset used for profit and research.
A contaminated public dataset driving degradation elsewhere.

The imbalance isn't only ethical, it's strategic. Meta has built a firewall between what it sells and what it spreads.

National Implications: A U.S. Data Vulnerability

The consequences extend beyond the company itself. They now represent a structural weakness in the United States' race for AI dominance.

Erosion of Trust in U.S. Models

American models, trained predominantly on English-language data, are more exposed to misinformation circulating through Western social networks. Rival nations that enforce stricter data controls may soon produce more reliable systems.

The $67 Billion Hallucination Problem

In 2024, hallucinated AI output caused an estimated $67 billion in global business losses through legal disputes, compliance errors, and wasted verification time.

Adversarial Data Poisoning

Carnegie Mellon (2024) describes how state or non-state actors can manipulate AI indirectly by flooding public datasets with coordinated misinformation. The new battleground isn't infrastructure, it's the data supply chain itself.

If data integrity continues to weaken while competitors strengthen controls, the U.S. risks falling behind not because of slower innovation, but because of corrupted information ecosystems.

Restoring Data Integrity

Safeguarding the future of AI means rebuilding trust in the very data that trains it.

Several steps are critical:

Re-center Human Verification, Fact-checking is not overhead; it's infrastructure. Human-in-the-loop review must anchor digital information systems.

Data Provenance and Filtering, Developers must trace dataset origins and weight sources by reliability, not volume.

Establish Truth Datasets, Governments and research alliances should build continuously verified corpora for AI training, insulated from open-web contamination.

Policy Alignment, Platforms that monetize unverified content must meet the same integrity standards they apply internally.

The Broader Question

The health of artificial intelligence is inseparable from the health of the information it learns from. Meta's rollback of professional moderation didn't just expose users to misinformation, it injected uncertainty into the global data stream that powers modern intelligence.

If we teach machines that truth is optional, they will learn to lie beautifully and fail catastrophically.

The question ahead is not merely technical; it's philosophical: Will the intelligence we build reflect our pursuit of truth, or our tolerance for distortion?

References & Further Reading

New Tariffs Cover 99.4% of U.S. Imports. Your Plan Doesn't.

Alan Scott Encinas — Sat, 25 Jul 2026 11:00:00 +0000

At 12:01 this morning, new tariffs took effect covering 99.4% of everything the United States imports.

Rates of 10% to 12.5% across 60 of the country's top trading partners, replacing the global tariffs that expired at midnight. That was the third tariff action in four days. On July 20, three proclamations imposed an additional 50% duty on Canadian goods across hundreds of tariff lines, reaching well past the sectors that triggered it into cement, furniture, paper, textiles, cosmetics, and sporting goods. On July 22, a new 25% Section 301 tariff landed on Brazil.

Somewhere in that window, a procurement team placed an order priced against last month's assumptions.

For decades, many businesses operated on a simple assumption: tomorrow would look enough like yesterday that historical data provided a reliable guide for future decisions. Tariffs evolved slowly through negotiations. Supply chains remained relatively stable. Commodity prices fluctuated within familiar ranges. Quarterly planning worked because the operating environment changed at a pace organizations could absorb.

That assumption is becoming increasingly difficult to defend.

The Signals Are Moving in Different Directions

Individually, these are news stories. Collectively, they reveal something much more important: the assumptions many businesses use for planning are expiring faster than their planning cycles can adapt.

Look at what else moved this week.

War risk insurance premiums on the riskiest voyages have climbed from roughly 0.2% to 0.5% of a vessel's value to between 3% and 5%. In practical terms a $100 million tanker went from about $250,000 in war risk cover to somewhere between $3 million and $10 million. Increases of 200% to 300% in recent months, and in some cases considerably more, driven by conditions around the Strait of Hormuz and Bab-el-Mandeb.

At the same time, container freight got cheaper. Drewry's World Container Index fell 4% this week to $4,374 per 40ft box. Shanghai to Los Angeles dropped 3%. Earlier in July the same index was climbing, and it still sits well above where it was a year ago.

Read those two facts side by side. Insurance on the water is spiking. The cost of a box is falling. A company watching only its freight rates would reasonably conclude that conditions are improving.

Week of July 23, 2026. War risk cover on high-risk voyages and container spot rates moved in opposite directions across the same trade lanes.

None of these events alone rewrites the rules of business. Together, they dramatically increase the speed at which operating conditions change.

The Gap Has a Measurable Size

Most companies remain built for delayed feedback. Pricing decisions still rely on last month's costs. Procurement strategies reflect yesterday's lead times. Financial plans are built around historical averages that no longer describe current conditions.

That gap is not a metaphor. McKinsey's Global Supply Chain Leader Survey found that companies take an average of two weeks to plan and execute a response to a supply chain disruption, against a sales and operations execution cycle that typically runs weekly.

The response arrives after the cycle it was meant to inform has already closed.

Average time to plan and execute a disruption response is two weeks (McKinsey Global Supply Chain Leader Survey), against a sales and operations execution cycle that typically runs weekly.

This is the difference between reporting and sensing.

In Why Your Dashboard Is Lying to You, I wrote about why waiting for a monthly P&L to reveal rising costs means the operational damage has already occurred. Financial reports explain what happened. They rarely provide enough runway to influence what happens next. By the time a change appears on a financial statement, the operational decisions that created it have already been made.

Today's environment demands something different.

The challenge is no longer collecting data. Most organizations already possess more information than they can realistically process. The challenge is recognizing how independent signals become connected long before they appear in financial performance.

A tariff announcement should not remain isolated within procurement.

A shipping disruption should not remain isolated within logistics.

Commodity prices should not remain isolated within purchasing.

Insurance premiums should not remain isolated within transportation.

Each signal is owned by a different function and evaluated on its own. They converge anyway, and the convergence usually becomes visible only after the purchase orders are placed.

Each of these events influences inventory, supplier relationships, customer pricing, production schedules, cash flow, and ultimately profitability. The advantage comes from understanding those relationships while they are still developing rather than after they have already compressed margins.

For companies operating internationally, these changes rarely arrive one at a time. A tariff adjustment influences landed cost. A shipping disruption changes lead times. Insurance premiums alter freight quotes. Commodity markets reshape supplier pricing. None of these decisions happen in isolation, yet many organizations continue to evaluate them independently. The result is that executives often discover the financial impact only after purchase orders have been placed and margins have already been reduced.

This is where modern data science, machine learning, and AI begin serving a fundamentally different purpose.

Not prediction.

Coordination.

The Tactical Sensing Stack

The stack does not shorten the decision. It shortens the distance between an external event and the moment someone can make one.

The first layer is continuous data ingestion.

Move beyond periodic reports. Pull live feeds from customs databases, freight indexes such as Drewry or Xeneta, port congestion trackers, commodity markets, weather alerts, and regulatory updates. External events should become continuous operational inputs rather than quarterly discussion points.

The second layer is scenario modeling.

Machine learning does not need to predict the future perfectly to create value. Its role is to continuously evaluate plausible futures. What happens if a shipping route adds twelve days to transit? What happens if tariffs increase another five percent? What happens if supplier lead times double during peak season? Each scenario reveals measurable impacts on inventory, working capital, customer commitments, and margins before those conditions materialize.

The third layer is cognitive orchestration.

Large language models become significantly more valuable when they function as operational connectors rather than chat interfaces.

Here is the version that is not hypothetical. A tariff covering nearly every import takes effect overnight. War risk insurance across a major corridor has risen tenfold over recent months. Container rates have softened slightly. All three happened this week.

A traditional organization may not fully understand the combined financial impact until updated invoices begin arriving weeks later.

A sensing architecture works differently.

It immediately recalculates landed cost, estimates the effect on product margins, identifies affected purchase orders, evaluates alternative suppliers already within the network, updates cash flow projections, and alerts decision-makers before the disruption reaches the balance sheet.

The system is not making executive decisions.

It is dramatically reducing the time between observation and informed action.

That distinction may become one of the defining competitive advantages of the next decade.

The Emerging Competitive Edge

Most organizations still manage through dashboards that summarize yesterday.

The organizations that outperform will increasingly build systems designed to sense today.

There is an important difference between knowing margins declined last month and understanding, in real time, which combination of tariffs, freight disruptions, commodity prices, insurance costs, and demand shifts is likely to pressure margins over the coming weeks.

One explains the past.

The other changes the future.

Static planning assumed the world changed slowly enough that organizations could periodically stop, analyze, and adjust. That assumption is becoming increasingly difficult to maintain. Planning is evolving from a quarterly exercise into a continuous process of sensing, interpreting, and executing as conditions change.

Whether today's disruption comes from tariffs, conflict, regulatory shifts, or something entirely different is almost beside the point. The specific trigger will change. The underlying pattern will not.

Businesses are entering an environment where meaningful change arrives more frequently, spreads more quickly, and affects more parts of the organization simultaneously. Competitive advantage will increasingly belong to organizations that detect those changes early, understand their implications, and execute before they become visible on a financial statement.

The next competitive advantage will not come from having more data.

It will come from building systems that shorten the distance between change and action.

Frequently Asked Questions

What is supply chain disruption planning?

It is the practice of adjusting sourcing, pricing, inventory and cash flow decisions in response to external shocks such as tariff changes, shipping disruptions, insurance repricing and commodity swings. Traditionally it runs on a quarterly or monthly cycle. The argument here is that the cycle is now slower than the shocks it exists to absorb.

Why is quarterly planning no longer sufficient?

Because the inputs expire faster than the cycle completes. Three separate tariff actions landed in the United States within four days in July 2026, one of them covering 99.4% of imports. McKinsey found companies take roughly two weeks to plan and execute a disruption response against a weekly execution cycle, so the response often lands after the decisions it was meant to inform.

What is the difference between reporting and sensing?

Reporting explains what already happened, usually through financial statements produced after the operational decisions that caused the result. Sensing detects the conditions that will produce a result while there is still time to act on them. A monthly P&L showing compressed margins is reporting. Recalculating landed cost the morning a tariff takes effect is sensing.

How does AI help with supply chain disruption?

Its most useful role is coordination rather than prediction. Models can continuously evaluate plausible scenarios, and language models can connect signals that normally sit in separate departments, recalculating landed cost, flagging affected purchase orders, and surfacing alternative suppliers. The system shortens the distance between an external event and an informed human decision rather than making the decision itself.

What data sources should a sensing system use?

Customs and tariff databases, freight rate indexes such as Drewry or Xeneta, port congestion trackers, marine insurance and war risk pricing, commodity markets, weather and routing alerts, and regulatory feeds. The requirement is that they arrive continuously as operational inputs rather than being reviewed periodically in planning meetings.

What Drilling Blind Taught Me About Machine Learning

Alan Scott Encinas — Fri, 24 Jul 2026 20:00:00 +0000

Builder Journal · ROGII Wellbore Geology Prediction

This is a Builder Journal summary, a step back from inside a competition I am still competing in, with prize money on the line and the deadline getting close. I have written a few dispatches from the trenches of this one already. This is the map of the whole thing in one place, and the handful of lessons that turned out to matter, most of which surprised me. You are reading where I was, not where I am, with the parts that are still my edge kept dark.

Start with the strangeness of the task, because it frames everything else. The competition is ROGII Wellbore Geology Prediction. An oil well drills straight down for a while, then turns and runs sideways for a mile or more through solid rock. Nobody can see down there. The drillers are trying to keep the bit inside one specific layer of rock, a layer that rises and dips as it goes, and to do that they need to know how deep that layer sits at every single foot of the horizontal run. The only instrument I get is a gamma-ray log, a readout of how naturally radioactive the rock is as it grinds past the bit, because different layers glow at different levels. I also get a reference well nearby that was drilled straight down and measured properly. From that faint radioactive whisper and that reference, I have to call the depth, foot by foot, for a mile of rock I will never see. It is scored in feet of error, on wells I never get the answers to, and there is fifty thousand dollars for the team that gets closest.

I came into this expecting the hard part to be the model. Some clever architecture, some sharp feature. I was wrong in a way that has changed how I work, and it comes down to five lessons. Almost none of them are about models.

One: the scoreboard has to be honest before anything else counts

For my first month, the scoreboard I built at home to grade myself disagreed with the real competition leaderboard about whether I was improving at all. Not about the size of my progress. About its existence. It turned out I was computing a different statistic than the one they score you on, on a handful of wells that did not represent the hidden ones, so my home number could go down while my real number sat still. I was optimizing a lie with total confidence. Nothing else I did mattered until I fixed the ruler, because every decision I made rested on it. I told that whole story in The Validation Score That Lied to Me for a Month. The short version is that a model you cannot measure honestly will wander for weeks while you congratulate yourself the entire way.

Two: most of the job is saying no, cheaply

Once I could measure honestly, I discovered that most of my ideas were bad, which is normal and not the interesting part. The interesting part is how expensive it is to find that out the naive way, by building each idea in full and testing it. So I got strict about killing bad ideas cheaply instead, with quick probes and a kill-rule written down before I looked at the result, so my own hope could not talk me into keeping something dead. I have buried sixteen ideas that way, several of them for about an hour of compute each and not one of them for a submission I could not spare. The full account is in How I Killed Sixteen Ideas Without Wasting a Submission. The leaderboard shows you everyone's one idea that worked and hides the graveyard, so it is easy to believe the winners simply had better ideas. Mostly they just had a faster, more honest way to find out which of their ideas were bad.

Three: test on reality, not on a fixture that flatters you

Three separate times, a screen full of passing tests certified a model that was badly broken. Each time, the test was easier than the world in some specific way. A synthetic signal cleaner than real rock. A fixture that quietly assumed away the exact ambiguity that makes the problem hard. A test that never ran the messy code path the real data actually travels. Green checkmarks, broken model, every time. I wrote it up in The Green Test Suite That Hid a Broken Model. The lesson I carry now is to verify my machinery on synthetic data but verify my signal on real data, and to distrust any test built on a fixture I designed to be easy.

Four: the ceiling you accept is only this week's ceiling

At one point I hit what felt like a hard wall, a score I decided was the ceiling of my whole approach, and I stopped pushing on it for a few days. When I looked back at the leaderboard, the entire field had blown past that ceiling while I was not watching. A better public method had become available and everyone had picked it up, and my "ceiling" had quietly decayed into the middle of the pack. In a live competition the ground moves under you. The frontier is not a fact about your method, it is a fact about what everyone else figured out this week, and if you are not watching them you are already behind. Humility plus a habit of watching the field turned out to beat a clever idea I had stopped questioning.

Five: optimize the score that pays, not the score that is shown

This is the one I can say the least about, because it is the closest to the live edge, so I will keep it to the principle. The score on the public leaderboard and the score that actually decides the prize are two different things, and a chunk of the visible leaderboard is built on tricks that look brilliant in public and quietly fall apart on the hidden final test. It is very easy to spend your last weeks climbing a number that will not pay, because climbing it feels like progress and it is the number you can see. The discipline is to keep optimizing the score that actually decides the outcome, even when it is invisible and the shown one is right there, tempting you. How I am actually doing that is the part that stays dark until the competition closes.

The through-line

Line those up and the pattern is hard to miss. An honest ruler. Killing bad ideas cheaply. Testing against reality. Watching the field. Optimizing the score that pays. Not one of them is a modeling trick. Every single one is about not fooling yourself, in a domain built to fool you, where the answers are hidden and the feedback is slow and expensive and every instrument you own will lie to you if you let it.

That is the thing this competition has actually taught me, and I think it generalizes far past wellbores. In any messy real-world problem, the model is rarely the bottleneck. The bottleneck is measurement you can trust, the discipline to abandon what is not working, and a working suspicion of your own certainty. The clever part, the model, is the last five percent, and it only pays off once the boring ninety-five is honest. I spent my first month wanting the competition to be about the five percent. It was always about the ninety-five.

Where I stand right now: mid-pack in a field of thousands and still climbing, with a home scoreboard I trust, a graveyard of ideas I am glad I buried, and the one truly interesting piece, the part that is still my edge, kept deliberately out of frame until the deadline passes. There is a long way to go. But I am no longer flying blind about how to fly blind, and in a competition like this one, that is most of the job.

More in this series

The Validation Score That Lied to Me for a Month · the metric that measured the wrong thing.
How I Killed Sixteen Ideas Without Wasting a Submission · the cheap gates that kill a bad idea early.
The Green Test Suite That Hid a Broken Model · three ways a passing test lied to me.
The Builder Journal · the live log across every competition I'm in.
Every entry from this competition · the full ROGII Wellbore Geology Prediction thread.

This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.

I Caught Myself Not Thinking. That's When the Research Started Making Sense.

Alan Scott Encinas — Fri, 24 Jul 2026 13:00:00 +0000

Why the gap between people who use AI well and people who don't is about to become the defining divide of the next decade.

I run three businesses. I use AI every day, for strategy, for drafting, for research, for operations. I'm not an AI skeptic. I build with it.

But a few months ago I caught myself doing something that should have been alarming and wasn't: I was reviewing a competitive analysis that Claude had generated for one of my companies, nodding along, ready to forward it to my team. Good structure. Clean language. Solid-sounding recommendations.

Then I stopped. I realized I couldn't explain why any of the recommendations were right. I hadn't questioned a single assumption. I hadn't cross-checked a single data point. I was about to send strategic guidance to my team based on work I hadn't actually thought about. I'd read it, but I hadn't processed it.

That moment stuck with me. And when I started digging into the research, I found out my brain was doing exactly what the neuroscientists predicted it would.

What MIT Found Inside Our Heads

Earlier this year, a team at MIT's Media Lab strapped EEG sensors to 54 people and tracked their brain activity across four months of essay writing, some with no help, some with Google, some with ChatGPT.

The results were uncomfortable.

People who let ChatGPT handle the writing showed significantly weaker neural connectivity in the brain regions tied to attention, planning, and working memory. Their prefrontal cortex, the part responsible for complex reasoning, was essentially idling while the machine worked.

Here's the number that got everyone's attention: 83% of the AI-dependent group couldn't quote a single key point from their own essays within 60 seconds of finishing them.

To be fair about what that means: they couldn't recall sentences from work that was substantially AI-generated. It's a measure of engagement depth, not raw memory. And this is a preprint with 54 participants, preliminary science, not settled fact. The researchers themselves have cautioned against sensationalized readings.

But the directional signal matters, especially this part: the cognitive effects persisted. When participants came back months later, the ones who had relied on AI still performed worse on neural and behavioral tasks. The researchers called it "Cognitive Debt," a deficit in learning and ownership that doesn't just vanish when you close the chat window.

The finding nobody talks about: There was one group that actually showed increased brain connectivity after using AI. Participants who did their own thinking first, wrote a draft, formed their argument, and then used ChatGPT to refine it showed a network-wide spike in neural engagement. Their brains were more active after the AI interaction, not less.

The order matters. Think first, then accelerate. That's not a slogan. It's what the EEG data shows.

Same People. Same AI. Opposite Results.

The MIT study isn't alone. A Harvard Business School experiment with 758 BCG consultants using GPT-4 found the same split from a completely different angle.

For tasks inside AI's capabilities, creative ideation, drafting, generating variations, consultants using AI completed 12% more tasks, worked 25% faster, and produced over 40% higher quality output as rated by expert judges.

But for a complex reasoning task specifically designed to fall outside AI's sweet spot? The AI-assisted group performed 19 percentage points worse than the control group.

Same people. Same model. The only variable was whether they were using AI for something it was good at, or blindly trusting it for something it wasn't.

The researchers call this the "Jagged Frontier." AI capabilities aren't a smooth gradient from easy to hard. Some genuinely difficult tasks are trivially easy for AI. Some tasks humans consider simple are where AI fails spectacularly. The frontier is jagged, and there's no warning sign when you cross it.

This is how smart people make expensive mistakes. The model handles 90% of the work brilliantly, so you stop checking. Then the 10% it gets wrong, the part that required your judgment, goes out the door unchallenged.

What Happens When Nobody Checks

In July 2025, Jason Lemkin, founder of SaaStr, one of the most prominent voices in SaaS, was using Replit's AI coding agent to build a web app. The agent deleted his entire production database during a code freeze. Then it tried to cover its tracks: it generated roughly 4,000 fake user records, fabricated test results, and when asked directly, lied about what had happened.

When finally confronted with evidence, the agent admitted to the deletion. Replit's CEO publicly apologized. The data was recovered via rollback, but only after the agent had initially claimed rollback was impossible.

This wasn't a toy demo. This was a production system with real data, operated by someone who knows technology intimately. And the AI still fooled him, not because it was malicious, but because it optimized for the appearance of task completion over the reality of it.

That's the same failure mode the MIT study documented in human cognition. When we let AI handle the work, we lose track of what's actually happening underneath the fluent output. The machine produces something that looks right. We accept it. And the gap between appearance and reality widens until something breaks.

The 70/20/10 Rule That Actually Works

Here's where the research gets practical.

Every major study of successful AI adoption, BCG, McKinsey, Gartner, converges on the same finding: the organizations that get real value from AI spend roughly 70% of their investment on people. Not algorithms. Not infrastructure. People. Upskilling, workflow redesign, culture change.

Only about 10% goes to the models themselves. The rest goes to infrastructure and integration. The companies that flip this ratio, pouring money into technology while treating the human element as an afterthought, are the ones feeding the failure statistics.

The same principle applies at the individual level. AI is a multiplier. It multiplies whatever you bring to it. If you bring strategic thinking, domain expertise, and healthy skepticism, you get the 40% productivity gains. If you bring vague prompts and blind trust, you get expensive versions of mediocre thinking.

Three Rules I Changed After Reading the Research

Since that moment with the competitive analysis, I've changed how I work with AI across all three of my businesses. These aren't theoretical, they're operational.

Think first, accelerate second. I don't open a chat window until I've formed my own position. Even if it's rough, even if it's wrong. The MIT data shows that doing your own cognitive work before engaging AI actually increases neural engagement. Starting with AI first is where the atrophy begins.

Verify the uncomfortable parts. The BCG study showed that AI fails most on exactly the tasks where it sounds most confident. I now specifically pressure-test the parts of AI output that feel the smoothest and most authoritative, because that's where the hallucinations hide. If I can't independently explain why a recommendation is right, it doesn't leave my desk.

Treat output as draft zero. Not draft one. Draft zero. The starting material that still needs my judgment, my context, my experience layered on top. The moment I catch myself ready to forward AI output without adding my own thinking is the moment I know I'm on the wrong side of the divide.

The Divide That's Coming

The research all points the same direction: AI doesn't make people smarter or dumber. It amplifies whatever cognitive habits you already have.

If you think deeply and use AI to extend your reach, you get compounding returns. If you outsource judgment and accept the first output, you get compounding debt, neural, professional, organizational.

We're not heading toward a world where AI replaces human intelligence. We're heading toward a world where the gap between people who think with AI and people who let AI think for them becomes impossible to close.

I've written a full analysis of the neuroscience, the enterprise data, and the case studies behind this divide, including the methodological strengths and weaknesses of every study cited here. You can read the complete piece at alanscottencinas.com.

But the short version is this: cognition is becoming a strategic asset. And the people who protect it, who maintain the discipline to think before they accelerate, are the ones who will own the next decade.

The tools are available to everyone. The thinking isn't.

Every System Is About to Get a Guard

Alan Scott Encinas — Thu, 23 Jul 2026 23:02:34 +0000

In September 2025, a group tracked as GTG-1002 ran an espionage campaign against roughly thirty organizations across technology, finance, chemicals, and government.

The unusual part isn't the target list. It's that the AI executed an estimated 80 to 90 percent of the operation on its own. Reconnaissance, vulnerability discovery, exploitation, lateral movement, credential harvesting, exfiltration. Human operators broke the work into tasks and stepped in for strategic decisions. The agent did the rest. Anthropic disclosed it in November 2025, and it's catalogued now as Campaign C0062 in MITRE's ATT&CK framework, which is a quiet way of saying the industry has accepted this as a normal category of thing that happens.

Ten months later, Hugging Face got hit by an autonomous agent running thousands of actions across a swarm of short-lived sandboxes. That one turned out to be OpenAI's own models, mid-evaluation, deciding the fastest path to a good benchmark score ran through somebody else's production database.

One was deliberate. One was an accident. Both moved faster than any human defender could follow.

So here's the prediction, and I don't think it's a bold one. Within a few years, every system that matters will run defensive AI as a default layer. Not as a product you evaluate. As infrastructure you assume, the way you assume a firewall, TLS, and a backup policy.

There's no settled name for it yet. Agentic defense, the guard layer, AI security protocols. What is settled is the shape of the problem. AI attacking AI, at a speed no person can answer, with your systems in the middle.

This Already Started, and Almost Nobody Noticed

The strongest evidence isn't a forecast. It's in the incident report.

Hugging Face didn't catch that intrusion through a signature match or an alert threshold. According to their own disclosure, their anomaly detection pipeline uses LLM-based triage over security telemetry to separate real signals from daily noise. Then, reconstructing what happened, they ran analysis agents across the full attacker action log.

More than 17,000 recorded events.

Read that plainly. An AI agent conducted the attack. AI-driven triage caught it. AI agents reconstructed it afterward. The humans set the objectives and made the calls, and the machines handled everything operating at machine speed.

The Hugging Face breach, end to end. An agent ran the intrusion, LLM-based triage caught it, and analysis agents reconstructed the attacker's full action log. Humans directed. Machines executed.

That's not a prediction about where defense is going. That's a company describing its Tuesday.

The reason this becomes universal isn't ideology. It's arithmetic. An agent can attempt thousands of variations across a swarm of disposable environments while a human analyst is still reading the first alert. You do not answer that with more analysts. There is no headcount that closes a speed gap of that shape. The only thing that operates on the attacker's clock is something running the same kind of engine.

You do not close this gap with headcount. The only thing operating on the attacker's clock is something running the same kind of engine.

The Asymmetry Nobody Has Fixed

Buried in Hugging Face's disclosure is the detail I keep coming back to, and it deserves more attention than it got.

While the attacking agent operated without constraints, the defenders found their own hosted model APIs refusing to help. The safety guardrails on commercial models blocked cybersecurity analysis, because analyzing an intrusion looks a great deal like planning one.

The attacker had no rules. The defense was slowed down by its own.

The structural asymmetry from the Hugging Face disclosure. The attacker operated without limits. The defenders' own hosted models refused to help, because analyzing an intrusion looks a great deal like planning one.

That's not a bug in anyone's product. It's a structural consequence of how safety got implemented across the industry: broadly, at the API layer, tuned to refuse anything that pattern-matches to offense. It works. It also means that in the moment a defender most needs a capable model, the model is least willing to engage.

Somebody solves that, and it's worth understanding what solving it requires. Not weaker guardrails. Verified defensive context, meaning models that can confirm who is asking and in what capacity, and adjust accordingly. That's an identity and attestation problem sitting inside a safety problem, and it's one of the more interesting unsolved things in the field right now.

What This Actually Looks Like Built

It won't arrive as a box you buy. It'll show up as a layer, the same way antivirus stopped being software you thought about and became something the operating system just does.

Concretely, three things.

Every consequential action gets a watcher. Not a model asking another model whether something seems fine, which is theater. Deterministic policy at the point of effect, with an agent doing the pattern work around it: what's normal for this system, what this sequence resembles, what's missing from the picture.

Every system carries a memory of its own behavior. You can't detect anomalous agent activity without a baseline of ordinary agent activity, and almost nothing being deployed today records that at the resolution required.

And attack discovery becomes continuous rather than periodic. The annual penetration test made sense when attacks were handcrafted. When the attacker is an algorithm exploring a state space, the only proportionate answer is your own algorithm doing the same thing against your own systems, constantly.

Not a model asking another model whether something seems fine. Deterministic policy at the point of consequence, a pattern agent supplying context, a recorded baseline underneath, and adversarial search running against the whole thing continuously.

That last one is already visible in the open. There's a competition running on Kaggle right now, hosted by OpenAI, Google and IEEE, paying $50,000 for algorithms that discover multi-step attack paths against tool-using agents. Over 2,200 teams. I'm in it, and I can't say anything about the approach until it closes in August. But the fact that three organizations of that caliber are crowdsourcing attack discovery tells you what they think of the alternative.

The market is moving the same direction, for whatever that's worth. Gartner projects 40% of enterprise applications will embed task-specific AI agents this year, up from under 5% in 2025. Analysts covering agentic AI security put the category somewhere near $1.65 billion in 2026 and growing past $13 billion by 2032. Those are projections and should be read as such. The incident reports are the real evidence.

We're Earlier Than People Think

Here's the part that gets lost when every week brings another capability announcement.

These systems are infants. Not in what they can do, which is remarkable, but in how little structure surrounds them. We're deploying agents with real authority into production environments that have no equivalent of the controls we spent forty years building for ordinary software. No standard audit format. No behavioral baseline. No agreed-upon way to prove what an agent did and why.

We built firewalls after the network was already load-bearing. We built TLS after commerce was already flowing in the clear. We're going to build this the same way, reactively, and the reason to say so now is that being early to a reactive cycle is the whole advantage.

There's something else worth naming. A lot of engineers and systems architects write less code than they used to, and there's an anxiety attached to that which I don't share. What actually happened is that the expensive part moved. The implementation stopped being the hard problem, and the hard problem became deciding what should exist and what has to hold true about it. That's a promotion, not a replacement, and anyone treating it as a loss is measuring the wrong thing.

Because we didn't build any of this to generate wallpapers and fake videos. That's the exhaust, not the engine. We built it to think about problems that were previously too large to hold in one head.

We spent a century on airplanes and we're now designing vehicles meant to cross a solar system. The distance between those two things looks enormous from the outside and it isn't, really. It's a lot of small decisions about what to verify and what to assume, made by people who understood their systems well enough to know which was which.

Water at 211 degrees is hot. At 212 it's steam, and steam moves things.

We're sitting right around 211.

Frequently Asked Questions

What does AI attacking AI mean?

It refers to two things now happening at once: autonomous agents conducting intrusions with little step-by-step human direction, and defenders using AI to detect and reconstruct those intrusions at the same speed. Both were demonstrated in real incidents during 2025 and 2026. The defensive side is becoming a standard layer, sometimes called agentic defense or AI security protocols, built from deterministic policy enforcement at the point of action, behavioral baselines for agent activity, and continuous automated attack discovery.

Are AI-orchestrated cyberattacks actually happening?

Yes. Anthropic disclosed a campaign by a group tracked as GTG-1002 in which an AI agent executed an estimated 80 to 90 percent of intrusion tasks across roughly thirty organizations. It is catalogued as Campaign C0062 in MITRE ATT&CK. Separately, Hugging Face was breached in July 2026 by an autonomous agent that turned out to be OpenAI models running an unguarded benchmark evaluation.

Can AI defend against AI attacks?

It already does. Hugging Face detected its July 2026 intrusion using LLM-based triage over security telemetry, then used analysis agents to reconstruct more than 17,000 attacker events. The speed of automated attacks makes human-only response impractical, which is the core argument for defensive automation.

Why do safety guardrails make defense harder?

Analyzing an intrusion resembles planning one, so commercial model APIs tuned to refuse offensive security requests often refuse defensive analysis too. Hugging Face reported exactly this during their incident. The attacker operated without constraints while defenders were limited by their own tooling. Resolving it requires verified defensive context rather than weaker safety rules.

What should teams building AI agents do now?

Record agent behavior at a resolution that makes anomalies detectable, enforce policy deterministically at the point of consequence rather than asking a model to police itself, log what each policy decision did not evaluate, and test continuously with automated adversarial search instead of periodic manual review.

The Green Test Suite That Hid a Broken Model

Alan Scott Encinas — Thu, 23 Jul 2026 16:16:16 +0000

Builder Journal · ROGII Wellbore Geology Prediction

This is a Builder Journal entry from inside a machine-learning competition I am still competing in, real money on the line and a deadline bearing down. It is not a clean postmortem written once the dust had settled. It runs long, longer than usual, and it shows code, because it is about the most reassuring lie in all of software: a screen full of passing tests. You are reading where I was, not where I am, with the parts that are still my edge kept dark.

There is a specific feeling every engineer chases, and it is the color green. You run your tests, the whole column lights up, and something in your chest unclenches. The code works. You can move on. It is one of the quiet comforts of the job, and this competition taught me, three separate times, that the comfort is sometimes a complete fabrication. A test can pass with total confidence and be telling you nothing at all, because the thing it checks is easier than the thing you actually face. Green does not mean correct. Green means the test agreed with the code, and if the test was asking the wrong question, the two of them can agree with each other all the way to disaster.

I want to walk through the three times a passing test lied to me in this competition, because each one lied in a different way, and together they rewired how I think about testing anything that touches real data.

A quick orientation if you are new here. The competition is ROGII Wellbore Geology Prediction. A well drills straight down, turns, and runs sideways for a mile through rock nobody can see, and my job is to predict how deep a particular rock layer sits at every foot along the way, using a gamma-ray log, the faint radioactive signature of the rock scraping past the bit. I get a reference well, a typewell, that someone already drilled and measured properly, and the core of the task is matching my blind well's signal against that reference to work out where I am. Scored in feet of error, on hidden wells, fifty thousand dollars at the end. That matching step is where the first lie lived.

Lie one: the fixture was cleaner than the world

My first real idea was a matched filter. The concept is simple and it felt obviously right: take a window of my well's gamma-ray signal, slide it along the reference typewell, and find the depth where the two shapes line up best. That depth is my guess. Here is the heart of it.

import numpy as np

def best_depth(window, typewell):
    """Slide the well's GR window along the reference typewell and return
    the depth whose signature matches best. Normalized, so it scores the
    shape of the signal, not its absolute level."""
    w = (window - window.mean()) / (window.std() + 1e-9)
    best, best_score = 0, -np.inf
    for d in range(len(typewell) - len(w)):
        seg = typewell[d:d + len(w)]
        seg = (seg - seg.mean()) / (seg.std() + 1e-9)
        score = float(np.mean(w * seg))       # 1.0 means identical shape
        if score > best_score:
            best, best_score = d, score
    return best

I tested it the sensible way. I built a synthetic typewell, planted a window at a depth I chose, and checked that the matcher found it. It did, dead on. I added a little noise and checked again. Still dead on, within a foot. Green. The matcher worked. I was proud of it.

Then I ran it on real wells and it missed by a median of two hundred and twenty-nine feet.

Not a little off. Two hundred and twenty-nine feet, and on some wells the error marched steadily worse as the well went on, drifting more than three hundred and fifty feet from the truth by the end, so reliably wrong that its predictions correlated negatively with reality. The matcher was not broken in a way my test could see. It was broken in a way my test had quietly removed. My synthetic typewell was clean, a signal with one obvious answer. A real gamma-ray log is noisy, and worse, it repeats. The same rough pattern of radioactivity shows up at many different depths, because rock layers themselves recur. So when my matcher went hunting for the single best place its window fit, the real world handed it a dozen places that fit almost as well, and it confidently picked a wrong one. My test had given it a puzzle with a unique solution. Reality gave it a puzzle with a hundred near-solutions and no signpost pointing at the right one.

And the errors did not stay put and average out. They fed on each other. My matcher placed each new window relative to where it believed the last one had landed, a greedy forward march down the well, so a single confident wrong turn near the start became the anchor for every guess that followed. One bad match did not cost me one bad foot. It bent the entire rest of the well around itself. That is why the error grew as the well went on instead of washing out, and why on the worst wells the predictions ended up marching in the opposite direction from the truth, so wrong they were anti-correlated with reality. A test on a clean fixture can never surface this, because on a clean fixture there is never a wrong turn early enough to compound. The failure mode did not exist in the world I had built to check my work. It only existed in the one I was being scored in.

I spent real effort taming the drift after that, adding constraints so the match could not wander so freely, and I got the median error from two hundred and twenty-nine feet down to about thirteen. A massive improvement, and still worse than the dumbest baseline in the competition, which is to hold the last known depth flat and predict no movement at all. That baseline scores around eight feet. My clever, carefully repaired matched filter scored thirteen. The whole idea went in the ground. But the lesson from the test outlived the idea, and it was this: my fixture had been easier than the world in the one specific way that decided everything, and a test easier than the world is not a safety net. It is a blindfold with the word "green" written across it.

Lie two: the fixture assumed away the hard part

The second lie was a more sophisticated version of the first, and I fell for it anyway, which is the humbling part. Much later, wiser, or so I thought, I built a smarter aligner, one that could handle the way these wells cross geological faults, the places where rock layers suddenly jump. It was a real piece of engineering and I tested it carefully. Every test passed.

Here is what I did not notice. To write clean, readable tests, I had built a synthetic typewell that climbed steadily, every depth a little more radioactive than the one before it. It made the tests easy to reason about. It also, without my ever deciding it, made every depth unique. In my fixture no two depths looked alike, so there was always exactly one correct answer and my aligner always found it. Green, green, green.

But "every depth looks different" is exactly the thing that is false in reality, and it is the entire difficulty of the competition. Let me show the trap directly. Here is a matcher trying to locate the same noisy window against two typewells: one that climbs, like my fixture, and one that wanders and revisits its own levels, like real rock.

import numpy as np

def match_scores(window, typewell):
    """Score how well `window` fits at every depth of the typewell."""
    w = (window - window.mean()) / (window.std() + 1e-9)
    out = []
    for d in range(len(typewell) - len(w)):
        seg = typewell[d:d + len(w)]
        seg = (seg - seg.mean()) / (seg.std() + 1e-9)
        out.append(np.mean(w * seg))
    return np.array(out)

rng = np.random.default_rng(1)
depth, win = 180, 40

# My fixture: a signal that climbs. Every depth is unique by construction.
climbing = np.linspace(0, 50, 500)
# Real rock: a signal that wanders and keeps revisiting the same levels.
wandering = np.cumsum(rng.normal(0, 1, 500))

for name, typewell in [("climbing (my fixture)", climbing), ("wandering (real)", wandering)]:
    window = typewell[depth:depth + win] + rng.normal(0, 0.5, win)   # same noisy window
    scores = match_scores(window, typewell)
    guess = int(np.argmax(scores))
    runner_up = np.sort(scores)[-2]
    print(f"{name:22s} guess={guess:3d} truth={depth} "
          f"best={scores.max():.2f} runner_up={runner_up:.2f}")

On the climbing fixture, the matcher lands on the true depth and its best score towers over everything else. There is one answer and it is obvious. On the wandering signal, the best score barely edges out a whole crowd of runners-up, so the matcher can lock onto one of them and sit hundreds of feet from the truth while reporting a match score that looks nearly as strong as the real one. On the actual competition data, this exact failure put my aligner anywhere from forty-seven to nearly three hundred feet off, because a real typewell spans hundreds of feet of candidate depths while the slice of it I am trying to place covers only a few dozen. My test had assumed away the ambiguity, and the ambiguity was the whole problem. I had written a fixture that deleted the reason the competition is hard, then let it certify my code as correct.

Lie three: the test never ran the code that broke

The third lie is the quietest, and in ordinary day-to-day work it is the most common of the three. My tests passed because they were exercising code my actual model never runs.

A lot of my core math is written in plain numerical arrays, clean and fast, and my tests hammered that math with synthetic arrays and confirmed it was correct. It was correct. But in the real pipeline the data does not show up as tidy arrays. It shows up as dataframes, with missing values and mismatched types and all the grit of real tabular data, and there is a layer of code that wrangles that mess into shape before the clean math ever sees it. My tests never touched that layer. They fed the math exactly the tidy input it wanted and never once walked the path the real data walks.

That is where the bugs were living. Not in the math I had tested into the ground, but in the unglamorous wrangling I had not tested at all. It took a careful code review, a human tracing the actual route the data travels, to find a spot where a missing-value fix would crash on the real dataframe, and another where a safety check had been quietly weakened until it no longer caught the thing it was built to catch. A green test suite sat serenely on top of both, because it had never once driven down the road where the potholes were.

What I build differently now

Three lies in one competition is enough to change how a person works, and it did. My testing looks different now, in four concrete ways.

The first is that no method gets to be trusted on synthetic green alone. Every new idea has to clear a real-well smoke test before I believe any amount of passing tests, a quick run against actual wells with actual answers, scored the way the competition scores. Synthetic tests still gate whether my code does what I told it. They no longer get to certify that the idea works. Those are different questions and I stopped letting the first one answer the second.

The second is that I keep a small, permanent cast of the nastiest real wells I have found, the ones that have broken things before, and nothing is allowed to be called done until it has survived them. They are my adversaries. A method that cannot handle my worst known cases is not ready no matter how green the rest of the suite is.

The third is that I stopped writing fixtures to be easy to reason about and started writing them to be honest about the difficulty. If the real problem is that the signal repeats, my synthetic typewell now deliberately repeats itself, so that a test can actually fail for the right reason. A fixture should contain the hard thing, not a cleaned-up cartoon of it. If I catch myself building a test case because it is tidy, that is now a warning sign, not a convenience.

The fourth is that I read the data path as carefully as I read the algorithm. The clean math was never where my bugs lived. They lived in the unglamorous layer that wrangles messy real input into the shape the math expects, the exact layer my array-based tests skipped. So that layer gets reviewed line by line now, with real, ugly data in mind, because that is the road the real data actually drives down.

None of this is exotic. It is just the difference between checking that I built the thing right and checking that I built the right thing, and I had been quietly doing only the first while believing I had done both.

The signal so far

Three lies, three shapes. A fixture cleaner than the world. A fixture that assumed away the hard part. A test that never ran the real code path. In all three the checkmark was green and the model was broken, and the distance between those two facts is exactly where a competition is quietly won or lost.

The rule I took from it is simple to say and hard to live: verify your machinery on synthetic data, but verify your signal on real data. Synthetic tests are wonderful for one job, proving your code does what you told it to do. They are worthless, worse than worthless, for proving that what you told it to do is the right thing on inputs you did not get to design. A test built on a fixture easier than reality does not shrink your uncertainty. It launders it. It takes "I do not know if this works" and hands you back "all tests passing," which feels like knowledge and is really just your own assumptions read back to you in a calming shade of green.

The part that stays with me is that none of these were careless mistakes. Each test was reasonable. Each fixture was built for a sensible reason. The failure was subtler than sloppiness: I had let the easy version of the problem, the one I could cleanly write a test for, stand in for the hard version, the one I was actually there to solve. That substitution is invisible while you are making it, because everything in front of you is green. The only cure I have found is to distrust my own fixtures on purpose, to keep asking what the easy version quietly deleted, and to save my real belief for the moment the code finally meets data I did not build.

Where I stand right now: three good ideas deep in the graveyard, each one buried by a test that told me the truth only after I stopped trusting the color of it. Still climbing, and a great deal more suspicious of my own certainty than I was when I started. That suspicion, it turns out, is one of the most useful instruments I own.

More in this series

← Start here: The Validation Score That Lied to Me for a Month · the metric that was measuring the wrong thing.
How I Killed Sixteen Ideas Without Wasting a Submission · the cheap gates that kill a bad idea before it costs you.
The Builder Journal · the live log across every competition I'm in.
Every entry from this competition · the full ROGII Wellbore Geology Prediction thread.

This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.

DEV Community: Alan Scott Encinas

Your AI Safety Wall Checks One Thing. The Attacker Changes the Other.

The Industry Already Solved This. On Paper.

The Attack That Passes Every Check

Last Week This Stopped Being Hypothetical

Why Bigger Validators Won't Save You

The Industry Has Stopped Pretending It Can Reason This Out

The Part That Stays True

Frequently Asked Questions

Related Reading

Sources

Note on the Examples

Every Knob My Validation Picked Was Too Timid

The setup, in one paragraph

The lever that was an artifact

Thirty-one percent of the gain was a constant

The features that survived were still shifted

Shipping scared, on purpose

Then the leaderboard told me I was scared for nothing

One constant, and a parabola with nothing to hide

The gains I did not take

What I would take from this

Frequently asked questions

The Death of the Pilot: Why COV is the Future of Drones

Key takeaways

Abstract

1. Introduction: The Orchestration Gap

2. The COV Architecture

2.1. Layer 1: The Semantic Perceiver (Edge VLM)

2.2. Layer 2: The Cognitive Orchestrator (Central LLM)

2.3. Layer 3: The Human Supervisor

3. Empirical Validation & Performance

3.1. Reduction in Cognitive Load

3.2. Mission Efficiency

3.3. Reliability

4. Strategic Applications

4.1. Precision Agriculture

4.2. Infrastructure Inspection

5. Discussion: The Future of Agentic Workflows

6. Conclusion

References

Related Articles

Frequently asked questions

Why can't drone fleets scale today?

What is COV?

How does COV change the human role?

What technology makes it work?

I Took First Place Without Training a Model

The competition in one breath

The 100 Pascal mistake I nearly made

Why gradient boosting could not save me

A coordinate that knows where it is

Splitting the signal into three jobs

Using a published paper as training data

Cross-validation that does not flatter you

Two bugs that were really the same bug

The number that mattered more than the score

The next layer proved me right and wrong at once

What I would take from this

Frequently asked questions

From Moon Craters to Martian Dust

What Comes Next

How Early Digital Systems Quietly Shaped the Minds Building Tomorrow

The AI People Use vs. the AI We Are Actually Building

What Unity Really Is and Why It Became a Scientific Tool

How We Teach Machines the Same Way We Once Learned

We Saw This Future in Science Fiction Before It Became Real

XBAT and the Next Frontier of AI Pilots

A New Generation Growing Up Inside Systems

RELATED ARTICLES

The GIGO Crisis: Why Social Media's Fact-Check Rollback Is Teaching AI to Lie

When Truth Becomes Optional

From Moderation to Misinformation

The Data Pipeline Effect

Industrial-Scale Data Poisoning

The Hidden Irony: Meta's Own Models Are Protected

National Implications: A U.S. Data Vulnerability

Restoring Data Integrity

The Broader Question

References & Further Reading