DEV Community: Alan Scott Encinas

What Is Cognitive Orchestration? Why AI Systems Can't Be Built Like Traditional Software

Alan Scott Encinas — Sat, 11 Jul 2026 22:06:27 +0000

Coordinating probabilistic cognition into reliable systems.

The first time I wired a handful of models together, it worked. The second time, it drifted. By the tenth, I couldn't tell you where the system had gone wrong, only that it had, somewhere between step three and step seven, quietly and with complete confidence.

What I eventually understood is that the model was almost never the thing that failed. The models were capable enough. What kept breaking was everything around them: the way they were connected, the assumptions living between the steps, the absence of anything watching the whole. Most AI systems today are not failing because foundation models lack capability. They are failing because we keep wrapping probabilistic cognition in architectures designed for deterministic machines.

For decades, software engineering has been based on predictability. Engineers define workflows, map dependencies, and write code that moves information through a system according to explicit rules. When an input changes, conditional logic handles the variation. When a component fails, the system returns a known error state. When execution succeeds, the output follows an expected structure.

In this traditional model, orchestration means coordinating reliable components. It involves scheduling processes, routing information, managing dependencies, and maintaining the infrastructure that allows different parts of a system to operate together. The underlying assumption is that individual components behave predictably and that failures can be anticipated through explicit rules.

Foundation models changed the nature of the components being orchestrated. Large language models and other AI systems do not behave like traditional software services. They do not simply receive an input and return a predetermined output. Instead, they interpret context, generate possibilities, and produce probabilistic results. This allows them to solve problems that were never explicitly programmed, but it also introduces uncertainty through hallucinations, ambiguity, and inconsistent interpretations.

Despite this shift, many AI applications are still built using traditional software patterns. Developers connect models together through prompt chains, passing the output of one probabilistic component directly into the next while expecting the overall system to behave like a deterministic pipeline.

This creates an architectural mismatch. Reliable AI systems cannot be created simply by connecting models together and hoping the chain remains stable. They require an additional layer that can maintain context, evaluate outcomes, manage uncertainty, and adapt execution as conditions change.

That layer is cognitive orchestration.

What Is Cognitive Orchestration?

Cognitive orchestration is the architectural discipline of coordinating probabilistic computing systems through synthesized state, adaptive planning, verification, and persistent memory to produce reliable, goal-directed behavior under uncertainty.

Traditional orchestration focuses primarily on coordinating execution. Cognitive orchestration extends this idea by coordinating the processes that allow a system to understand a situation, determine an appropriate action, evaluate the result, and adapt its strategy over time.

The purpose of cognitive orchestration is not to remove uncertainty. Probabilistic systems will always contain uncertainty. The purpose is to create the architecture required for those systems to operate reliably in real-world environments where information is incomplete, objectives may be ambiguous, and conditions can change during execution.

Why Does Cognitive Orchestration Exist?

The challenge with modern AI is no longer simply generating an answer. Foundation models are already capable of producing impressive outputs. The larger challenge is making those outputs reliable enough to support complex systems, businesses, and real-world decision-making.

A single model call can be useful, but production systems require more than isolated responses. They require systems that can maintain awareness of a larger objective, determine what information is missing, select the right tools, verify results, recover from mistakes, and continue execution when conditions change.

This is where traditional software architectures begin to struggle. Deterministic systems work extremely well when the rules are known and the possible states can be defined ahead of time. However, many real-world problems involve uncertainty, incomplete information, and changing environments.

Cognitive orchestration provides a framework for building systems that can operate within those conditions.

The Evolution of Orchestration

Orchestration itself is not a new concept. Operating systems orchestrate processes. Kubernetes orchestrates containers. Workflow engines orchestrate business operations. Distributed systems orchestrate services across complex infrastructure.

The need for orchestration has always existed because complex systems require coordination. What has changed is the nature of the components being coordinated.

Traditional software components are primarily deterministic. They execute defined logic and return predictable responses. Foundation models are different. They are probabilistic systems capable of interpretation, generation, and inference.

This difference fundamentally changes the role of orchestration.

Traditional orchestration manages predefined execution paths. Cognitive orchestration manages adaptive decision-making. Instead of only determining where information should move, a cognitive orchestrator evaluates the current state, understands the objective, considers available actions, and determines the next appropriate step based on available information.

The system is no longer simply moving data through a workflow. It is managing the process required to accomplish a goal.

Cognitive Orchestration vs Prompt Chaining

Prompt chaining was one of the earliest approaches for building AI applications. The concept is simple: when a single prompt cannot complete a complex task, divide the task into smaller steps and connect those steps together.

For example, a document processing workflow might use one prompt to summarize information, another prompt to extract entities, and another prompt to transform those entities into structured output.

For narrow and controlled applications, this approach can be effective. The problem appears when prompt chaining becomes the primary architecture for complex systems.

Each step in a probabilistic chain introduces uncertainty. Picture a deep-space probe. A trajectory that is off by a fraction of a degree at launch looks flawless for the first million miles, and by the time it reaches the outer planets it has sailed past its target into empty space. A prompt chain drifts the same way. A slightly incorrect interpretation early in the process influences every downstream operation, and missing context, incorrect assumptions, or hallucinated information become embedded into later stages until the system can no longer recognize that it has moved away from the original objective.

The issue is not that prompt chaining is wrong. It is that chaining alone lacks awareness.

A chain can move information forward, but it cannot determine whether that information remains accurate, complete, or aligned with the original goal.

Cognitive orchestration treats chaining as one execution pattern inside a larger adaptive architecture. The orchestrator maintains awareness of the overall system state, evaluates intermediate results, and determines whether the next action should involve continued execution, correction, additional retrieval, verification, or escalation.

A prompt chain moves forward blind. An orchestrator holds shared state and checks itself.

Cognitive Orchestration vs AI Agents

AI agents are often described as systems capable of planning tasks, using tools, and taking actions toward an objective. However, an agent is only one component within a larger cognitive architecture.

A collection of agents does not automatically create intelligence. Without coordination, shared context, verification mechanisms, and a way to evaluate progress, adding more agents can increase complexity without increasing capability.

Cognitive orchestration provides the architecture that allows agents and other AI components to work together effectively. It manages shared state, determines task allocation, evaluates results, and ensures that individual actions contribute toward a larger objective.

Agents provide specialized capabilities. Cognitive orchestration provides coordination and reliability.

The Architecture of Cognitive Orchestration

A cognitive orchestrator relies on four interconnected capabilities that transform probabilistic components into reliable systems.

Four capabilities working together. Remove one and the reliability it was holding up comes down.

State Synthesis

Traditional applications store information. Cognitive systems maintain understanding.

State synthesis transforms raw data, previous actions, constraints, objectives, and observations into an operational representation of the current situation. This goes beyond storing conversation history or expanding a context window. Think of the difference between a pile of raw sensor logs and the bridge viewscreen on a starship. The logs contain everything that happened; the viewscreen turns it into a single live picture the crew can actually act on. A system needs more than information. It needs an organized understanding of what is known, what remains uncertain, what constraints apply, and what actions are still required.

State represents the system's current understanding of the environment and task.

Adaptive Planning and Decision-Making

Traditional software follows predefined workflows. Cognitive systems determine execution dynamically.

Using synthesized state, the orchestrator evaluates possible actions and selects the strategy most likely to advance the objective. This may involve calling a language model, retrieving information from memory, querying an external system, executing deterministic code, parallelizing tasks, or requesting clarification from a human.

Execution is not completely determined before runtime. The system continuously evaluates new information and adapts its approach. A Mars rover does not blindly follow a line drawn on a map back on Earth. When it meets a boulder that was never in the plan, it finds its own way around and keeps heading for the target. Adaptive planning gives a cognitive system the same freedom.

Planning is not simply routing. It is decision-making under uncertainty.

Verification Loops

Traditional software verifies whether an operation completed successfully. Cognitive systems must also evaluate whether the result is trustworthy.

Because probabilistic models can generate plausible but incorrect information, verification becomes a core requirement. It is the ship's computer re-running the jump calculation before it commits, because a confident answer and a correct answer are not the same thing. A verification loop evaluates whether outputs are supported by evidence, satisfy constraints, remain consistent with existing state, and contain sufficient confidence to continue.

When verification fails, the system can retry with additional context, retrieve more information, decompose the problem, or escalate to human review.

Verification transforms uncertainty from a failure condition into a manageable part of system behavior.

Persistent Memory

Context windows provide temporary awareness. Memory provides continuity.

Persistent memory allows cognitive systems to retain information beyond a single interaction. Without it, a system is the protagonist of Memento, walking into every task with no memory of the last one, doomed to repeat the same mistakes because it cannot remember making them. It enables systems to remember previous outcomes, successful strategies, user preferences, recurring failures, and historical patterns.

State represents the system's current understanding.

Memory represents accumulated experience.

Together, they allow systems to become more informed over time rather than treating every execution as an isolated event.

Real-World Applications of Cognitive Orchestration

Cognitive orchestration becomes increasingly important as AI systems move beyond simple assistance and into complex environments.

In enterprise AI, orchestration enables systems to work across documents, databases, applications, and human teams while maintaining reliability and accountability.

In robotics and autonomous systems, cognitive orchestration allows perception models, planning systems, control systems, and human oversight mechanisms to work together in uncertain environments.

In scientific research, it enables AI systems to combine models, simulations, external knowledge, and human expertise to accelerate discovery.

In multi-agent systems, orchestration provides the coordination layer required to prevent isolated agents from becoming disconnected processes.

In my own work building multi-agent coordination systems, I have found that adding more agents does not automatically increase capability. Without a central orchestration layer maintaining shared state, detecting drift, evaluating progress, and coordinating execution, complexity can grow faster than intelligence.

The Field Caught Up

For a while, saying this out loud felt contrarian. The public story of AI was a story about size: bigger models, longer context windows, higher benchmark scores. If you argued that the real work lived in the architecture around the model, you were talking about the plumbing while everyone else admired the engine.

Then the ground shifted, and the language of the field started to catch up to what the working systems were already doing.

In February 2024, Berkeley's AI research lab said it plainly: state-of-the-art results are "increasingly obtained by compound systems with multiple components, not just monolithic models," and called it one of the most impactful trends in AI that year. They pointed out that most real applications were already compound, with around 60 percent using retrieval and 30 percent using multi-step chains. The bare model was the exception, not the rule.

A month later, Andrew Ng named the four patterns doing the heavy lifting, reflection, tool use, planning, and multi-agent collaboration, and showed that an agentic workflow beats asking a model to produce its answer in a single pass. By December, Anthropic's guide to building agents read, from its first page, as a document about architecture: workflows versus agents, and an orchestrator that breaks work down, delegates it, and synthesizes the results.

The other half of the story arrived at the same time. The scaling narrative hit a wall in public. Ilya Sutskever told a room at NeurIPS that "pre-training as we know it will end," because compute keeps growing but data does not. Reports followed that the newest frontier models were showing smaller gains than the leap before them. Attention started moving from making the model bigger to making the system smarter.

Then came the number that settles it. In June 2025, Anthropic reported that a multi-agent version of its system outperformed the single-agent version by 90.2 percent on an internal research evaluation, using the same model family. The improvement did not come from a better model. It came from the architecture around it.

Anthropic's multi-agent research system vs. its single-agent version, same model family, on their internal research eval (June 2025). The lift is architecture, not a bigger model. *At roughly 15× the token cost, reliability under uncertainty is not free.

Even the vocabulary moved. "Prompt engineering" gave way to "context engineering," which Andrej Karpathy described as the art of filling the model's context with exactly the right information for the next step. The name changed because the job changed. The work was never really about the prompt. It was about the system deciding what the model sees, when, and what to do with what comes back.

None of this is a claim that I called it first. It is that the field's language finally caught up to the architecture the work was already demanding. The model was always the easy part.

Intelligence Emerges From Architecture

Foundation models are powerful engines of probabilistic inference, but they are not complete intelligent systems by themselves.

The intelligence users experience in production emerges from the architecture surrounding the model. State provides understanding. Planning drives action. Verification establishes reliability. Memory provides continuity.

Together, these capabilities transform individual model calls into systems capable of pursuing objectives in uncertain environments.

The future of AI will not be defined solely by who creates the largest models or the longest context windows. It will be defined by who builds the most effective architectures for coordinating intelligence.

Models generate possibilities.

Architectures create reliable cognition.

Key takeaways

Most production AI fails on architecture, not the model: probabilistic components wired together through deterministic pipelines.
Prompt chaining compounds uncertainty. A linear chain moves information forward but has no awareness of whether it is still accurate, complete, or on-objective.
Cognitive orchestration is the discipline of coordinating probabilistic systems through four capabilities working together: synthesized state, adaptive planning, verification loops, and persistent memory.
The field agrees now: Berkeley's shift to "compound AI systems," Anthropic's orchestrator-workers pattern, and a 90.2% jump from multi-agent orchestration over the same single model all point to architecture over raw scale.
The intelligence users experience in production emerges from the architecture around the model, not the model alone. Models generate possibilities; architecture creates reliable cognition.

FAQ

What is cognitive orchestration? Cognitive orchestration is the architectural discipline of coordinating probabilistic cognitive systems, such as large language models, through synthesized state, adaptive planning, verification, and persistent memory to produce reliable, goal-directed behavior under uncertainty. Traditional orchestration coordinates predictable execution; cognitive orchestration coordinates adaptive decision-making, managing the uncertainty a probabilistic model introduces so the overall system stays dependable.

How is cognitive orchestration different from prompt chaining? Prompt chaining passes the output of one model call directly into the next along a fixed sequence. It works for narrow tasks but has no awareness of its own accuracy, so errors compound as the chain grows. Cognitive orchestration treats chaining as one execution pattern inside a larger adaptive architecture that maintains state, evaluates intermediate results, and decides whether the next action should be execution, correction, retrieval, verification, or escalation.

How is cognitive orchestration different from AI agents? An AI agent is a single component that can plan, use tools, and take actions toward an objective. Cognitive orchestration is the architecture that coordinates agents and other AI components so they work together reliably: it manages shared state, allocates tasks, evaluates results, and keeps individual actions aligned with the larger goal. Adding more agents does not create intelligence on its own; without a coordination layer, complexity grows faster than capability. Agents provide specialized capabilities; cognitive orchestration provides coordination and reliability.

What are the four pillars of cognitive orchestration? State synthesis (turning raw information into an operational understanding of the situation, distinct from stored data), adaptive planning (deciding the next best action under uncertainty rather than following a fixed workflow), verification loops (evaluating whether a result is trustworthy, not just complete), and persistent memory (retaining experience across interactions so the system improves rather than treating each task as isolated).

Why can't AI systems be built like traditional software? Traditional software is deterministic: the same input produces the same output, and failures are explicit. Foundation models are probabilistic: they interpret and generate, can vary between runs, and can produce confident but incorrect results. Wiring probabilistic components through a deterministic pipeline lets those errors pass through undetected, which is why reliable AI systems need an orchestration layer designed specifically to manage uncertainty.

The Infrastructure Fallacy: Why Most Sales Problems Are Actually Strategy Problems

Alan Scott Encinas — Sat, 11 Jul 2026 20:20:50 +0000

In the last decade, I have worked with companies ranging from struggling startups to multi-million dollar OEM manufacturers. Despite operating in completely different industries, many of them shared the same underlying pattern. When growth slowed, leadership responded by adding more volume to the system. More advertising, more salespeople, more software, more AI tools, all deployed with the assumption that increased activity would eventually resolve stalled revenue.

It rarely worked that way. What actually happened was closer to pouring water into a leaky bucket, where each new initiative increased motion inside the business but failed to repair the underlying structure that was leaking value in the first place. Over time, this creates what I think of as an illusion of progress. The organization becomes more active, more automated, and more instrumented, yet the core outcome remains unchanged because the system responsible for turning market demand into revenue was never properly designed.

Most companies do not have a sales problem. They have an architecture problem.

The Hidden Signal

Across industries, a consistent signal appears when you look closely at underperforming organizations. They tend to invest heavily in visible infrastructure. Websites get rebuilt, CRM systems get implemented, sales teams get expanded, and increasingly, AI tools are layered into every part of the workflow. From the outside, everything looks like it is moving in the right direction.

Internally, revenue often tells a different story. The issue is not effort or intent, but the absence of a defined system that reliably connects market demand to the sales process. The product exists, the team exists, and the tooling exists, but the path that moves a potential customer from awareness to purchase is either unclear or inconsistent.

That missing path is the signal.

Infrastructure Is Not Strategy

A common misconception is that building infrastructure is equivalent to building a strategy. Call it the infrastructure fallacy. In practice, they are fundamentally different layers of the business.

A website, a CRM, a sales team, or an AI automation stack are all infrastructure components. They enable execution, but they do not determine direction. Strategy is what defines who the customer is, why they would buy, and where they can consistently be reached in a way that creates predictable demand.

Without those answers, infrastructure does not reduce uncertainty. It amplifies it. Each new system increases the surface area of activity without improving the quality of the underlying decisions, which is why many organizations find themselves scaling complexity rather than scaling outcomes.

The Automation Multiplier

AI has introduced a new layer to this problem. Many organizations now treat automation as a substitute for strategy, assuming that enough tooling can compensate for unclear positioning or weak distribution.

In reality, automation does not replace structure. It magnifies it.

Well-designed systems become more efficient and more predictable when automated. Poorly designed systems become harder to diagnose and more expensive to operate. If the qualification process is weak, AI will generate more weak leads. If messaging is misaligned with the market, AI will distribute that misalignment faster. If the sales process depends on intuition instead of defined criteria, automation simply accelerates the inconsistency.

AI is not an alternative to strategy. It is a multiplier of whatever already exists.

Distribution Is the Operating System

Sales is often treated as a standalone function, but in practice it is the output of a larger system. That system can be understood as a sequence that moves demand from the market into revenue.

Customer need flows into discovery, then qualification, then the sales conversation, followed by purchase, retention, and eventually referral. Each stage depends on the quality of the stage before it, which means failure rarely originates where it becomes visible.

Most organizations focus heavily on improving the sales conversation while neglecting everything that happens upstream. They invest in better closers while ignoring lead quality. They automate outreach before validating customer fit. They optimize dashboards that track activity rather than signals that indicate real buying intent.

When the upstream system is weak, downstream performance cannot compensate. Closing is rarely the bottleneck. Distribution is.

Where Good Talent Becomes Bad Data

One consulting engagement made this visible in a very direct way. Leadership believed their sales team had become ineffective because conversion rates had dropped, and the instinct was to replace the closer.

When we traced the pipeline in reverse, a different structure emerged. Marketing was generating large volumes of leads, sales was making a high number of calls, and management was tracking strong activity metrics across the board. On paper, each function appeared to be performing well.

The issue was that the campaigns were attracting people who had limited alignment with the product being sold. Each department was optimizing for a different metric. Marketing focused on lead volume, sales focused on close rate, and leadership focused on activity reporting, but no system was optimizing for customer fit.

Individually, every team looked successful. Collectively, the system was misaligned.

The salesperson had not lost capability. The system had lost signal integrity.

Systems Reveal Problems That Opinions Hide

A consistent pattern emerges when you examine enough organizations. Most failures are not caused by lack of effort. They are caused by the absence of reliable feedback loops.

Without clear feedback, assumptions begin to replace evidence. Meetings replace measurement, and activity begins to look like progress even when it is not producing meaningful outcomes. Over time, each department develops its own internal definition of success, and the organization drifts away from the actual behavior of the market.

At that point, additional effort does not solve the problem. Better observation does.

A Practical Operating Model

When evaluating a business, I return to four questions that cut through most of the surface noise.

Detect: What signal is the market actually producing through customer behavior, purchasing patterns, and engagement data, rather than what leadership assumes is happening.

Validate: Does the available evidence confirm that signal consistently, or is the organization reacting to isolated anecdotes that feel meaningful but do not repeat.

Design: How does demand actually move through the system from discovery to qualification to revenue, and where does friction repeatedly appear in that flow.

Execute: Once the system is understood, it must be deployed, measured, and adjusted continuously based on what the data reveals rather than what the plan originally assumed.

The Real Competitive Advantage

Markets produce signals continuously through how customers search, compare, purchase, ignore, and recommend. These signals are always present, but not always interpreted correctly.

The organizations that outperform over time are rarely those with the most advanced tools or the largest budgets. They are the ones that build systems capable of detecting those signals early and translating them into consistent execution before competitors adjust.

Revenue is not created by websites, CRMs, or AI platforms. It emerges when a clear offer consistently reaches the right customer through a distribution system that reflects how the market actually behaves.

Infrastructure supports execution. Strategy determines where execution should happen.

Confusing the two is one of the most expensive mistakes an organization can make. Technology will continue to evolve and automation will continue to expand, but the underlying system remains constant. Businesses rarely fail because opportunity is missing. They fail because their systems are not designed to recognize it in time.

The signal is almost always present. The difference is whether the architecture is built to see it.

Frequently asked questions

Does my business have a sales problem or a strategy problem?
If you keep adding activity, more ads, more salespeople, more software, and revenue stays flat, it is almost never a sales problem. It is a strategy problem wearing an infrastructure costume. The real question is whether a defined system reliably moves a customer from market demand to purchase. When that path is unclear or inconsistent, no amount of extra effort at the close will fix it.

Can AI automation fix a struggling sales pipeline?
Not on its own. Automation does not replace structure, it magnifies it. A well-designed system gets more efficient when automated, but a weak one gets harder to diagnose and more expensive to run. If qualification is weak, AI generates more weak leads. If messaging is misaligned, AI distributes that misalignment faster. Fix the architecture first, then automate it.

What does "distribution is the operating system" mean?
Sales is the output of a larger sequence, need, discovery, qualification, conversation, purchase, retention, referral, and each stage depends on the one before it. Most companies pour resources into the sales conversation while ignoring everything upstream. Closing is rarely the bottleneck. The way demand reaches the right customer in the first place, distribution, is the system everything else runs on.

Why do good salespeople suddenly start underperforming?
Often they have not lost capability, the system has lost signal integrity. When marketing optimizes for lead volume, sales for close rate, and leadership for activity reporting, but nothing optimizes for customer fit, the pipeline fills with people who were never a match. Each team looks successful in isolation while the system quietly misaligns. Trace the pipeline in reverse before you replace the closer.

I made my agent more capable and it got worse

Alan Scott Encinas — Fri, 10 Jul 2026 13:00:00 +0000

Builder Journal · ARC Prize 2026

There is a moment in every role-playing game where you load your character with so much heavy gear that they can barely walk. Strongest sword in the game, can't reach the fight. I did the machine-learning version of that this month. I kept making my agent more capable, and the scoreboard kept punishing me for it, and it took me two tries to understand that the upgrades were the problem.

A quick frame, in case this is your first entry in this thread: I'm in the ARC Prize 2026, building an agent that has to learn small games it has never seen, with no instructions. As the benchmark's creator measured it, the hardest part by far is the piece that figures out the rules of a game by experimenting on it. So that piece is where I have been pouring my effort.

The obvious upgrade

The obvious way to make that piece better is to teach it more kinds of games. If it can model three families of puzzle today, teach it a fourth, and it should win more. So I did exactly that. I built support for a new class of game it could recognize and solve, wrote it carefully, tested it, and confirmed the thing I wanted to confirm: the agent now beat a game it provably could not beat the day before. Real, verified, new capability. Not a story I was telling myself, a genuine new skill on the board.

Then I submitted, and the score went down.

Twice

This is the part I want to be honest about, because one bad result is noise and two is a pattern. My agent's attempts to use this theory-building component had already been underwhelming on the real board, landing around 0.05, 0.07, and 0.09 across earlier tries, all of them under the 0.25 my plain, careful agent scores when it does not reach for the fancy component at all. The fourth skill was supposed to turn that corner. Instead the next submission came in at 0.04, the worst of the lot. I had added ability and the number had dropped, again.

So I stopped adding and started counting. I ran a survey across twenty-five of the practice games to see what my agent was actually doing on each one. The shape of it was sobering. It cleanly won five. It correctly recognized five more as hopeless and skipped them. And on most of the rest, it poked and prodded at the game, spent real effort trying to build a theory, and then gave up with nothing to show for it.

The reason, when I finally cornered it

The failure was not in the new skill. It was in the cost of reaching for the skill at all. Every time the agent tries to build a theory of a game, it has to spend real moves poking at that game to test its guesses. On a game it can crack, that effort is an investment that pays off in a win. On a game it cannot crack, the effort is pure loss, spent and gone, and most of the hidden games turn out to be ones it cannot crack.

So every new skill I added had the same hidden effect. It made the agent attempt more games, which meant it spent more effort on more games it would ultimately lose anyway, and paid that toll on every single one. I thought I was buying wins. I was buying more expensive losses. Adding the fourth skill is exactly what dragged 0.09 down to 0.04: more reasons to try, more failed attempts, more cost.

What it means

This is the trap of optimizing the wrong variable. I had been quietly counting how many games my agent could solve, treating raw capability as the goal. The competition does not pay for capability. It pays for net result, and a capability that costs more than it returns is a liability wearing the costume of progress.

The lever I thought I was pulling, make it able to do more, was the wrong lever. The right one is closer to the opposite, and that part I am keeping to myself for now, because it is still live and still my edge. What I can hand over freely is the principle underneath it: more powerful is not the same as better. A bigger toolbox is only an asset if you also know when to keep it shut. I spent weeks adding tools, and the breakthrough, when it comes, is going to be a lesson in restraint, not capability.

Where I'm standing right now

0.25, same as ever, and the plain agent still wears the crown over every cleverer version I have built. The difference is that I now understand why all that added capability never moved the number, which is not nothing. Knowing precisely why a thing fails is the toll you pay on the road to the thing that works.

Next entry: the time I spent two full work sessions reasoning from something I believed about my own code that turned out to be flatly untrue.

More in this series

← Start here: I'm trying to teach a machine to play a game it has never seen, what ARC-AGI-3 is, and the bet I'm making.
Why the leaderboard is the only oracle: I looked unbeatable in practice and lost the real fight, why my home bench doesn't get a vote anymore.
The full ARC Prize 2026 thread, every entry from this competition.
The Builder Journal, the live log across all my competitions.

This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.

Building LunarSite: the director's commentary

Alan Scott Encinas — Fri, 10 Jul 2026 13:00:00 +0000

The other half of the story

There are two ways to write up a project like this. One is the results piece: the architecture, the held-out numbers, how it stacks against the benchmark. That version already exists. If you want the metrics table, go read the results write-up.

This is the other one. LunarSite is a solo ML pipeline that screens lunar south-pole landing sites: the kind of pre-mission analysis the IM-2 crash in March 2025 made concrete, when classical geometry aged badly against kilometer-deep shadow and lit peaks ringed by craters.

What I want to talk about is how it got made: the decisions, the forks where I picked one path and parked another, the tooling I built before training anything, the negative result I published so nobody re-runs it. A held-out mIoU tells you what happened; it tells you nothing about the encoder I talked myself out of or the scaffolding that's the only reason a one-person project this wide shipped instead of dying at 70% in a branch somewhere. Director's commentary, not the trailer.

Scoping so it actually ships

The biggest risk to a solo project this broad isn't a hard ML problem. It's scope. Crater detection, terrain segmentation, a geospatial feature pipeline over 315k grid cells, uncertainty quantification, a dark-terrain story, a live demo. Any one of those is a project. Strung together by one person on a laptop, the default outcome is a half-finished mess where nothing ships because everything is 80% done.

So before I trained a model, I wrote a ship contract. Three layers, each with one done-definition:

Layer 1: Foundation. Stage 2 segmentation only: a trained segmenter, a sim-to-real qualitative eval, and a Streamlit v0 demo. Done means the core ML capability works and I can show it to someone. That's it. Not "LunarSite". Just a good lunar terrain segmenter.
Layer 2: Deepening. This is the real ship. Layer 1 plus Stage 1 crater detection, the Stage 3 XGBoost scorer with LOLA features, deep-ensemble uncertainty on Stage 2, and a Streamlit v3 demo that takes coordinates and returns a site score with SHAP. Done means the end-to-end pipeline works. This is the definition of "LunarSite exists."
Layer 3: Validation. Layer 2 plus MC Dropout calibrated uncertainty, PSR-aware Stage 3 features from PGDA, and cross-instrument validation against ShadowCam at Cabeus. Done means the pipeline has been checked against something it didn't train on.

The discipline isn't in writing those three bullets. It's in treating Layer 2 as the ship and refusing to let Layer 3's shiny ideas bleed forward. Calling Layer 2 "the real ship" out loud, in the spec, is a commitment device. When a new idea showed up mid-Layer-1 (and they always do), it had a place to go that wasn't "start it now."

That place was a BACKLOG. Every "ooh, I should also..." went into a file, not the working tree. A full ShadowCam HORUS denoising pipeline. A DINOv2 encoder swap. LuSNAR supplementary training data. Shadow-from-depth validation against LOLA. All real, all interesting, all written down and not done. The backlog is the pressure valve: it lets you acknowledge the idea (which is what your brain actually wants) without paying for it. Several of those items are still in there, filed under "skipped: low credibility-per-effort." That phrase did more work than any model in this project; more on it in the encoder section.

The other rule was: demo at every layer. Streamlit existed from Layer 1, with cached artifacts so the first paint showed a finished overlay and a per-class IoU table instead of an empty upload box. And each version (Stage 2 only, then the real-moon gallery, then crater output, then the coordinate-to-score flow) was independently shareable. The point is psychological as much as technical: a solo project dies the moment you're in a "can't show this yet" position, because that's when momentum becomes a story you tell yourself instead of a thing on a screen.

The harness before the work

Here's the decision I'd defend hardest: I built the Kaggle automation before the training it was for.

Free GPUs are great until you count the clicks. The manual loop is: import the notebook, attach the right datasets, set the accelerator, hit Run All, wait, download outputs. About five minutes of browser clicking per run, error-prone in the worst way: the failure mode is forgetting to attach a dataset, which you discover twenty minutes later when the kernel dies on a missing path. Fine once. Not fine when you're training a 5-seed ensemble, sweeping configs, or re-running an eval for the tenth time after a bug fix.

So scripts/kaggle_run.py wraps the Kaggle CLI and makes every kernel a registered spec. A KernelSpec is a dataclass (slug, title, notebook path, datasets to attach, accelerator, GPU/internet/privacy flags, output dir) and the whole project's kernels live in one KERNELS dict. The metadata you'd otherwise click into the UI is baked into code, version-controlled, and impossible to forget:

"crater_v2_finetune_seed1": KernelSpec(
    slug=f"{KAGGLE_USERNAME}/lunarsite-crater-v2-finetune-seed1",
    title="LunarSite Crater v2 Finetune Seed1",
    notebook=REPO_ROOT / "notebooks" / "finetune_crater_southpole_kaggle.ipynb",
    datasets=[
        f"{KAGGLE_USERNAME}/lunarsite-weights",
        f"{KAGGLE_USERNAME}/lunarsite-southpole-finetune",
    ],
    output_dir=REPO_ROOT / "outputs" / "crater_v2_finetune_seed1",
),

The subcommands are list, push, status, wait, pull, and run. The first five are the obvious primitives; run earns its keep as the full loop: push the notebook, poll until it finishes, pull the outputs to disk.

def cmd_run(name: str) -> None:
    """Full loop: push, wait for completion, pull outputs."""
    cmd_push(name)
    print("Waiting for kernel to complete (polling every 60s)...")
    final = cmd_wait(name)
    if "error" in final.lower():
        print(f"Kernel finished with error: {final}")
        sys.exit(1)
    cmd_pull(name)

cmd_wait polls kaggle kernels status every 60 seconds against a terminal-state set (complete, error, cancelAcknowledged, cancelled) and prints elapsed minutes each tick so you can glance at a terminal instead of babysitting a browser tab. From the command line the whole thing collapses to python scripts/kaggle_run.py run eval_v1_vs_v2.

That is the entire payoff. Training the 5-seed deep ensemble (seeds 2 through 5 are four near-identical entries in KERNELS, two of them generated in a dict comprehension because Kaggle's slug-from-title behavior forced a naming quirk I only wanted to encode once) became four fire-and-forget commands, not an afternoon of clicking the same five buttons twenty times and praying I attached the landscape dataset each time. The registry doubles as documentation: kaggle_run.py list prints every kernel, its notebook, datasets, and output path. Six months later that's the only artifact that remembers how crater_v2_finetune was wired.

The principle, and why it belongs in a making-of and not a results piece: if a manual step will happen more than about three times, the tool to automate it is cheaper than the clicking, and cheaper up front, before you know how many runs it'll take. Building the harness first is a bet that you'll iterate more than you think, and on a research-shaped project you always do.

Fork #1: the encoder I didn't use

The original spec called for trying a DINOv2 encoder as a possible Stage 2 v3. On paper it's the obvious move: a domain-agnostic ViT, self-supervised on a huge corpus, exactly the backbone you reach for when your domain is weird and you want features that don't assume natural-image statistics. Lunar terrain is about as far from ImageNet as it gets. The pitch writes itself.

I didn't do it, and I want to be precise about why, because "I didn't get to it" and "I decided not to" are different statements, and only one is honest here.

The ResNet-34 baseline was already strong, strong enough that the case for the swap was speculative rather than evidenced. I had no signal the encoder was the bottleneck. Sim-to-real transfer was already working: a model trained purely on synthetic Unreal Engine scenes generalized to real moon photography with no domain adaptation, and the class-coverage distribution held up. When the thing you'd replace is already clearing the bar, "a fancier encoder might help" is a hypothesis, not a plan. And a ViT swap is not a cheap experiment: new training recipe, new failure modes, real GPU hours, and a non-trivial chance the answer is "about the same, sometimes worse."

So I weighed it on credibility-per-effort against everything else competing for the same hours, and it lost. The deep ensemble and the MC Dropout calibration delivered better marginal value at lower risk: they make the uncertainty story real instead of making a plot point marginally prettier. DINOv2 went into the backlog under "explicitly skipped," and that's where it still is.

The thing to flag is that this is not a disproof. I'm not claiming DINOv2 wouldn't help. I don't know, because I didn't run it. It's a deferral: speculative upside, real cost, a stronger competitor for the same time. On a solo project with finite hours, "I chose not to spend the budget here" is a defensible engineering decision, and pretending every parked idea was secretly bad is just dishonesty with better PR. It might be worth doing later. It wasn't worth doing instead of calibrated uncertainty.

Fork #2: when the right knobs lose

The most instructive result in the project looks like a footnote on the metrics table: a tiny negative delta. The story behind it is the one I'd want a peer to take away.

For Stage 2, every instinct said the minority classes were the place to push. The classes that matter most for landing safety (small_rocks and large_rocks) are exactly the ones underrepresented in the pixel distribution, dominated by background. Textbook setup for the textbook fixes. So v2 reached for all of them at once: a heavier ResNet-50 encoder for capacity, FocalDiceLoss to down-weight the easy background pixels and focus on the hard minority ones, and inverse-frequency class weighting to rebalance the loss toward the rare classes. Three independently reasonable knobs, all aimed at the same correct-looking target.

It lost. v2 came in at test mIoU 0.8429 with the same flip TTA, 0.0027 below plain v1, which is just a ResNet-34 with vanilla Dice + Cross-Entropy at equal weighting. And it wasn't a one-eval fluke: stacking multi-scale TTA on top of v2, which should have helped if anything, also degraded it.

The lesson is easy to nod along to and hard to internalize: complexity has to earn its place against a measured baseline, every time, and your priors about which knob should help are not evidence. Three changes that each "obviously" target the right failure mode can still, in combination, land below a baseline that did none of them.

The decision I'm most deliberate about is what I did with the loser: I published it. v2 is up on Kaggle as a documented negative ablation, not quietly deleted. The framing I keep coming back to is the simpler thing won, here's the data so nobody else has to redo this. That's useful science, especially when it's undramatic. Someone else screening lunar terrain will have the same instinct, reach for the same three knobs, and now there's a public artifact that says: I tried that, here's the number, spend your afternoon on something else. A results piece reports the winning config; a making-of tells you about the afternoon I spent proving the obvious upgrade was worse, and why I left the evidence out for the next person.

Fork #3: "wrong model" or "wrong data?"

The crater detector is where I almost burned a week on the wrong fix. Stage 1 is a binary U-Net (ResNet-34 backbone) trained on DeepMoon synthetic DEMs; on synthetic validation it was fine. Pointed at real LOLA south-pole tiles, recall fell off a cliff to 0.155, missing roughly 84% of the craters. A screener that finds one crater in six isn't a detector; it's a liability.

The instinct is to reach for the model (bigger backbone, fancier loss) because the architecture is the knob you can turn. I made myself name the question first, because the fix differs completely depending on the answer. Two hypotheses: a modality shift, where synthetic and real LOLA DEMs are different kinds of image and the learned features don't transcribe at all (fine-tuning can't save you: no shared representation to adapt), versus a distribution-within-domain shift, same modality but the real south pole is a harder, shifted slice: degraded rims, overlapping basins, lighting and scale the synthetic generator never sampled (the features are mostly right and the fix is data).

You tell them apart by ruling out the cheap confounder first. I ran a small grid (four preprocessing configs crossed with nine thresholds) to check whether the gap was just a normalization or scale mismatch fixable at inference. It wasn't; no config recovered the recall, but the predictions weren't random either. They fired in the right neighborhoods and dissolved on the degraded rims: the signature of distribution-within-domain, not modality. A model that had never seen this slice would miss diffusely, not fail selectively on the hardest cases. So: don't change the architecture. Show it the slice.

The fix was 334 real LOLA south-pole tiles: 256×256 patches from LDEM_80S_20MPP_ADJ.TIF, resampled to 118 m/px to match the DeepMoon crater scale, labels from the Robbins 2018 catalogue (craters ≥3 km). Fine-tune, don't retrain: warm-started from v1, 25 epochs on the 4070. Recall went from 0.155 to 0.372, a +140% improvement, and v2 is the production Stage 1.

Now the honest part, because that number has a cost. IoU here is brutal: my targets are 1-pixel-wide rasterized rims, so a prediction that's morphologically correct but a few pixels fat scores as mostly wrong, and pushing recall up takes precision down almost mechanically. I took that trade on purpose: for pre-mission screening, a missed crater is a hazard you never knew to avoid, while a thick rim is just a cell you double-check. The point isn't that 0.372 is triumphant. It's that I knew which knob to turn, and why.

Why a deterministic rule still needs a model

Stage 3 scores every cell on the south-pole grid, and the landability criteria are, on paper, a hard rule. NASA's CASSA thresholds are deterministic: slope ≤ 5°, illumination ≥ 33%, Earth visibility ≥ 50%. Three comparisons and a logical AND. So why train a gradient-boosted model on rule-based pseudo-labels? Isn't that an expensive way to reproduce an if statement?

That objection is right about what the rule is and wrong about what it's for. The hard threshold gives you a binary mask, and over 315,034 cells × 29 features that mask is useless as a ranking. Under strict CASSA only 419 cells (~0.13%) come back "suitable," and within those the rule says nothing about which is better: a 4.9° cell and a 0.5° cell are both just "pass," a 5.1° cell is "fail," indistinguishable from a 30° cliff. What I want is an ordering: a soft, rankable score that degrades gracefully across the boundary, so a near-miss on one criterion can be redeemed by excellence on the others. That's what boosting on the pseudo-labels buys: it learns the rule's shape and interpolates the continuous score the rule can't give you.

Why not a neural scorer? The data is tabular: 29 hand-built features, the regime where gradient-boosted trees still beat MLPs at a fraction of the tuning. Interpretability is non-negotiable for mission analysis, and SHAP is the trust layer: it lets me say this cell ranked here because of slope and illumination. The validation backs the design: slope dominates the SHAP ranking, which is what every published study finds, so the model rediscovered the field's known #1 feature on its own.

The dead code that became the uncertainty story

My favorite decision in the project is the one where I wrote almost no new code. A file (src/lunarsite/utils/uncertainty.py) had sat in the tree unused since an early scaffolding pass: real, working MC Dropout helpers (enable_mc_dropout, add_mc_dropout, mc_predict, uncertainty_map_to_rgb) that nothing imported. Dead code. The kind you delete in a cleanup PR.

The interesting bit is add_mc_dropout. segmentation_models_pytorch U-Nets ship without dropout, so flipping eval/train flags gets you nothing, nothing stochastic to enable. This helper walks the module tree and inserts a Dropout2d after every ReLU:

for name, module in model.named_children():
    if isinstance(module, nn.ReLU):
        setattr(model, name, nn.Sequential(
            nn.ReLU(inplace=True),
            nn.Dropout2d(p=p),
        ))
    else:
        add_mc_dropout(module, p)   # recurse

Run that over the ResNet-34 U-Net and it lands 27 Dropout2d modules. At inference, enable_mc_dropout flips just those into train mode while batch norm stays in eval, and mc_predict runs 20 stochastic forward passes, turning their spread into the metrics that matter: predictive entropy for total uncertainty, and mutual information for the epistemic part: does the model know what it's looking at.

The decision was: resurrect this, don't train another ensemble. I already had a 5-seed deep ensemble, and the obvious "more uncertainty" move was a sixth and seventh seed. But ensembles and MC Dropout answer different questions. The ensemble gives me disagreement across independently-initialized models (good aleatoric-leaning coverage), but it's expensive and doesn't, on its own, tell me whether its confidence is calibrated. MC Dropout gives me a cheap per-pixel epistemic map from one fine-tuned checkpoint. The right move was one of each, not five of one.

That meant fine-tuning from best_resnet34.pt (10 epochs, lr 2e-5, p=0.1) on the 4070 to produce models/best_segmenter_mcdropout.pt, then wiring the helpers into the three call sites: streamlit_app.py, scripts/mc_dropout_calibrate.py, and scripts/train_segmenter.py --mc-dropout. Code that imports nothing is a liability; the work was as much in the call sites as the modeling.

And it paid off in the one way uncertainty work has to: it was honest. Expected Calibration Error came in at 0.0072 across 46 million validation pixels: textbook-calibrated, the confidence means what it says. Better, the epistemic signal pointed the right way out of distribution: on the 72 real moon photographs the model never trained on, mean mutual information ran 4.7× higher than on in-domain validation. That's exactly the property a screening tool needs: visibly unsure where it's out of its depth, which on the moon is the shadowed terrain you most need flagged. The best code I shipped in Layer 3 was an import.

Validation as a design choice

By Layer 3 the tempting move is to grind another tenth of an mIoU point. I didn't, because an internal metric improving is the least convincing thing you can show a skeptic: you chose the metric and the split. So I spent that budget on two external yardsticks I didn't get to define. First, overlap with NASA's Artemis III shortlist, a human-curated list from a completely different process: my top 1,000 cells hit 5 of the 9 regions, Mons Mouton dominating. Second, a cross-instrument check against ShadowCam, a separate sensor with separate physics: at Cabeus, 81–85% of its deepest-observed-shadow pixels fall inside my PGDA-predicted PSRs. And the screener excludes them on its own, 0 of the top 100 cells containing permanently-shadowed ground. Agreement with an independent shortlist and an independent instrument buys credibility no internal tuning can, because I couldn't have rigged either.

The cuts, and the judgment behind them

I want to be straight about what I left out, because the cuts are a design decision too. HORUS-style PSR denoising, the LuSNAR supplementary training set, the DINOv2 encoder revisit, shadow-from-depth validation against LOLA, all parked under one phrase: low credibility-per-effort. Each would move a number; none would change the story. HORUS needs the raw 19 GB ShadowCam cubes to validate properly, for a marginally cleaner image; LuSNAR is 108 GB to maybe nudge an mIoU already clearing the bar. A cut isn't an omission when you can name exactly what it would and wouldn't buy you. It's triage. The skill on a solo project isn't doing everything; it's saying why not.

What I'd do differently

If I ran it again: I'd diagnose modality-vs-distribution before training Stage 1, not after a recall of 0.155 forced the question. The synthetic-to-real gap was foreseeable and I paid in a re-train I could have skipped. I'd write the calibration eval before the ensemble, because I built five seeds before I had a way to ask whether any were calibrated. And I'd treat dead code as deferred code from day one; the MC Dropout helpers were the project's best ROI and nearly got swept in a cleanup. The throughline: the leverage on a project this wide isn't the models, it's the order you ask the questions.

If you want the story without the engineering weeds, I wrote the narrative version. For the full numbers, see the results write-up.

The Paintbrush Paradox: Why the Monolithic Era of AI Is Crumbling

Alan Scott Encinas — Thu, 09 Jul 2026 21:00:00 +0000

Over the past week, two narratives have been colliding everywhere I look.

On one side, there's panic. AI is expected to replace marketers, engineers, and entire categories of knowledge work almost overnight. On the other, there are quieter but far more consequential signals: enterprise teams discovering their AI infrastructure is burning through API budgets far faster than expected. This isn't because the underlying models are weak, but because the systems built around them are fundamentally inefficient by design.

These aren't separate stories. They're the same failure showing up in different places.

A conversation with another developer made that gap visible in real time. He argued that auditing a 150,000-line codebase requires feeding the entire repository into a model in one single, massive pass. It's still a common assumption in mainstream tech: that an LLM works like a giant biological brain that you must fully load with raw text before it can begin to think.

But that assumption is already outdated. Modern AI systems don't scale through brute-force context. They scale through structure. And that shift changes everything.

Key takeaways

Bigger context windows did not solve AI. Treating a frontier model as a monolithic processor that re-reads an entire system on every query is wasteful, dilutes attention, and hides bugs under raw volume.

ARC-AGI-3 makes the gap stark: frontier models scored under 1% on interactive reasoning tasks that untrained humans solve at nearly 100%. The gap is architecture, not memory.

The teams pulling ahead treat the model as one narrow component inside a larger system: intelligent routing, task decomposition, retrieval, and only the minimum necessary context.

The next advantage is not the biggest model or the longest prompt. It is the system designed around the model. Prompting was the first generation; systems architecture is the next.

The Myth of the Infinite Context Window

When context windows expanded into the hundreds of thousands or even millions of tokens, many assumed the core limitations of AI had been solved. The prevailing logic became: if a model can "see everything," then it can reason about anything.

In practice, that approach collapses under its own weight. The problem is not intelligence; it's the physics of attention and compute efficiency. Treating a frontier model like a monolithic processor that must re-read an entire system for every query is computationally wasteful, economically unsustainable, and structurally naive. It introduces massive token waste, causes severe attention dilution, and hides execution bugs under raw volume.

True intelligence has never been about how much information you can hold in a single pass. It's about how little you need to solve a problem correctly.

This is where production-grade systems are breaking away from the hype cycle. The newly launched ARC-AGI-3 benchmark makes this boundary explicit. Moving completely away from static pattern-matching tests, ARC-AGI-3 drops AI systems into hidden, interactive environments where they must actively explore, infer rules on the fly, and build a world model under strict compute constraints. Larger context windows alone cannot solve these tasks. As a result, frontier models scored under 1% at launch, while ordinary humans natively solved them.

The human baseline is efficient not because we possess an infinite working memory, but because we excel at hyper-compression, abstraction, and maintaining long-term structured schemas. The gap in AI performance right now is not a lack of memory. It is a lack of architecture.

Bigger Canvas, Worse Paintings

There is a useful analogy here. Giving an artist a larger canvas does not inherently make them better; it simply gives an amateur more space to make mistakes.

AI systems have reached that exact threshold. Expanding raw context windows without radically restructuring how information is dynamically retrieved, filtered, and applied does not improve outcomes. It merely increases cost, noise, and the failure surface area. The real constraint is no longer access to information. It is how intelligently that information is staged.

However, the metaphor is evolving. The future isn't just about a better artist guiding a static brush. The hand (system architecture) and the brush (the frontier model) are co-evolving. Frontier models are rapidly improving their native reasoning depth, execution loops, and instruction-following capabilities. The true differentiator is no longer just the single tool; it is the design of the entire production studio built around it.

The Rise of Cognitive Orchestration

The teams building highly effective enterprise systems today are no longer treating models as the final product. They are treating them as powerful, narrow components inside a larger, smarter scaffold.

Instead of feeding an entire codebase into a single prompt, advanced architectures break complex codebases into structured, multi-scale, and hierarchical representations. Abstract syntax trees become navigable graphs. Repositories become searchable, segmented semantic spaces. Tasks are decomposed into highly targeted, isolated queries.

Orchestration layers manage this workflow, maintaining multiple levels of abstraction simultaneously. The system can evaluate a high-level architectural map while executing on-demand deep dives into specific sub-routines. This orchestration relies on a robust engine: persistent memory systems, dynamic retrieval pipelines, verification loops, self-critique structures, and specialized routing mechanisms.

The goal is no longer maximum input. It is the minimum necessary context.

A disciplined agentic workflow built to audit a codebase does not "read everything." It navigates it surgically. It isolates dependencies, tracks external state, and deploys deterministic instruments for tasks where models are naturally weak (like exact computation or maintaining long-term consistency). It cuts token waste down to a fraction, not by writing better prompts, but by entirely redesigning the flow of information itself.

Where the Line Is Drawn

There is a stark separation forming between two classes of builders.

The first group still treats AI as a universal input-output machine. They rely entirely on raw scale, prompt length, and ever-growing commercial context windows to compensate for a lack of internal system structure. They are the ones currently generating massive technical debt, facing brittle agent behavior, and watching their budgets evaporate.

The second group designs systems around the model. They assume the model is highly capable but fundamentally limited, and they compensate with rigorous engineering: intelligent routing, task decomposition, explicit planning, and human-in-the-loop escalation paths. They leverage clever routing layers to seamlessly pass tasks between cheap, ultra-fast local LLMs for routine data parsing, and expensive, heavy frontier models only when genuine reasoning depth is required.

This difference is not cosmetic. It directly dictates cost, reliability, and scalability. It determines who can operate at true enterprise scale and who gets priced out of production.

The Next Generation of AI

As frontier models continue to commoditize, the ultimate bottleneck has officially shifted away from raw model intelligence and toward deliberate system design.

This is the direction the industry is moving: prioritizing truth-seeking, curiosity-driven architectures that maximize clean signal and eliminate compute waste. Compound scaling, the intersection of smarter models working inside even smarter architectures, is where the real value is being captured.

The first generation of AI rewarded the people who learned how to prompt. The next generation will belong exclusively to the systems architects who know how to design the cognitive machine.

Frequently asked questions

Why aren't bigger context windows enough to make AI smarter?
Because the bottleneck was never how much a model can hold, it is how efficiently the system uses it. Forcing a frontier model to re-read an entire codebase on every query wastes tokens, dilutes attention across irrelevant material, and hides bugs under raw volume. ARC-AGI-3 makes the gap explicit: frontier models scored under 1% on interactive reasoning tasks that untrained humans solve at nearly 100%. The limit is architecture, not memory.

What is AI orchestration?
Also called cognitive orchestration, it is the practice of treating a model as one narrow component inside a larger system rather than the whole product. Orchestration layers decide what the model sees and when: breaking a codebase into navigable graphs and semantic spaces, decomposing tasks into targeted queries, routing cheap models to routine work and expensive ones only to real reasoning, and retrieving the minimum necessary context instead of loading everything.

Does a larger context window increase cost and errors?
Often, yes. Expanding the window without restructuring how information is retrieved and filtered does not improve outcomes; it increases cost, noise, and failure surface area. More room to load is more room to make expensive mistakes. The real constraint is no longer access to information, it is how intelligently that information is staged.

What skills will matter most in the next generation of AI?
System design. As frontier models commoditize, the advantage shifts from raw model intelligence to deliberate architecture: routing, task decomposition, retrieval, verification, and human-in-the-loop escalation. The first generation rewarded people who learned to prompt; the next rewards the systems architects who design the machine around the model.

→ The Scale Wall: Why AI Gurus Are Building Toys While the World Needs Architects

→ Companies Replaced Their Workers With AI. Now They're Hiring Them Back to Babysit It.

I looked unbeatable in practice and lost the real fight

Alan Scott Encinas — Wed, 08 Jul 2026 13:00:00 +0000

Builder Journal · ARC Prize 2026

Every boxing movie has the sparring montage. The hero looks untouchable, gloves snapping, trainer grinning in the corner. Then the bell rings on the real fight and none of it holds, because sparring was never the fight. This month I built an AI agent that was crushing it in sparring. The real fight went differently, and the gap between the two finally taught me how this competition actually works.

This is a thread in my builder's log from inside the ARC Prize 2026, where my agent has to win small games it has never seen before, with no instructions, and the games it actually gets graded on are hidden from me. An earlier entry laid out what the competition is and why it might be the most important benchmark in AI. This one is the day I learned to stop trusting my own practice scores.

The gym every competitor has to build

To improve the agent I had to measure it, and I cannot measure it on the real test, because the real games are secret. There is no way around that wall. So, like everyone in this competition, I built a practice gym at home: the handful of example games the organizers do hand out, a scoring script, and a way to run two versions of the agent head to head and see which did better.

That gym is supposed to be the edge. It is where you try ten ideas cheaply, keep the one that wins, and only then spend one of your scarce real submissions. The leaderboard gives you only a trickle of attempts, so the whole game is supposed to be won at home, on the bench, long before you ever submit. Get the gym right and you compound. Get it wrong and you are sprinting on a treadmill.

The number that looked like a win

One change tested beautifully. On my home bench it gained about five hundredths of a point, and in a competition scored from zero to one, where the leaders are separated by thin margins, that is a real, ship-it-with-confidence number. Every signal pointed the same way. So I sent it.

It came back worse. Not by a hair. The change that gained roughly 0.05 at home cost me around 0.14 on the real, hidden board. Nearly three times the size of the supposed win, pointing the opposite direction. My practice gym had not merely failed to predict the game. It had confidently marched me the wrong way down the road and told me I was winning the whole time.

The part that actually scared me

A bad submission is survivable. You lose a slot, you learn something, you move on. What unsettled me was what happened when I tried to understand the loss. I had a known regression sitting right there on the official scoreboard. A fact. Undeniable, in writing, public. And my own tooling, run against every practice game I had, could not reproduce it. I went hunting for the failure on my bench and the bench kept calmly insisting nothing was wrong.

That is worse than a wrong number. That is an instrument that is structurally blind to the exact thing that decides your score. Digging further, I found that my home proxy had at one stage been overstating performance by something like twenty times. I had been steering by a gauge that was not just miscalibrated but pointed at the wrong quantity entirely, and I had trusted it because it was the only gauge I had.

What it actually means

This rewired how I work here, and the new rule is blunt: the leaderboard is the only oracle. It is the single place the real, hidden behavior ever shows its face, and it rations submissions to a precious few, so each one has to be a sharp question with a real answer attached.

My home gym does not get a vote on whether something works anymore. I demoted it. Its job now is smaller and more honest. It invents candidates worth trying. It guarantees, before I risk a submission, that a change physically cannot make things worse than the version already on the board. And after a score lands, it helps me explain why. What it is no longer allowed to do is pretend to be the truth, because the one time I let it, it lied to my face, twice, with total composure.

There is a cousin to this lesson over in my other competition's journal, where a test set I had carefully built told me a confident lie for almost the opposite reason. Same disease, different host. A measurement you cannot fully trust is more dangerous than no measurement at all. No measurement keeps you humble. A lying one makes you bold, right before it walks you off a cliff.

Where I'm standing right now

Still 0.25 on the board. That number has become the wall I keep failing to climb, because the clever upgrades keep losing to it. The floor is stubborn precisely because it is honest: it is the version of my agent that does the safe, simple thing and never overreaches, and so far nothing fancier has reliably beaten it where it counts. Climbing past it for real, on games I will never get to see, is the entire problem, and I now know for certain it cannot be faked from the bench.

Next entry: the change I was certain made my agent smarter, that made it measurably dumber, and the uncomfortable reason why.

More in this series

← Start here: I'm trying to teach a machine to play a game it has never seen, what ARC-AGI-3 is, and why it's AI's uncomfortable mirror.
The cousin lesson from my other competition: My own test set lied to me, same disease, different host.
The full ARC Prize 2026 thread, every entry from this competition.
The Builder Journal, the live log across all my competitions.

This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.

I spent a few months teaching a laptop to read the Moon

Alan Scott Encinas — Wed, 08 Jul 2026 13:00:00 +0000

In March 2025, a lander called IM-2 reached the lunar south pole and tipped over.

It had survived launch, the long coast, the descent burn. And then, in the last moments, on terrain no spacecraft had ever touched, it came down wrong and couldn't get back up.

I keep thinking about why that place is so unforgiving. At the lunar south pole the sun never climbs overhead the way it does in the photos you grew up with. It skims the horizon at a knife's-edge angle, all year, so even small rocks throw shadows that run for kilometers, and whole crater floors have sat in darkness for a billion years. A flat patch and a deadly drop can look almost identical when the only light you have rakes across them sideways. It is the strangest lighting any human machine has ever tried to land in.

And it is exactly where everyone wants to go. The permanently shadowed places at the pole are cold enough to have trapped water ice: fuel and air and water for whatever comes next. It is where Artemis III is headed to put people back on the surface. The most valuable real estate in the solar system, and it happens to be the hardest ground we have ever tried to read.

That gap between most valuable and hardest to read is the thing I couldn't put down.

The question I couldn't drop

The flight computers that pick landing spots in real time work by geometry. They measure slopes and edges and decide, in seconds, whether the ground below is safe. That logic is fast and proven, and it was built for places where the light behaves. At the south pole the light does not, and the geometry starts seeing cliffs where there are none and smooth ground where there is a hole.

So I started wondering about a different job entirely. Not the split-second decision during descent. Leave that to the engineers who do it for a living. The job before that. Could a machine look at the whole south pole ahead of time, the way a scout reads a map, and quietly rule out the obviously lethal ground so humans only argue about the handful of places that might actually work?

Nobody handed me this. No client, no deadline, no team. Just a blank page, a single laptop, and the free GPUs Kaggle hands out to anyone willing to wait in line. I wanted to find out whether one person could build the whole thing end to end, and whether the result would be worth anything or just confident nonsense.

A video-game Moon taught it about the real one

The first real surprise is also the one I'm still a little smug about.

The first job was to teach a model to look at a picture of the lunar surface and label every single pixel: this is rock, this is ground, this is sky. (The jargon for that is "segmentation"; think of it as very patient, very literal coloring-in.) To learn that, a model needs thousands of example pictures a human has already colored in. Real, labeled photos of the Moon's surface barely exist in that quantity. So the training pictures I used weren't real at all. They were rendered in a video-game engine, the same kind of software that makes mountains in a shooter. Thousands of fake moonscapes, each with a perfect answer key baked in because the computer drew it.

The obvious worry is that a model raised entirely on a video game learns the video game, not the Moon: that it would ace the fake images and fall apart the instant it saw a real photograph from an actual mission.

It didn't. I fed it real moon photos it had never seen, with no special coaxing, no retraining, nothing. And it just... worked. It found the rocks. It traced the ground. A stranger looking at the results can't tell which pictures are the fake training kind and which are real. Something built inside a game engine had learned something true about a real place a quarter-million miles away. I sat there grinning.

It isn't flawless, and the flaw is my favorite part. Every fake training image had a bright, smooth, uniform sky at the top, so the model quietly invented a rule of thumb: bright and smooth means sky. Usually right. But every so often a big sunlit boulder on the real Moon is also bright and smooth, and the model, very confidently, labels the rock as sky. It's wrong in the most human way possible: it learned a shortcut that usually works, and gets caught by the exception. I find that charming.

The part that nearly broke me

Then I built the crater detector, and it humbled me completely.

This piece does a different job. Instead of color photos, it reads elevation maps (the shape of the ground itself) and tries to find craters, because a rim or a pit is exactly what you don't want under a lander. Same plan as before: train it on synthetic data, then point it at the real Moon and watch it shine.

It did not shine. The first time it looked at real lunar elevation maps, it missed the large majority of the craters that were actually there. Not a few. Most of them. It was, plainly, nearly useless: staring at a landscape full of craters and shrugging.

I want to be honest about how that felt, because the tidy version of this story skips it. The first model had worked on the first try and made me feel clever. This one made me feel like I'd gotten lucky and was now being shown the truth. I spent real days in that hole, second-guessing the whole approach, wondering if the project was the kind of thing that demos well and means nothing. The fun part of building something (and I do mean fun, eventually) is the moment the thing that obviously should have worked just... doesn't, and you have to figure out why.

The why, it turned out, was subtle. The synthetic craters and the real ones weren't different in kind. They were the same sort of thing, just shaped a little differently than the fakes had taught it to expect. So the fix wasn't to throw it all out. It was to show the model a small dose of reality: a few hundred patches of genuine lunar terrain, real craters with real labels, and let it adjust. (In the field this is called "fine-tuning": you don't start a new student over, you let an experienced one apprentice on the real job for a bit.)

That small dose changed everything. The number of real craters it could find jumped by more than double (a 140% improvement) from a model that missed most of them to one that pulls its weight. It's still not perfect; crater-finding on raw elevation data is hard, and I won't pretend otherwise. But it went from the thing that nearly made me quit to a piece I trust. That recovery is the part of the project I'm proudest of, precisely because the low was so low.

Ranking 315,000 patches of Moon

With the two "looking" models working, the last stage was the one the project was secretly for: actually picking places.

I carved the entire south pole into a grid (over three hundred thousand little square cells, each about a kilometer across) and for every one gathered everything I could measure. How steep is it? How much sunlight does it ever get? Can you see Earth from there, for talking to home? Is it crawling with rocks and craters, from the two models I'd just built? Then I trained a final model to weigh all of that into a single score: how landable is this patch of Moon?

That gives you a ranked list of the whole pole, best to worst. Satisfying. But it raises the obvious, slightly terrifying question: is the ranking any good, or did I just build an elaborate opinion machine?

Here's the gut-check that mattered to me more than any other moment. NASA, through years of expert work, has publicly shortlisted nine candidate regions for Artemis III. I had never told my model about that list. It had never seen it, never been graded against it. So I asked where its own top picks landed. Five of NASA's nine regions showed up among them. The model, working alone from raw measurements on my laptop, had independently pointed at the same ground a room full of NASA scientists pointed at. I don't think I exhaled for a few seconds.

Teaching it to say "I don't know"

There's one more piece I care about, and it's a quiet one.

A model that's always confident is dangerous, because it's just as confident when wrong as when right. So I added a way for the model to notice when it's looking at something truly unfamiliar (terrain weirder than anything it trained on) and to raise its hand and say, in effect, I'm not sure about this one. It's the difference between an expert who knows the edges of their own knowledge and a know-it-all who bluffs. When the eventual stakes are a real spacecraft and a real crew, I'd take the one who admits doubt every time. Building the humility in felt as important as building the skill.

Knowing when to stop

There's always another feature. Another model to try, another dataset to chase, another clever idea at 1 a.m. that promises to make everything better. At some point the honest move is to look at a thing that works end to end and call it finished instead of fiddling forever. So I did. The pipeline runs, the parts agree with each other and with NASA, and I'd rather ship something real than polish forever. Stopping was its own small skill to learn.

See it for yourself

If you'd like to poke at it, the whole thing is live and the code is open. Hand it a stretch of the Moon and watch it work: find the rocks, flag the craters, score the ground, and tell you when it isn't sure.

Try the live demo: lunarsite.streamlit.app
Read or fork the code: github.com/AlanSEncinas/LunarSite

If you want the engineering decisions behind all this (the dead ends, the tooling, the forks I argued with myself about), I wrote a separate making-of: the director's commentary.

I'm trying to teach a machine to play a game it has never seen

Alan Scott Encinas — Mon, 06 Jul 2026 13:00:00 +0000

Builder Journal · ARC Prize 2026

Hand a seven-year-old a video game they have never seen. No manual, no tutorial. Within a few minutes they have worked out which button does what, what kills them, and what they are supposed to be chasing. Nobody explained the rules. They learned them by poking at the world and watching what happened.

That casual, almost invisible act is one of the hardest unsolved problems in artificial intelligence. I'm competing to build a machine that can do it.

This is a new thread in my builder's log, and a second competition running alongside the hyperspectral one I have been writing about. Completely different game. Same deal: you get it as it happens, from behind a delay, with the parts that are still my edge kept dark.

What this actually is

The competition is the ARC Prize 2026, built on a benchmark called ARC-AGI. To understand why it matters, you have to know what it was built to embarrass.

ARC stands for the Abstraction and Reasoning Corpus, and it came from François Chollet, the researcher who also built Keras, one of the tools that put deep learning in everyone's hands. In 2019 he wrote a paper called On the Measure of Intelligence with an argument that has aged into a quiet prophecy. Most of what we call AI progress, he said, is not intelligence at all. It is skill. A model that has seen ten million chess games is not smart at chess, it is well-rehearsed. Real intelligence is not how much you know. It is how efficiently you handle something you were never prepared for.

So he designed a test that you cannot rehearse for. ARC is a set of little puzzles where every single one follows a rule you have never seen, and you get only a handful of examples before you have to generalize. You cannot memorize your way through it, because nothing repeats. The only thing that helps is the raw ability to look at a few examples of a brand new pattern and grasp what is going on.

Why it's the field's uncomfortable mirror

Here is the part that keeps ARC in the headlines. Humans, including children, solve these puzzles comfortably. The most powerful AI systems on earth, for years, scored close to nothing.

That gap is the whole point. The same models that pass the bar exam, write working code, and draft a passable essay would sit down in front of ARC and fall apart, because passing the bar exam is a memory-and-pattern feat and ARC is not. It is the benchmark that refuses to be impressed by scale. Every time the models get bigger and quietly start to crack one version, the benchmark gets sharpened into a harder one, and the gap reopens. ARC is the mirror the field keeps walking past, the standing evidence that "knows almost everything" and "can actually think" are not the same sentence.

The ARC Prize exists to do something about that. It is an open competition, with real money behind it, deliberately built to drag the field's attention away from "make the model bigger" and toward "make the model adapt." It rewards efficiency and generalization, the things scale alone does not buy.

The interactive leap

Earlier versions of ARC were static. Look at a few examples, produce one answer, done. ARC-AGI-3, the one I'm competing in, makes it interactive, and that changes everything.

Now the AI is not staring at a snapshot. It is dropped inside a small game it has never encountered and has to win. It is not told the goal, the controls, or the rules. It has to take an action, watch what the world does in response, form a guess, test the guess, and slowly build a working theory of a place nobody described to it. It is the difference between reasoning about a photograph and figuring out a machine by pressing its buttons. That is far closer to how a real mind learns, by doing, by consequence, and it is far harder for a computer.

And you still cannot study for it. The games it gets graded on are hidden. There is no dataset to scrape, no test to pretrain on. The only thing that helps is learning on the fly, in a world you have never seen, from a standing start.

Why anyone should care

Strip away the contest and here is the stakes. Today's AI is dazzling inside the conditions it was trained on and brittle the moment reality steps outside them. It is a brilliant student who has memorized every past exam and panics at a question phrased a new way.

The skill ARC measures, getting your footing in a situation nobody prepared you for, is the exact thing standing between the narrow tools we have now and machines that can handle the truly unfamiliar. A robot in a kitchen it has never seen. A system meeting a problem that was in no textbook. A scientific assistant facing a result nobody predicted. All of that lives or dies on the ability to learn an unfamiliar world quickly, which is precisely what a frontier model still cannot reliably do and a child does before breakfast. That is why a benchmark about cartoon grid-games is, underneath, one of the most serious questions in the field.

The bet I'm making

Here is where I am stubborn, and where a lot of the field would disagree. The fashionable move is to point an enormous language model at the problem and let it sort everything out. I am doing close to the opposite. I am hand-building the mind, the perception, the exploration, the decision-making, all of it as my own code that I can open up and explain, and I am letting a language model do exactly one job: the theory-building part, the place Chollet's own breakdown says most of the difficulty actually lives.

The reason is not nostalgia. I want to be able to explain and defend every single thing my agent does, and "the big model decided" is not an explanation. Learning first, score second. If a shortcut would buy me a better rank at the cost of understanding why it works, it is not a shortcut I am willing to take.

Where I'm standing right now

My best agent wins about a quarter of the hidden games. The number on the board is 0.25.

That is further than it sounds and nowhere near where I want to be. Most of what I have learned here came not from the runs that worked but from the ones that embarrassed me, and this competition has handed me some of the most instructive failures I have had on any project. Those are the entries coming next: the times I was certain I was improving the agent, and the scoreboard told me, flatly, that I had made it worse.

There is a particular kind of humbling that only happens when a number you cannot argue with disagrees with a belief you were sure of. I have a few of those to tell.

More in this series

Why my 21 AI agents aren't allowed to talk to each other, the same bet in another project: hand-build the system instead of pointing one big model at it.
My other live competition: the Hyperspectral Object Tracking Challenge, where this builder's log started.
The full ARC Prize 2026 thread, every entry from this competition.
The Builder Journal, the live log across all my competitions.

This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.

I Built an AI Sales Team for Under $1 a Day. It Closed $29,201 and Beat Salesforce and HubSpot's AI

Alan Scott Encinas — Mon, 06 Jul 2026 13:00:00 +0000

The first version of Albert cost about $15 to run, and my reaction wasn't pride but suspicion. This is too expensive, I thought, and went hunting for what I could cut. That instinct is the whole story, so hold onto it.

I run a B2B sales operation across 14 international markets and 5 languages, by myself. The standard way to handle that is to rent a stack of software, one tool for each slice of the job: a CRM at $150 a month, a lead-generation tool at $120, an email assistant at another $150, and a meeting-prep bot at $40. Add it up and you are paying $460 a month, every month, for work that in my case exactly one person does, and you are renting four products from four companies to do it. When the month ends you own none of it, only the bill. So I built my own instead, named it Albert, and it has cost $15.85 to run across its entire life. That isn't a cost comparison, it's a different category of tool.

Albert isn't a chatbot or a single clever script. It's a workforce of a dozen or so small, specialized agents wired into the CRM, inbox, and tools I already run, each owning one slice of the work, with me approving every decision that reaches a real person. The architecture matters and I'll get to it, but the architecture was never the point.

The trap I was trying to avoid

The honest reason most operators rent is that building sounds harder than it is while renting sounds safer than it is, and both feelings are wrong. Generic software is built for everyone, which means it is built for no one in particular: the CRM doesn't know that a free email address with no company attached is almost never a real wholesale buyer in my world, the lead tool doesn't know my 5 sales lanes, and the email assistant writes in a voice that belongs to nobody. You spend the first month bending four tools toward a business none of them understand, and then you do it again every time one of them ships an update you didn't ask for. A thing you build knows your business because you taught it, and a thing you rent forgets you the moment you stop paying.

What Albert actually is

Picture an org chart rather than a program. There are two front doors, a command line I run from a terminal and a web dashboard I open in a browser, and both reach the same workforce underneath: a dozen or so single-purpose agents, each very good at exactly one job and completely ignorant of the rest. Some go out hunting for new business, one grades the leads, one builds the day, one writes the email, one reads the replies, one keeps score, and a few do the unglamorous maintenance that keeps a CRM from quietly rotting from the inside. None of them tries to be the whole salesperson, and that ignorance is the design rather than a flaw in it. Let me walk a lead through the ones that do the selling, then show you the crew that keeps the data clean.

It runs on the stack I already had, with HubSpot holding the records, Outlook handling the mail, and Claude doing the thinking, and none of the data ever leaves the building. Albert was never trained in the machine-learning sense, since there's no fine-tuned model anywhere in it; instead it's grounded three ways, by a company bible that tells it who we are, a library of lane-and-language templates, and a learning log that records every correction I make, so over time it gets better at how I write, how my business actually works, and what a real deal looks like.

Spread that across the world and it stops looking like a program and starts looking like reach. On Albert's map the leads scatter across all 14 target markets on four continents, each dot sized by how many sit in that country and colored by temperature, so at a glance I can see not just where the pipeline is but where it is heating up.

The work, end to end

Follow a single lead and the whole machine comes into view. A contact arrives, usually from a web form, and the grader reads everything attached to it, the email history and the company and the website, then hands back three things: a lane, which is whether this is wholesale or B2B development or an event or OEM or retail or a partner, a score of hot, warm, or cold, and a one-line reason I can read in three seconds. It runs a free check before it spends anything, so a throwaway address with no company never reaches a paid model and only the genuinely uncertain leads do.

Then the planner takes everyone hot or warm, plus the deals already in motion, and builds the day. Not a to-do list but a real calendar: the just-replied hot leads get the early block at 15 minutes each, deals in active negotiation get mid-morning, warm follow-ups get late morning, the long shots, deals that went cold months ago, get the early afternoon, and cold verification gets the end of the day. Everything is grouped by language so I am not whiplashing between Spanish and Portuguese every fifteen minutes, and anything I don't finish quietly reschedules itself to tomorrow.

For the contacts ready for a message, the drafter writes one, and it chooses how to write from the history it can see: a first touch for someone brand new, a follow-up that softens with each attempt for someone who has gone quiet, a continuation that reads the existing thread and answers on topic for someone who has already replied, and a different opening entirely for someone who came to us first. It writes in whichever of the 5 languages the prospect uses, theirs rather than mine.

The rest of the workforce runs in the background while I work the calls. The lead-generation agent goes looking for operators who never filled out a form, searching 14 countries in their own languages across five verticals, from distributors and eco-resort developers to outdoor-gear retail, camping and overlanding, and events. The partner prospector hunts a different kind of contact entirely, the surveyors and architects and contractors across 50 states who send business rather than buy it. An inbox-triage agent reads every reply as it lands and sorts it into one of six buckets, flagging the ones that want pricing or a follow-up or are asking a real question as buying signals the moment they arrive. When one of those conversations turns serious, Albert can open the deal in the pipeline and line up the invoice behind it, all through the same HubSpot connection, so the path from a first reply to signed paperwork never has to leave one system. A dead-lead detector keeps a quiet watch on anything that has gone 45 days without a heartbeat. And an outcome tracker sits at the very end of the line, matching the deals that close back to the outreach that touched them. Every one of those steps stops at the same gate, which is me, because the system proposes and I dispose.

And underneath all of that runs a quieter crew whose only job is keeping the data honest, because a CRM rots from the inside the moment nobody tends it. One agent finds duplicate contacts and merges them, another clears out duplicate tasks before they pile up, and another fills in missing company phone numbers and reconnects orphaned contacts to the businesses they actually belong to. It is the unglamorous half of the job, the part no per-seat subscription ever quite does for you, and it runs for the same fractions of a cent as everything else.

Making it write like me, not like an AI

The drafts were the part I was most afraid would give the whole thing away, since nothing announces "a machine wrote this" like a cold email that reads like a machine wrote it. So most of the craft went into the opposite. A forbidden-phrase list is baked into the prompt and a plain regex strips the tells out afterward as a safety net, so the em-dashes and the "leverage" and the "seamless" and the rest of the corporate-AI vocabulary are gone before I ever see a draft. It matches the prospect's own language, reading how they describe themselves and then picking one word, glamping or eco-resort or experiential, and using it consistently instead of guessing at five. It knows exactly how long the silence has been, so a note written after 29 days of quiet acknowledges the 29 days rather than pretending we spoke yesterday. And every draft is mine before it goes anywhere: I can rewrite it myself or just tell it what to fix, make it shorter, warm up the opening, mention the Star Valley job, and it reworks the draft on the spot, with each edit logged so the next one comes back a little closer to how I would have written it in the first place.

I put it up against HubSpot and Salesforce

A few days ago I ran a simple test, the kind a skeptical buyer would actually care about. Imagine hiring a new sales rep and handing them the AI inside HubSpot, the AI inside Salesforce, or Albert, then asking the only three questions that matter: could they find and work a lead, would it be accurate and easy, and would it stay cheap? I picked an industry, hospitality, went looking for glamping operators, and pulled the kind of cold leads you actually get in the wild, where some came with an email, some with only a phone number, some with just a name, and some with nothing but a business address. Then I dropped them into each system and tried to turn them into someone I could contact.

HubSpot did well when a lead already had a website to pull from and laid out plenty once it had something to work with, but hand it a bare name or an address and it had nowhere to go. Salesforce had more features, yet they only helped when the contact's fields were already filled in, and the moment I wanted it to find the missing pieces I was looking at bolting on more paid services, which is exactly the direction you never want a cost to move. Both tools quietly assumed I would show up with the data already in hand.

Albert assumed the opposite. Handed almost nothing, a name or a number or a single email address, it used the email to find the company's domain, pulled up the site, wrote the missing details back into the CRM, summarized the business, graded the lead hot, warm, or cold with a reason I could read in a sentence, and drafted a first outreach email for me to approve. A scrap of contact information went in, and a researched, scored, ready-to-send opportunity came out, for a few cents. The rented tools wait for you to feed them; Albert goes and gets it, and the only time either incumbent could keep pace was the moment it offered to find the rest for an extra fee. It also remembers what it has seen, so when I tag a company as a competitor it stays tagged and drops out of every future search, which means I am never handed the same name twice and never spend a cent researching someone I will never sell to.

The receipts

I didn't want a demo, because demos are easy, so the only question that mattered was whether a thing I'd built alone could move real money. The very first version of Albert was crude, and a single run of it cost $15.07, but that one expensive pass scored 1,051 contacts and surfaced $82,998 in pipeline I didn't know I had, leads that had been sitting invisible in my own CRM for months, and it pulled $18,000 in quotes out of contacts I had already written off. The idea clearly worked. The price tag told me it wasn't finished.

So I did what the opening instinct demanded and went hunting for the waste. I split the work across agents sized to each task, a cheap model where I only needed a label and a stronger one only where the writing had to be good, and I pushed everything I could off the paid models and onto free Python and the HubSpot API, which it turns out will do a startling amount of the busywork for nothing. The cost of a run fell by more than 90 percent, from $15 to well under a dollar, often closer to 30 cents. What once paid for a single pass now runs the whole operation for cents a day, and across 42 days of real work the finished system has cost $15.85. The first version spent $15.07 in one morning; the version I kept has spent about the same in six weeks.

And the version I kept closes business. Over the quarter two deals worth $29,201 came in, both carried by Albert from the first touch through the follow-up, with a $5,000 deal still live in the pipeline.
Here is the part I won't airbrush. For weeks, Albert's own performance dashboard read zero across the board, zero revenue attributed and zero deals won, and not because anything had broken. Sales cycles run 3 to 18 months, so the software is simply faster than the business it serves, and while my dashboard insisted nothing had happened the bank account quietly disagreed. Both were telling the truth, just on different clocks. Full attribution won't mature until late this year, deal by deal, on its own schedule, and a system honest enough to measure its own results has to be patient enough to wait for them.

Why it never fell over

Two more decisions did the load-bearing work on reliability. The first is that Albert refuses to redo finished work, because the cheapest model call is the one you never make, so the grader only looks at a contact when something about it has actually changed since the last time. That sounds trivial, and it caused the nastiest bug in the project. The first version decided whether anything had changed by watching the field that logs the last email reply, a field that often stays empty because the mail integration doesn't always catch the inbound message, so Albert quietly concluded that nothing had ever changed for anyone and skipped every contact, confidently and without throwing a single error. When you build a system that skips work to save money, the signal it uses to decide what to skip is load-bearing, and choosing it wrong turns your optimization into an outage that never announces itself.

The second is that the agents never talk to each other. No agent calls another and the grader never hands off to the planner; instead they share a destination rather than a conversation, so the grader writes a score to the CRM and later, independently, the planner reads it back. The payoff shows up in how the system fails, because nothing waits on anything else: any agent can crash and be restarted without the rest of them noticing, and there's no fragile chain where step four dies the moment step three hiccups. Independent systems fail gracefully, while monolithic ones fail all at once, and I wanted the kind that degrades a little over the kind that collapses completely.

One decision matters more than those two. Albert has no autonomous mode, and not because I haven't built it yet, since there's no code path anywhere that writes to the CRM or reaches a real person without an explicit yes from me. Skynet's whole problem was that nobody had to press the button, and an AI that acts on the world at machine speed, on imperfect information, with no human standing between the decision and the consequence, isn't a productivity tool but a liability with good marketing. The human in the loop isn't a feature I haven't gotten around to removing yet, it's the feature.

Why I turned it off

Because the experiment was over, and it had won. I didn't build Albert to run forever, I built it to answer a single question, which was whether one person with no team and no SaaS budget could own the entire machine around the selling and have it pay for itself. At $15.85 in, with five figures closed and the dashboard still catching up to reality, the answer was obvious, and a test that has already succeeded doesn't need to keep running to stay true, so I switched it off the way you stop a stopwatch once you have the time.

We were sold a dream about the AI that replaces the person, and the one I actually built does the opposite, because it replaced a $460 invoice and a team I never hired, and it answers to me. I can turn it on when I need it and off when it has made its point, because I own it instead of renting it. That is the part worth keeping, more than anything about Albert itself: the leverage didn't come from the AI being smart, it came from the AI being mine, infrastructure I own instead of software I rent. That isn't a smaller version of the dream we were sold. It might be the better one.

Why it isn't really about my business

Albert stands on three things: a company bible, the HubSpot API, and the agents. The agents are the workers and the API is the plumbing, but the company bible is the part that actually knows the business, who I am, who my competitors are, what we sell and to whom. It is a plain document, and I can rewrite it. Swap that one file and Albert's whole personality and commercial instinct change with it, because the agents never needed to know what business they were in. The company bible tells them, so the same machine that sells hospitality could sell energy, or loans, or finance, or anything with a pipeline and a customer on the other end of it.

Which means the real ceiling on Albert was never the model or the code. It was me. Underneath everything, Albert is a new sales rep: it learns from you, from your website and your materials and every draft you correct, and it gets a little better at the job every day while keeping the business tidy enough that you can spend your hours on the one thing software still cannot do for you, which is close. I turned this version off because it had made its point. The next one is just getting started.

Frequently asked questions

What is Albert?
Albert is a multi-agent sales system built and run by a single operator. It scores leads, plans the day, drafts outreach in 5 languages, sorts inbound replies, finds new prospects and partners, diagnoses the pipeline, tracks which deals close, and keeps the CRM clean behind the scenes, all from one command line and one web dashboard on top of HubSpot and Outlook.

How much did Albert cost to build and run?
Its total spend to date is $15.85, against roughly $460/month for the conventional SaaS stack it replaced, which was a CRM, a lead tool, an email assistant, and a meeting-prep bot.

Did Albert actually generate revenue?
Yes. Albert's outreach closed two deals worth $29,201 this quarter, with a $5,000 deal still open, against $15.85 in total cost. Because sales cycles run months, the system's own attribution dashboard read zero for weeks before those deals finally closed.

Why did you turn it off?
It was a time-boxed experiment to prove that a solo operator could own the full sales machine cheaply and have it pay for itself, and once it had, the test was complete. Because I own it rather than rent it, I can run it again whenever I choose.

How does Albert keep costs so low?
It filters the obvious cases out for free before any paid model is called, never reprocesses a contact that hasn't changed, and routes work by economics, with a cheap fast model for high-volume scoring and a stronger model only where writing quality matters.

Does Albert send anything without human approval?
No. There is no autonomous mode in the code, and every contact creation, score change, and outbound email passes through an approval queue first.

Can Albert work for an industry other than yours?
Yes. Everything Albert knows about a specific business lives in one plain document called the company bible. Swap that file and the same agents sell something else, because the workforce and the plumbing never depended on the industry, the company bible supplied it. It could run sales for hospitality, energy, finance, loans, or anything with a pipeline and a customer.

The Scale Wall: Why AI Gurus Are Building Toys While the World Needs Architects

Alan Scott Encinas — Sun, 05 Jul 2026 12:00:00 +0000

"Day 5: Finished learning Hugging Face. Built a script that passes a PDF to a pipeline() wrapper. Big lesson: the model is the brain. Day 6: moving on to dominate AI architecture."

I did not make that up. Some version of it scrolls past me every single morning, and every morning it lands the same way. It is the technical equivalent of skimming the index of a biology textbook and then offering to perform open-heart surgery by lunch.

We are living through a strange kind of whiplash. On one side, autonomous agentic architectures, localized models, and cognitive orchestration are quietly rewiring how real industries run. On the other, my feed is an endless parade of people who speedran a single high-level API tutorial on Monday and rebranded as a Senior AI Architect by Tuesday. It treats artificial intelligence like one more trendy JavaScript framework, as if you only need to memorize a few import statements, copy a UI template, and call it a career.

So we trap ourselves in a digital playground. We build Jarvis-style second brains and slick automated email carousels because they look incredible and give us that Iron Man rush, completely blind to whether the thing underneath is actually good software. If a model sits on the desktop and answers our prompts, we fall in love with the novelty and stop asking the only question that matters at scale: does this hold up?

Because while the gurus sell courses on how to build flashy novelties, real enterprise systems are quietly shattering under the weight of terrible architecture.

Notes from the field: the scale wall

Over the last six months I have been brought in to audit and re-engineer AI systems for roughly two to three companies a month. The spread is chaotic on purpose: real estate marketing agencies, cannabis compliance firms, overseas logistics providers, OEM manufacturers, unsecured lending underwriters. Different worlds, identical failure.

Every one of them fell for the same thing. Call it the Guru Mirage. They had sharp ideas and knew exactly what their endgame was. They had seen a flashy video of a cool little tool that scrapes Reddit, Twitter, and TikTok and instantly spins up optimized marketing copy, and they thought: perfect, let's build a whole enterprise workflow around that loop.

And it worked, at first. It produced some solid concepts. Then they tried to scale that linear pipeline to real business volume, and the engine choked.

The systems went stagnant, and the reason was always the same. They were built on vibes and brittle, linear chains, forcing enormous context windows to pass raw data back and forth on every call, spending 30 to 100 times the compute a task actually needed to do work that should have been cheap. They had built a fragile spaceship out of cardboard, pointed it at the stars, and wondered why it came apart the moment it cleared the atmosphere.

That is the scale wall. It is where a thing that demos beautifully meets the volume of an actual business and falls over. And it is almost never a model problem.

The hierarchy I use now

From six months of pulling these systems apart and rebuilding them, here is the map I use to place any AI project. We used to get away with a rough five-level curve. Production reality needs ten.

Level 1: Basic prompting. Raw text in, reliance on system instructions. The starting line where everyone begins.

Level 2: The toy box. API wrappers, off-the-shelf image and video generation, simple linear scripts. This is where the Jarvis second brains live. If your entire strategy sits here, you are playing checkers.

Level 3: The playground. Advanced prompt engineering, sequential chaining, iterative loops, basic out-of-the-box retrieval-augmented generation.

Level 4: Multi-agent orchestration. Multiple baseline agents working together inside shared execution environments, instead of one single stream of code.

Level 5: Deep systemic architecture. Where real systems engineering starts. You assign specialized, finely tuned models to hyper-specific tasks rather than asking one giant model to do everything.

Level 6: Infrastructure and state management. Custom code for complex state dependencies, memory, and deterministic execution hooks. The system stops forgetting what it was doing.

Level 7: Cognitive orchestration. The system no longer just passes text. It manages dynamic routing, self-correction, and algorithmic control flow.

Level 8: Fine-tuning and small language models. Domain-specific adaptation, embedding optimization, and distilling large weights into small specialized models that slash latency.

Level 9: Edge deployment and localized mainframes. Off the standard commercial endpoints entirely: secure, distributed, localized model clusters running on owned hardware.

Level 10: Fully autonomous malleable frameworks. Self-learning, self-optimizing architectures that adapt their own parameters, memory structures, and tool use in response to real-time data.

The distance between a Level 2 script and a Level 7 system is not elegance. It is survival. That brute-force Level 2 pipeline burns its 30 to 100 times overhead because it makes one large general model reason over raw data on every single call. Re-architected properly, the same job runs at orders of magnitude lower cost per task, the difference between paying dollars and paying cents for identical output, at higher throughput, without the whole thing buckling at volume.

The Kaggle blueprint: building Albert

True AI engineering was never only about large language models. Ten years ago the field lived in statistical mechanics, computer vision, and the hard math of machine learning. The interface changed. The math did not. If you want to build things that do not break, you have to step away from the influencer tutorials and look at where real code is still forged: high-level data science hackathons and Kaggle challenges.

When I engineered Albert, my autonomous sales orchestration system, I did not build a linear chain of prompts. I built a self-learning engine that cleans CRMs, tracks client profiles, and matches products in real time. To teach it the chaotic dynamics of live negotiation, we did not hand it a static script. We built a full simulation environment inside the Unity game engine.

The architecture for that simulation was inspired by the structural logic of the Pokémon Trading Card Game AI Battle Challenge, the competition Kaggle and The Pokémon Company are running right now. I did not compete in it. I studied how it is framed, because the framing is the lesson. It is an imperfect-information game: the deck order and your opponent's hand are hidden, so a winning agent has to read a hidden strategy in real time, resolve shifting dependencies, and survive long stretches of chaos. That is not a card-game problem. That is the exact shape of an enterprise sale.

The same instinct solved a very different problem. To parse dense underwriting documents and legal files, we did not dump raw text into an expensive context window and pay to reason over all of it. We took inspiration from hyperspectral imaging, the systems built to detect and isolate specific wavelengths of light invisible to the human eye, and borrowed that mindset of layered, targeted filtering to pull only the few critical variables out of messy compliance documents. Precision over brute force, at a fraction of the cost.

Step outside the box

It is impressive that a baseline tutorial lets a web developer spin up a marketing banner or an app carousel in an afternoon. But we owe ourselves a harder question: is that really the ceiling of our ambition?

We have unprecedented, world-shaping compute sitting at our fingertips. We are standing at a crossroads where the same tools could optimize overtaxed energy grids, accelerate targeted cancer research, and cut the staggering water and power footprint of the very data centers keeping this whole thing alive. Yet a depressing share of the current gold rush goes into building the easiest possible thing that looks good on a timeline.

If you are just getting into this space, stop speedrunning the surface. Drop the guru courses, find a hard machine learning hackathon, and start asking the heavy questions. How does this actually process data? Why is this latency here? How do I optimize this formula instead of praying to it?

The moment you stop treating the model as a magical, isolated black box and start seeing it as one malleable compute node inside a wider, deterministic system is the moment you step out of the toy box. The compute on our desks could change the world. Let's stop building digital pets and start building engines that do.

Frequently asked questions

What is the "scale wall" in AI systems?
The scale wall is the point where a linear, prompt-chained AI pipeline that worked in a demo collapses under real business volume. Throughput stalls, compute cost runs away, and the system fails in brittle ways, because it was assembled from ad-hoc chains instead of built on deterministic architecture and state management.

Why do AI projects cost 30 to 100 times more than they should?
Naive pipelines force huge context windows to pass raw data back and forth on every call and lean on one large general-purpose model for every task. Routing specialized or smaller models, managing state deterministically, and filtering inputs before inference cuts cost per task by orders of magnitude, often the difference between paying dollars and paying cents for the same output.

Do you need machine learning skills to build production AI, or just API knowledge?
Production-grade AI requires systems engineering and data-science fundamentals: state management, model specialization, and orchestration. API fluency alone gets you to about level two of a ten-level maturity hierarchy, enough for a demo but not for a system that survives real volume.

What is a better way to learn AI engineering than guru courses?
Work a hard machine-learning hackathon or a Kaggle-style competition and ask architectural questions, how the system processes data, where latency comes from, and how to optimize the underlying math, instead of speedrunning surface-level API tutorials.

I built an AI agent that reads my CRM emails and scores leads for $0.01 each

Alan Scott Encinas — Sat, 04 Jul 2026 13:00:00 +0000

I built an AI agent that reads my CRM emails and scores leads for $0.01 each

It scored 1,051 leads in one run for $15.07, and surfaced $82K+ in pipeline I didn't know I had.

The problem

I run a company in the outdoor gear and hospitality industry. We manufacture luxury glamping structures, geodesic domes, and event tents for resort developers, hospitality operators, and event organizers worldwide. A single deal can be worth tens of thousands of dollars. Some clients buy 50-200 structures at a time.

I also do everything alone. Sales, lead gathering, qualification, follow-up, customer service, all of it. My HubSpot CRM had 1051 contacts. Some were resort developers in Costa Rica looking to buy 20 units. Some were people on Instagram asking "how much for a tent?"

They all looked the same in my CRM.

I was spending hours every week scrolling through contacts, trying to remember who was worth calling. Meanwhile, good leads went cold because I couldn't get to them fast enough. The people who called my cell phone got attention. Everyone else got whatever time was left.

What I built

I built a Python script that connects to HubSpot and uses Claude (Anthropic's AI) to score every contact in my pipeline. But here's what makes it different from every other "AI lead scoring" tool: it actually reads the email conversations.

Not metadata. Not "opened email 3 times." The actual words. The full thread. It reads that a lead wrote "we have 30 acres in Tulum and we're looking to build a 20-unit eco-resort, what's your lead time on safari tents?" and it understands that this is a completely different animal than someone who wrote "prices?"

For each contact, the agent:

Pulls the full email conversation history from HubSpot
Checks if they have a company linked, if not, extracts the domain from their email and creates one
Visits their website to verify it's a real business
Sends everything to Claude for intelligent scoring
Writes the score, lane, and reasoning back to HubSpot

The results

Metric	Value
Cost per lead	$0.01
Leads scored	1,051
Total API cost	$15.07
Pipeline surfaced	$82K+

On a single production run, the agent scored all 1,051 contacts in my CRM and surfaced over 100 actionable leads, hot and warm prospects I didn't know were sitting in my pipeline. But the agent didn't just score them. It told me exactly who to call first, in what order, and what to say when they picked up.

So I called them.

Within days, I'd generated $18,000+ in quotes directly from following the agent's follow-up recommendations. Total pipeline value identified: $82,998. All from leads that were already in my CRM, already had email conversations with me, and were completely invisible because I didn't have time to read through hundreds of threads.

One was a resort developer in Baja Mexico with a 12-unit glamping project and construction platforms already built. The original form data was thin, just a name and email from a Facebook ad. But the email conversation revealed everything: specific unit counts, a location, a timeline, and active planning. The agent changed it from Retail/Cold to B2B Development/Hot and told me to call immediately.

Another contact had been tagged as Retail/Warm for weeks. The agent read the email thread and discovered they were an established campground owner with two properties, asking detailed questions about specific models, and had already committed to a call by Thursday. Upgraded to Hot with a specific recommended action: "Prepare pricing on cost-effective models for both campground locations."

It also found leads I should stop wasting time on. One contact that looked promising had actually been trying to sell promotional services TO my company, not buy from me. The agent caught that from the email content and tagged it as Cold with the note: "No buying intent, attempting to sell services to Trend Tents."

What it costs vs. everything else

The total monthly cost for this system is about $35, that's $20 for HubSpot Starter and roughly $15 in Claude API costs for a full CRM sweep. For context, HubSpot's built-in lead scoring requires the Professional plan at $890/month. And their scorer doesn't read your emails. Mine cost $15 and generated $18K in quotes on the first run.

The ROI is absurd

$15.07 in API costs. $18,000+ in quotes generated. $82,998 in total pipeline surfaced. I called the leads the agent told me to call, and they converted. Those leads were already in my CRM, already had email conversations, and were completely invisible to me because I didn't have time to read through a thousand threads.

Before and after

Before: manual process

Metric	Value
Time to qualify a lead	5-15 min
Daily qualifying time	1-2 hours
Leads reviewed per day	10-20
Hot leads missed	Unknown
Email reading	Skim at best
Company enrichment	Manual lookup
Monthly cost	Your time

After: AI sales agent

Metric	Value
Time to qualify a lead	~3 seconds
Daily qualifying time	0 min
Leads reviewed per day	1,000+
Hot leads missed	Zero
Email reading	Every word
Company enrichment	Automatic
Monthly cost	~$15

How it works

The architecture is simple. No ML models to train. No databases to maintain. One Python file, two API keys.

HubSpot CRM
  → Pull contacts with recent activity (skip already reviewed)
  → For each contact:
      → Read email conversation history (last 10 emails)
      → Check company association (create from email domain if missing)
      → Visit website (verify legitimacy)
      → Send everything to Claude API
      → Claude returns: lane, score, reason, recommended action
      → Write results back to HubSpot
      → Stamp review date
  → Print summary: hot leads, score changes, companies added

The "intelligence" is a system prompt that describes my business, my customer types, and what signals indicate a hot vs. cold lead. It's written in plain English, no code, no model training, no feature engineering. You literally describe your business to Claude the way you'd describe it to a new sales hire, and it scores leads based on that understanding.

The scoring logic classifies every lead into a commercial lane (I have five: Wholesale, B2B Development, Event, OEM, and Retail) and then scores them as Hot, Warm, or Cold based on lane-specific criteria. A Hot wholesale lead looks different from a Hot event lead, and the agent knows that.

The email reading is what makes it work

Every other lead scoring tool I looked at scores based on form data and activity metadata, "opened 3 emails, visited pricing page, job title is VP." That's useful, but it misses the most important signal: what the person actually said.

When Claude reads an email thread where someone writes "We're converting 30 acres in Guanacaste into an eco-lodge. Our architect has the platform layouts done. We need 15 safari tents delivered by Q1 2027, what's your lead time and bulk pricing?", it doesn't need a scoring algorithm. It knows that's a hot lead because it can read.

Similarly, when it reads a thread where someone asked for a price list, received one, and then went completely silent for 6 months despite 5 follow-ups, it knows that's cold. And it writes that specific reasoning into HubSpot so when I see the contact, I know exactly why and what to do.

It also cleans up your CRM

One unexpected benefit: company enrichment. About 40% of my contacts had no company linked in HubSpot. The agent checks each contact's email domain, if it's a business email (not Gmail/Yahoo), it searches HubSpot for a matching company. If one exists, it links them. If not, it creates the company and links it.

Over two runs, it added 8 companies to my CRM that I never would have created manually. That's not just tidiness, it's data quality that makes every future report, filter, and search more useful.

Get the code

The agent is open source. One Python file. One dependency (pip install requests). Customize the scoring logic by editing plain English, no ML knowledge needed.

GITHUB REPOSITORY

github.com/AlanSEncinas/ai-sales-agent

Free and open source. Works on Mac, Windows, and Linux.
One dependency. Customize scoring in plain English.
Works with HubSpot Free, Starter, Professional, or Enterprise.

Setup takes about 5 minutes: clone the repo, paste two API keys (Claude + HubSpot), create four custom properties in HubSpot, describe your business in the scoring config, and run it.

If you're a solo operator or small sales team drowning in leads you don't have time to evaluate, this might be the highest-ROI hour you spend this month.

I was sure, and I was wrong

Alan Scott Encinas — Fri, 03 Jul 2026 13:00:00 +0000

Builder Journal · Hyperspectral Object Tracking Challenge 2026

Every origin story has the genius idea scrawled on a napkin. Mine had one too. I sketched it, I believed in it, I built it carefully, and then the data walked up and quietly tore the napkin in half.

This is the sixth entry in a log from inside the Hyperspectral Object Tracking Challenge 2026, where I'm tracking a single object through video shot in light the eye was never built to see. The goal is a podium and a paper. This entry is about the contribution I was most certain about, and the afternoon it died.

The idea that sounded obviously correct

My first real contribution, the one I expected to be the centerpiece, was a smarter way to choose which information the tracker pays attention to. The logic was airtight on the napkin. Be selective. Find the signal that best separates the target from its background, feed the model that, and drop the rest. It is the kind of idea that sounds so obviously correct that nobody bothers to check it.

That should have been my first warning. The ideas that feel unarguable are the ones most worth arguing with.

What the data said

I built it properly, written test-first, then measured it head to head against the cruder approach already sitting in the pipeline. The incumbent does roughly the same job in a blunter, less principled way, the kind of thing you would be slightly embarrassed to put in a paper.

My elegant version lost. Not by a hair. It was beaten clearly by the blunt incumbent, and when I stacked more cleverness on top to rescue it, the gap only widened.

The reason, once I dug in, was humbling in a way I keep having to relearn. My principled method quietly assumed it had enough clean data to estimate what mattered. In the hardest cases it simply does not. The estimate goes noisy at exactly the moment the stakes are highest, and the crude method, precisely because it was not trying to be clever, shrugged off that noise. Elegance was the liability. Bluntness was the feature.

Why I'm telling you a story where I lose

I did the thing this whole log exists to do. I published the loss. The failed contribution is written up as a documented negative result, not deleted, so the next person with the same airtight-on-paper idea can find my receipt and spend their week on something better.

Complexity has to earn its place against the simple baseline, every single time. The fact that your idea is more sophisticated is not evidence that it is better. It is only evidence that it is more sophisticated, which is a different thing wearing the same suit.

There is a quieter gift buried in a failure like this. A confident idea dying tells you something true about the shape of the problem that a success never would have. I can't say more about what it told me, because that part is still my edge and this log runs on a deliberate delay. But that dead end was the most useful hour of the week.

Where I'm standing right now

A small change I'm keeping to myself for now nudged the board up to around 0.547, a hair under the podium line. The elegant idea that was supposed to carry me there is in the graveyard. The thing that actually moved the number was almost embarrassingly simple. That is, reliably, how it seems to go.

I'm going to set this competition down for a little while. Not because it is over, it is very much not over, but because I have six other competitions running and one of them has a story that has aged far enough to be safe to tell. This log was always going to jump between them. That is the whole reason I can write it truthfully while the races are still live.

More in this series

← Start here: I entered a competition to track objects in light you can't see, what the challenge is, and why it's hard.
Why my 21 AI agents aren't allowed to talk to each other, the same discipline in a different project: complexity has to earn its place against the simple version.
The full Hyperspectral Object Tracking Challenge thread, every entry from this competition.
The Builder Journal, the live log across all my competitions.

This is part of an ongoing builder's log written from inside live competitions. You're reading where I was, not where I am.

DEV Community: Alan Scott Encinas

What Is Cognitive Orchestration? Why AI Systems Can't Be Built Like Traditional Software

What Is Cognitive Orchestration?

Why Does Cognitive Orchestration Exist?

The Evolution of Orchestration

Cognitive Orchestration vs Prompt Chaining

Cognitive Orchestration vs AI Agents

The Architecture of Cognitive Orchestration

State Synthesis

Adaptive Planning and Decision-Making

Verification Loops

Persistent Memory

Real-World Applications of Cognitive Orchestration

The Field Caught Up

Intelligence Emerges From Architecture

Key takeaways

FAQ

Related reading

The Infrastructure Fallacy: Why Most Sales Problems Are Actually Strategy Problems

The Hidden Signal

Infrastructure Is Not Strategy

The Automation Multiplier

Distribution Is the Operating System

Where Good Talent Becomes Bad Data

Systems Reveal Problems That Opinions Hide

A Practical Operating Model

The Real Competitive Advantage

Frequently asked questions

Related reading

I made my agent more capable and it got worse

The obvious upgrade

Twice

The reason, when I finally cornered it

What it means

Where I'm standing right now

Building LunarSite: the director's commentary

The other half of the story

Scoping so it actually ships

The harness before the work

Fork #1: the encoder I didn't use

Fork #2: when the right knobs lose

Fork #3: "wrong model" or "wrong data?"

Why a deterministic rule still needs a model

The dead code that became the uncertainty story

Validation as a design choice

The cuts, and the judgment behind them

What I'd do differently

The Paintbrush Paradox: Why the Monolithic Era of AI Is Crumbling

The Myth of the Infinite Context Window

Bigger Canvas, Worse Paintings

The Rise of Cognitive Orchestration

Where the Line Is Drawn

The Next Generation of AI

Frequently asked questions

I looked unbeatable in practice and lost the real fight

The gym every competitor has to build

The number that looked like a win

The part that actually scared me

What it actually means

Where I'm standing right now

I spent a few months teaching a laptop to read the Moon

The question I couldn't drop

A video-game Moon taught it about the real one

The part that nearly broke me

Ranking 315,000 patches of Moon

Teaching it to say "I don't know"

Knowing when to stop

See it for yourself

I'm trying to teach a machine to play a game it has never seen

What this actually is

Why it's the field's uncomfortable mirror

The interactive leap

Why anyone should care

The bet I'm making

Where I'm standing right now

I Built an AI Sales Team for Under $1 a Day. It Closed $29,201 and Beat Salesforce and HubSpot's AI

The trap I was trying to avoid

What Albert actually is

The work, end to end

Making it write like me, not like an AI