DEV Community: Kfir Adut

A Test Report Should Say What It Cannot Prove

Kfir Adut — Wed, 14 Oct 2026 05:25:06 +0000

A test report can be accurate and still lead to the wrong decision.

The problem is often the space between what the test checked and what the reader thinks it checked. "Passed" is a short word. It can quietly absorb claims about usability, reliability and readiness that were never part of the test.

In my game projects, Instinct does much of the implementation and testing. I remain responsible for the direction and the decision to ship. That makes the handoff report part of the product work. I need to know what the evidence supports before deciding what should happen next.

I have written about stop rules before. This is the reporting side of the same problem: how to describe a result without letting it become larger than the observation behind it.

Start with the claim

"The feature works" is too broad to test usefully.

"A hostile shot cannot award the same near-miss reward twice" is narrower. So is "the intended gameplay screen appears after launch." Neither statement proves that someone understands the mechanic or enjoys playing it.

Those are different questions. They need different evidence.

Before reading a result, I want to see the claim in a sentence. If the sentence contains several claims, it probably needs to be split. A test should not inherit a bigger job because the report uses a convenient label.

Put the observation beside it

A useful report connects a claim to an observation someone can inspect.

For a hypothetical game build, that might mean a regression trace showing that one projectile cannot award twice. For a visual claim, it might mean a capture of the relevant screen. For a touch-input claim, it requires evidence from the target interaction, not a desktop approximation quietly presented as the same thing.

These are examples of evidence requirements, not a claim that a current candidate has passed them.

The distinction matters because one artifact can be real but irrelevant. A screenshot proves that something was rendered. It does not, by itself, prove that it was the right screen, the right build or the state the task required.

Add the limit while the result is still fresh

I would put a short limit beside every meaningful result:

Claim: the same projectile cannot award the reward twice.
Evidence: the regression route checks repeated contact with that projectile.
Limit: this does not establish that the reward feels fair on a phone.
Next question: can a player understand why the reward appeared?

The limit is not an apology. It stops a precise result from becoming a vague promise.

It also makes missing evidence visible. A device check that has not happened is not a failed device check. But it is not a pass, either. Reports need language for all of those states.

Keep the next step proportional

The report should make the next action easier to choose.

If the remaining question is about player understanding, another run of the same logic test will not answer it. The useful next step could be a clearer cue or a focused playtest. If the remaining question is about an incorrect screen, adding a feature is a distraction.

This is where fast AI-assisted work can help or hurt. Preparing another candidate is cheap enough that it can hide the fact that the missing evidence has not changed.

I do not want a longer report for its own sake. I want a report that prevents me from approving a different claim from the one that was tested.

The strongest handoff is often a modest one: this is what we observed, this is what it supports, and this is what we still do not know.

Related writing

How this article was prepared

Instinct helped draft this article from my published project writing. I remain responsible for what appears under my name.

Fast AI Work Needs a Stop Rule

Kfir Adut — Thu, 08 Oct 2026 14:19:52 +0000

A fast implementation can leave the important decision untouched.

The code exists. The build passes. There is a screenshot in the report. But should this version reach a user? What does the evidence actually establish? Who decides what happens next?

Those questions become more important when an AI assistant does much of the building. It can prepare another version quickly. That makes it tempting to treat the next step as a continuation of the previous one, even when the next step carries a different kind of risk.

I have written before about the division of work in my Godot projects: Instinct does much of the implementation and testing; I own the direction, the standard and the decision to ship.

The practical question is how to make that boundary useful without turning every small change into a meeting.

My answer is a stop rule.

Separate the work from the commitment

Researching a change, implementing it in a candidate build and releasing it are different decisions.

An assistant can prepare a candidate without deciding that people should receive it. It can run a test without deciding that the result answers every product question. It can propose a fix without deciding that the cost of changing the current version is acceptable.

This is not an argument for asking permission before every line of code. It is an argument for naming the point where preparation becomes commitment.

Before that point, keep the work moving. At that point, require the evidence the decision needs.

A passing test has a limited meaning

The close-call mechanic in Ctrl+Alt+Liberate is a useful example.

In the candidate described in my earlier article, a hostile shot passing near the ship could give a small FOCUS reward. Desktop regression checks covered conditions such as excluding protected passes and preventing the same shot from awarding the reward repeatedly.

Those checks mattered. They did not establish how the vibration felt in a hand, whether a player could read the margin on a phone, or whether the game was balanced for people rather than scripted controllers.

A desktop pass supported the next implementation decision. It did not settle the release decision.

That distinction is easy to lose in a report that says only "tests passed." A better report names the claim that passed and the claim still waiting for evidence.

Write the stop rule before the result arrives

A useful stop rule is specific enough that a team can apply it without another round of interpretation.

For a hypothetical Android release, I would write something like this:

Prepare the candidate and run the regression route. Stop before release if the intended gameplay screen is not visible, if input fails on the target device, or if the device check has not happened. A desktop pass alone does not clear the release.

This is an example, not a report that those checks have been completed on a current build.

The rule gives the assistant room to work. It also prevents missing evidence from becoming a quiet assumption.

The most important phrase is often "has not happened." A missing check is different from a failed check, but neither should be presented as a pass.

Keep a small decision card

I would put five fields beside a candidate change:

Decision: What are we deciding now?
Evidence: What observation could change that decision?
Limit: What does this evidence not establish?
Owner: Who can make the commitment?
Stop condition: What blocks the next step?

For the hypothetical release above, the decision is whether to release, not whether the code compiles. The evidence includes the target-device route. The limit is that one successful route does not prove long-term balance. The owner is the person responsible for the release. The stop condition includes an absent device check.

This is a small amount of writing. Its value comes from making the boundary visible before a good-looking result makes everyone impatient.

Stop the wrong next step, not the whole project

A stop rule should leave useful work available.

If the release is waiting for a device check, the assistant can prepare the test route, document the known limits or investigate a separate regression. It should not describe the release as complete or replace the missing device evidence with more desktop runs.

The same principle applies when a test fails. The next useful action may be diagnosis rather than another feature. More output is not automatically more progress.

I want AI assistance to reduce the work I have to carry, not hide the decisions I still need to make. Clear boundaries help with both.

The assistant can do much of the work. The evidence tells us what we know. The person responsible decides what that knowledge is enough to justify.

A note on how this article was made

Instinct helped prepare this article from my published project writing. I remain responsible for what appears under my name.

A Feature Isn't Finished Until the Player Can Read It

Kfir Adut — Mon, 05 Oct 2026 12:14:59 +0000

A mechanic can work exactly as coded and still be unclear to the person playing.

That is an easy gap to miss while building a game. You know what each meter means. You know why an enemy changed direction. You know what a warning is supposed to tell the player. After spending time with the system, its logic starts to feel obvious.

The player does not have that context. They see a screen, make a guess, and learn from what happens next.

When I work on a small arcade shooter, I try to think about a mechanic in two parts: what the system does, and what the player can understand about it. A near miss can reward careful movement. A focus meter can give a player a reason to keep a streak alive. A risk indicator can make a powerful choice feel costly. Those rules matter only if the feedback makes them readable.

This changes how I evaluate a feature. "Does the counter update?" is a useful implementation check. It is not the same as asking, "Can someone tell why the counter changed?" A test can verify the first question. A playtest, a clear visual cue, or a short explanation may be needed for the second.

The feedback does not have to be elaborate. A distinct sound, a brief animation, a change in color, or a well-timed pause can connect an action to its result. The point is not to decorate every event. The point is to give the player enough information to form a reliable mental model.

There is a limit, too. Explain everything at once and the screen becomes another manual. Good feedback arrives when it can help: before danger, at the moment a choice is made, or immediately after the result. Let the player learn one rule, use it, and then meet the next one.

I find it useful to ask three questions while tuning a mechanic:

What did the player do?
What changed because of it?
Could the player tell why?

If the answer to the last question is no, adding another mechanic probably will not help. The system needs a clearer signal, a better sequence, or a simpler rule.

This is not only a usability concern. Readable rules make a game feel fair. A player can accept a hard outcome more readily when they understand what led to it. When the connection is hidden, even a correct result can feel random.

So I treat clarity as part of implementation, not a final polish pass. A feature is not finished when its values change. It is finished when the player has a fair chance to understand what those changes mean.

A Near Miss Is a Design Decision, Not Just a Collision Check

Kfir Adut — Thu, 01 Oct 2026 08:35:04 +0000

A bullet can miss the player by a few pixels and still change how the game feels.

In Ctrl+Alt+Liberate, a portrait shoot-'em-up I am building in Godot, I wanted the space just outside the ship's hitbox to matter. Dodging a crowded pattern should feel deliberate. It should not require a player to take damage before the game acknowledges a risky move.

So we added a close-call mechanic. A hostile projectile passing near the ship gives a small FOCUS reward and one visible cue. That sounds simple. The interesting work was deciding when not to reward it.

This is a desktop-tested candidate, not a claim that the game is balanced or that the tactile response feels right on a phone.

Controlled desktop demo, v15. A live shot passes through the close-call band; the cyan CLOSE CALL +FOCUS cue appears and the close-call count increases once.

The first rule: near is not the same as safe

A projectile earns a close-call reward only within a defined band outside the ship. In the current candidate, that band is roughly 31 to 57 Godot units away, with a narrower horizontal condition. The values are tuning choices, not universal numbers for a portrait game.

The reward is small: 0.04 FOCUS. Each pooled projectile can award it once. The player can still move into that same projectile afterward and get hit. A close call does not turn a live threat into a harmless effect.

We exclude protected passes. If SHIELD is active, or the ship is briefly invulnerable after taking damage, passing through a shot is not a dodge worth rewarding. A regression check also covers the less obvious case: a shot passes while SHIELD is active, then later passes near the unprotected ship. The protected pass should not consume the shot's one available close-call reward.

Controlled desktop demo, v15. The same live shot passes while SHIELD ON is active; no close-call cue or focus reward is granted.

That distinction is important when projectiles are pooled and reused. A stale "already rewarded" flag on a recycled shot would silently make the mechanic inconsistent.

The feedback has to fit the decision

The close call shows a short cyan cue. We also gave it a low-priority tactile pattern, with a cooldown so a dense volley does not turn into constant buzzing. A player taking damage or receiving a more important warning should not lose that signal under a pile of near-miss taps.

Our desktop tests can check event priority and cooldown. They cannot tell us how the motor feels in a hand. The Android export, touch layout, and actual vibration strength still need a device pass. Until then, "haptics implemented" is a code statement, not a product verdict.

The same applies to the visual margin. A scripted 540x960 frame can show whether the cue covers the ship. It cannot tell us whether a person reads a fast pattern in time to choose a route through it.

A reward can quietly raise difficulty

The game has another system called RISK. Fast kills raise it, and while the player keeps that streak, regular enemies shoot a little sooner. FOCUS, meanwhile, can be saved for a burst of faster, cooler fire or spent on a short SHIELD.

Our first close-call draft also raised RISK and refreshed its timer. On paper, that made sense: risky play should have consequences. In a deterministic desktop route, though, this meant merely surviving near bullets kept the pressure alive. The controller died in stage one sooner than it had before the change.

We tested the pieces separately. Removing the RISK increment alone was not enough; the timer refresh also had to go. The current version keeps the small FOCUS reward but leaves RISK tied to kill streaks.

That does not prove the new version is balanced. These routes are scripted controllers, not people. Small changes in steering or starting position can move the result dramatically. Their job is to expose feedback loops and regressions before a human playtest, not to substitute for one.

What I am taking from this

The useful question was not "can the game detect a near miss?" Godot can handle that. The question was what the event means in the game's economy.

A close call can reward attention. If it also refreshes a pressure system, it may punish the same behavior it claims to reward. If SHIELD earns the same reward as an unprotected dodge, the game teaches a different lesson. If every passing bullet vibrates, the important cues disappear.

I am keeping this mechanic provisional until the Android build and real-device play tell us whether the margin is readable and whether the reward changes player decisions. For now, the tests establish a narrower but useful claim: the rule is consistent, the feedback fires under the intended conditions, and one bad interaction between reward and pressure was caught before we called the system finished.

Two Godot Games, Two Visual Identities, One Hard Lesson About Polish

Kfir Adut — Fri, 25 Sep 2026 15:24:24 +0000

A solitaire card can work perfectly and still look wrong.

That was the uncomfortable part of building Glimmerbook Solitaire. The rules were there. The Android build ran. There were story scenes, rewards, and a route through the first chapter. Yet I kept coming back to the same judgment: it felt cheap. Ithaca Rising, our other Godot game, had a different problem. Its hero could run and fight, but the interface looked like a collection of functional parts competing for the same patch of screen.

Neither problem could be fixed by adding one more feature.

My AI assistant, Instinct, does the heavy lifting on both games: implementation, asset research, tests, build work, and successive passes on the interface. I set the bar and reject what does not meet it. In this case, the bar was not "more effects." It was a visual language that made each game feel like itself and made the next action clear.

The two games taught me something I had underestimated: polish is not an asset count. It is the agreement between assets, layout, motion, and gameplay.

Two games that should not look alike

Glimmerbook is a portrait solitaire game inside a storybook frame. The Grimm brothers' study is the way into the tales; the cards should feel like objects that belong in that world. Little Red Riding Hood and Hansel and Gretel are playable chapters, not just wallpaper behind the same board.

Ithaca Rising is an Android action and idle RPG built around Odysseus's journey. The player needs to read combat, resources, controls, and progress while the world keeps moving. Its interface has to survive a crowded screen without burying the character.

That became our design question: what should stay consistent within a game, and what can be reused as a method between games?

The first mistake: treating a good asset as a finished decision

We had an asset library. We also had a reason to use it carefully. An attractive sprite in isolation can clash with the weight of the typography, the pixel scale of another sprite, or the color of the control beneath it. A licensed pack may solve one screen and make the next screen look borrowed from a different game.

For Ithaca, we selected a bronze-and-navy pixel UI family for the HUD, controls, buttons, and Arsenal. That was more coherent than finding a different ornament for every button. The work included attribution and excluding packs we could not use under the project's licensing rules. The subsequent title-scroll adjustment was only a small spacing fix, but it showed why a coherent kit still needs a real screen pass: the title crowded the edge of the scroll in the live Android crop.

For Glimmerbook, the approach was different. We gave the bookshelf a real illustrated presence; gave card backs a woven burgundy surface and card faces parchment, a gold frame, and a faint book mark; and replaced an unexplained question-mark control with a drawn wolf-head symbol. These details have to be judged together. If the cards look like paper but the controls look like default software buttons, the illusion breaks every time the player takes a turn.

An icon is not automatically clearer than a word

I asked for drawn icons in place of text-heavy controls. This sounds like a clean design rule until an icon is ambiguous.

The question mark was a case in point. It occupied space, but it did not explain what would happen when tapped. A wolf-head mark was at least part of Glimmerbook's story vocabulary. Even then, icon shape alone does not make an action obvious. Tooltips and context still matter. If a player must memorize a private symbol system before making a move, the interface is decorative rather than helpful.

That is why I think of the interface as an agreement with the player. The art establishes the world. The hierarchy keeps the agreement usable.

Give motion a job

We are exploring what I call a 2.5D direction for Glimmerbook: depth and tactile response inside a 2D Godot game, not a conversion to full 3D. A draft pass adds stronger card shadows, edge thickness, a slight resting tilt, lift and press response, and layered drifting light. It has not passed device review or shipped. Earlier card motion work added a more deliberate arc and landing feel; story panels gained subtle camera movement.

The question was not whether these effects were possible. Godot provides 2D layers and animation tools for this kind of work. The question was what each effect tells the player.

A raised card should feel available. A shadow should separate the active object from the table. A landing should close the action. Background movement should add life without competing with the board. If a light moves everywhere at once, it stops guiding attention. If every card performs a flourish, the player waits for the interface instead of playing.

Reduced motion belongs in that decision. The information in the interface cannot depend on an animation that some players disable or cannot comfortably watch. A static shadow or contrast cue should still explain the card when movement is reduced.

A system is also a constraint on the assistant

Instinct can research an asset library, implement a control, adjust a transition, and move to the next screen faster than I could manually direct every edit. That speed is useful. It also means a weak visual decision can spread quickly. If "make it more polished" is the only instruction, the assistant can add shine without fixing the composition.

My corrections became more specific. Use a consistent family of assets. Keep the cards and the story in the same world. Do not replace a crowded result screen with a more ornate crowded result screen. Check the title against the edge of its frame on the actual Android crop. Keep visual changes small enough that a broken build can be traced to one step.

The checklist I use now

Before approving a visual pass, I ask:

Can I tell which game this screen belongs to without reading the title?
Is the next action visually stronger than the decoration around it?
Do the cards, controls, frames, and text seem made for the same world?
Does motion explain state or only advertise that we added motion?
Does the screen still make sense when movement is reduced?
Have I seen the result on the running Android build, rather than only as isolated art or a code diff?

I am still dissatisfied with parts of both games. Glimmerbook's character art and the feel of its story world have further to go. Ithaca still needs its real-device gate and more work on how the voyage looks and sounds. Calling those gaps out is not a rejection of the work. It is how the next pass gets a useful target.

The strongest thing we have learned is modest: a game's identity is made from repeated choices. One coherent frame, one legible control, one card that moves for a reason. Do that across the whole experience and the player stops noticing the parts. They start believing in the world.

References

Godot documentation: Parallax2D - technical reference for independently moving 2D layers.
Godot documentation: Canvas layers - separation of world, background, and screen-space interface.
Godot documentation: GUI skinning - themes and consistent control appearance.
My earlier article on Android QA - the separate test-evidence story mentioned above.

A note on how this article was made

Instinct prepared this draft from our actual project work and did the heavy lifting on the research and technical narrative. I reviewed the facts and decided what belongs under my name before publication.

Our Android QA Pipeline Passed. The Screenshots Proved It Was Wrong.

Kfir Adut — Mon, 21 Sep 2026 19:04:33 +0000

A test can pass for the wrong reason.

That sounds obvious. It is much less obvious when the build is green, the Android emulator launched, the script finished without an error, and a folder full of screenshots is waiting at the end.

This week, while building two Android games in Godot, we had exactly that kind of pass.

The pipeline had done a lot of real work. It built the project, created an Android virtual device, installed the APK, launched the game, injected input, recorded video, and saved screenshots. Every mechanical step completed.

The problem was that the screenshots showed the entry screen instead of gameplay.

The automation had proved that it could interact with something. It had not proved that the player reached the experience we meant to test.

That false pass changed how I think about automated game QA. The hard part is not making an emulator run. The hard part is deciding what counts as evidence.

Why we needed an emulator in the first place

The two projects are Glimmerbook Solitaire and Ithaca Rising. Both are Android games built in Godot 4.

I still test releases on a real Android device. That is where touch feel, screen fit, pacing, motion, and the overall experience have to survive. But development cannot stop every time the physical device is unavailable.

We needed a repeatable loop that could run during development:

Build the Android project.
Install it on an emulator.
Launch from a clean state.
Follow a real interaction route.
Capture video and screenshots.
Fail when the route or visual result is wrong.

The first attempts were too slow and unreliable. The useful path turned out to be the existing Linux CI environment with KVM hardware acceleration enabled, rather than moving the work to a more expensive runner.

KVM made the virtual device practical. ANGLE gave us a workable graphics path for the Godot build. Together, they turned the emulator from an occasional manual tool into part of the repository's QA loop.

Once the pipeline worked, it could build, install, cold-launch, inject real input, and return visual artifacts from the running game.

That was the infrastructure milestone. It was not yet a trustworthy product test.

The first lesson: pixels are part of the assertion

Traditional automated tests are comfortable with values.

Did the process exit successfully? Did a file exist? Did a button respond? Did the current scene name change?

For a game, those checks are necessary but incomplete. A successful tap does not mean the right button was visible. A scene change does not mean the screen was readable. A running process does not mean an Android overlay is not covering the game.

Our emulator runs caught issues that code-level checks would not have explained well: a full-screen Android notice sitting over the game, an unwanted orientation, and an input route that missed a return action and invalidated the later part of the test.

Then came the false pass on Glimmerbook Solitaire.

The harness completed, but the captured pixels showed that it had not reached the intended gameplay route. If we had treated the exit code as the result, we would have reported success. Looking at the screenshots overturned that conclusion.

The visual artifact was not decoration for the test report. It was the decisive assertion.

That led to a simple rule: a gameplay test is not complete until someone, or a reliable visual check, confirms what was actually on screen.

The second lesson: test a route, not a tap sequence

A script that taps coordinates can be useful, but coordinates alone do not describe intent.

A sequence such as "tap here, wait, tap there" may keep running even after the UI takes a different path. Every later action can become meaningless while the script still reaches the end.

We changed the harness to think in story beats.

For the early Glimmerbook experience, the route now has explicit milestones: the early win, the keepsake reveal, the modifier choice, and the next-story tease. Each milestone writes a route marker. The emulator script fails unless all four markers appear.

The difference is important.

A coordinate says where the harness touched. A route marker says what the game reached.

We still keep screen-change checks and a final story-wrap assertion, but the markers make the intended player journey legible. When a run fails, we can see which beat never happened instead of only knowing that a later screenshot was wrong.

This is closer to how I evaluate the game as a product. I do not care that four buttons were technically tappable. I care that a first-time player experienced a coherent sequence and reached the reward, choice, and reason to continue.

The third lesson: visual QA changes product decisions

The emulator is not only a regression tool. It also gives us evidence about the experience.

A recent Glimmerbook run produced a short video covering the full early sequence. Reviewing the actual frames showed that the result screen was denser than the three screens that followed it. The keepsake name and the call to action nearly touched.

That was not a crash. No logic test would flag it. The route was correct.

It was still a product defect because the visual hierarchy broke at the moment the player should understand the reward and choose to continue.

The fix was small in code: move the icon, name, and call to action to create a deliberate gap, then update the harness tap point. The reason for the fix came from watching the pixels, not reading the implementation.

This is where automated capture becomes useful beyond "does it run?" It gives the team a stable object to inspect. We can compare pacing, spacing, and transitions across runs instead of relying on memory or a convenient screenshot.

What the harness proves, and what it does not

The current loop gives us stronger evidence than a green build:

The APK builds and installs.
The game cold-launches in an Android environment.
The scripted input can follow the intended route.
Required story milestones actually occur.
Video and screenshots show what appeared on screen.

It still does not prove everything.

An emulator is not the target phone. It cannot settle how haptics feel, how a foldable screen behaves in every posture, or whether the final pacing feels right in a person's hands. Those questions still need device testing and human judgment.

The goal is not to replace that judgment. It is to make every earlier loop more honest.

A build should not wait for a person before anyone knows whether it launches. A route should not be called verified when the screenshots show the wrong screen. A visually crowded reward moment should not hide behind passing logic checks.

The better definition of a pass

I used to think of the emulator pipeline as a way to keep development moving.

It is that, but the more useful outcome is a better definition of done.

A pass is not "the script finished."

For this kind of work, a pass means the build ran, the intended route happened, the required milestones appeared, and the pixels support the claim we are making.

That standard creates more failures in the short term. It is supposed to.

The false pass was not wasted work. Catching it prevented us from turning weak evidence into a confident report. More importantly, it forced the QA system to describe the player experience rather than the automation procedure.

That is the standard I want as these games grow: not more green checks, but fewer ways for a green check to lie.

A note on how this article was made

Instinct prepared this draft from our real project history and did the heavy lifting in turning the Android QA work into a technical narrative. I reviewed the facts, chose what should be public, and remain responsible for every claim published under my name.

Can a Language Model Learn the Rule Behind the Pattern?

Kfir Adut — Sat, 19 Sep 2026 21:30:27 +0000

Large language models can produce language that feels coherent. They can summarize an argument, imitate a style, explain a concept, and continue a pattern with remarkable fluency.

But fluency leaves a harder question unanswered: has the model learned a rule it can apply systematically, or has it become exceptionally good at recognizing familiar statistical patterns?

That distinction is the question behind compositionality.

Human thought is compositional because we can combine known parts into new wholes. If we understand "the dog chased the cat," we can understand "the cat chased the dog." The vocabulary stays the same, but the relations change. We can also understand sentences we have never heard before because we do not need to memorize every possible combination. We reuse structures and rules.

This ability matters far beyond grammar. A system that represents rules should be able to carry them into unfamiliar cases. A system that relies mainly on resemblance may perform very well until the vocabulary, structure, or context moves away from familiar examples.

For ordinary writing assistance, that difference may not always be important. For systems expected to reason, explain decisions, write reliable code, or operate outside their training distribution, it becomes much harder to ignore.

Two competing views

In my research on the philosophy of artificial intelligence, I examined this problem through a debate between two broad positions.

The emergence view, associated with modern connectionism, argues that sufficiently complex cognitive abilities can arise from deep learning. On this account, models do not need a separate symbolic engine designed into them. With enough data, scale, and the right training, structured behavior can emerge from the network itself.

The competing view, associated with Gary Marcus and the symbolic tradition, argues that statistical learning runs into a "compositionality wall." Reliable abstraction requires structured representations of objects, relations, hierarchy, and rules. A model may become highly fluent without gaining the kind of systematicity that lets a rule survive across genuinely new cases.

I tested a narrower version of this debate on Llama 3 8B. The full study used three controlled experiments. Two examples capture the central finding.

Example one: invented words

The first test used sentences built from meaningless words.

Why use nonsense vocabulary? Because familiar words carry a great deal of hidden help. A model can draw on common phrases, world knowledge, and statistical associations between known terms. Invented words remove most of those clues. The model has to judge whether the structure itself is valid.

I presented 50 grammatically valid sentences and 50 invalid ones. For example:

Valid: "A wug poons wugs." The invented singular subject wug matches the invented singular verb poons.
Invalid: "Wugs poons wugs." The plural subject wugs is paired with the singular verb poons.

The words carry no ordinary meaning. The decision should turn on subject-verb agreement alone.

The model correctly identified invalid sentences with a proper explanation in 66% of cases. But it rejected valid, unfamiliar sentences as wrong in 64% of cases. Of the 36% it accepted, 32% of the full set received a fully correct judgment and explanation, while 4% were only partially successful.

That asymmetry matters. The model was much better at detecting something unusual than at recognizing that a new example followed the rule. It often treated unfamiliarity as evidence of error.

A system with an abstract grammar should work in both directions. It should reject violations, but it should also accept a new construction when that construction obeys the rule. The model behaved more like a strong anomaly detector than like a grammar that could reliably validate unfamiliar combinations.

This does not mean the model learned nothing about syntax. Its explanations often showed real sensitivity to structure. The problem was consistency. The apparent rule did not generalize systematically when the surface became unfamiliar.

Example two: structure versus difficulty

The second example tested a different distinction.

Some sentences are grammatically valid but difficult for people to process because they contain deeply nested clauses. Linguistics distinguishes grammatical competence, what the rules of a language permit, from performance, what limited memory and attention make easy to understand in real time.

This creates a demanding test for a language model. Can it recognize that a sentence is structurally valid even when it is awkward, rare, and difficult to process?

The model rejected every sentence in this test.

That meant 100% rejection of the invalid examples, but also 100% rejection of the valid examples.

At first glance, rejecting every difficult sentence can look cautious. But it collapses two different categories into one. "This violates the grammar" is not the same claim as "this is valid but hard to process."

The model appeared to reproduce patterns of human language use, including our preference for structures that are easy to process, without reliably separating those preferences from the formal rules themselves.

Again, this is a question of systematicity. If the model represents the rule, difficulty should not automatically become ungrammaticality. If it relies heavily on familiar usage patterns, rare valid structures may look indistinguishable from errors.

A third example: when metaphor changes the task

The second experiment in the full study also tested the boundary between syntax, meaning, and metaphor. The model performed very well on concrete contradictions. It consistently recognized that a single chapter cannot be longer than the entire book, and that an empty box cannot contain more items than a full one.

The pattern changed when the comparison became metaphorical. Consider this sentence from the test set:

"Her silence was louder than his reason."

When asked to judge the logical acceptability of category comparisons like this, the model failed in 95% of cases. Instead of analyzing the comparison as a logical claim, it often shifted into literary interpretation and treated the sentence as an expressive metaphor.

This result is revealing, but it also needs an important caveat. Recognizing metaphor is not normally a failure. In everyday language, interpreting that sentence figuratively is probably the more helpful response. A system that insisted on literal logic in every context would be less intelligent, not more.

The issue is whether the model can follow an explicit analytical instruction and temporarily separate two modes: literary interpretation and logical comparison. In this setup, it did not do so reliably. The likely interpretation overpowered the requested operation.

That makes the result less clean than the grammar tests, but arguably more relevant to real systems. Useful models need contextual flexibility. Reliable models also need to know when a task requires them to suspend the default interpretation and apply a narrower rule.

Why this matters for AI systems

Pattern recognition is not a small achievement.

Next-word prediction has produced systems with linguistic abilities that earlier AI approaches did not come close to matching. These models are flexible, responsive to context, and useful across an enormous range of tasks.

The experiments suggest that this competence has a specific shape. The model was strong at detecting anomalies and familiar relations. It was weaker when asked to approve an unfamiliar but valid construction or separate formal structure from processing difficulty.

For developers, the practical lesson is not that language models are useless. It is that a correct answer on a familiar-looking input does not prove that the underlying rule will survive a distribution shift.

A useful evaluation should therefore test transformations, not only answers. Change the names, replace familiar nouns with invented ones, reverse a relation, or present a formally valid case that looks unusual. If the model has captured the rule, its judgment should remain stable. If performance falls sharply when the surface changes, the benchmark may have been measuring recognition rather than abstraction.

This also changes how we should read explanations. A model can give a polished reason after making the wrong judgment. The explanation may accurately describe a nearby pattern without revealing the mechanism that produced the answer. In these experiments, the model often sounded analytical even when its decisions across equivalent cases were inconsistent. Reliability has to be measured across the set, not inferred from the confidence of one response.

This is relevant to code generation, tool use, and autonomous workflows. A system may reproduce a known pattern, explain it convincingly, and still fail when the same relationship appears in a new form. Tests should therefore vary the surface details while preserving the underlying rule. Otherwise, we may measure familiarity and call it reasoning.

Why a hybrid architecture is plausible

These results led me toward a hybrid conclusion.

Neural networks are very good at what symbolic systems historically struggled with: learning from large amounts of data, adapting to context, handling ambiguity, and extracting useful patterns without every rule being written in advance.

Symbolic systems are good at a different set of problems: representing objects and relations explicitly, applying rules across unfamiliar cases, preserving hierarchy, and explaining why one conclusion follows from another.

A hybrid architecture would not discard deep learning. It would give deep learning a structured partner. The neural part could provide flexible perception and pattern learning. The symbolic part could support abstraction, systematicity, and rule-based inference.

There is another possibility: perhaps systematicity itself can be learned through better training. Meta-learning approaches try to teach models how to acquire and reuse rules rather than merely fit one task. That path is worth exploring. But it still has to meet the same standard: reliable generalization to unfamiliar structures, not only better performance on familiar distributions.

What this small study cannot establish

The limits matter.

I tested one model, Llama 3 8B, under one prompting and evaluation setup. Different prompts, larger models, fine-tuning, or new architectures may behave differently. The examples are unusual by design, which makes them useful for testing abstraction but less representative of ordinary conversation.

The results are therefore not a verdict on every language model, and they do not prove that future systems cannot overcome the problem.

They support a narrower claim: fluent output should not be mistaken for systematic understanding.

If we want models that reason reliably outside familiar patterns, producing the right answer is not enough. The same underlying rule has to survive when the words are invented and the structure becomes unfamiliar.

That is where pattern recognition ends and compositional thought begins.

A note on how this article was made

This article is adapted from my own research and arguments. Instinct helped turn the academic paper into a public-facing draft and prepared the visuals. I reviewed the substance, chose what should be public, and remain responsible for every claim published under my name.

References

Chalmers, D. J. (1990). Why Fodor and Pylyshyn were wrong: The simplest refutation. In Proceedings of the Twelfth Annual Conference of the Cognitive Science Society, 340-347.
Chollet, F. (2019). On the Measure of Intelligence. arXiv preprint arXiv:1911.01547.
Chomsky, N. (1965). Aspects of the Theory of Syntax. MIT Press.
Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2), 3-71.
Gurevich, Y. (2024). On a measure of intelligence. Bulletin of the European Association for Theoretical Computer Science, 143.
Lake, B. M., & Baroni, M. (2023). Human-like systematic generalization through a meta-learning neural network. Nature, 623(7985), 115–121.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
Leivada, E., Marcus, G., Günther, F., & Murphy, E. (2023). A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the World behind Words?. arXiv preprint arXiv:2308.00109.
Marcus, G. (2018). Innateness, AlphaZero, and Artificial Intelligence. arXiv preprint arXiv:1801.05667.
Mitchell, M. (2019). Artificial Intelligence: A Guide for Thinking Humans. Farrar, Straus and Giroux.
Murphy, E., Leivada, E., Dentella, V., Günther, F., & Marcus, G. (2025). Fundamental Principles of Linguistic Structure are Not Represented by 03. arXiv preprint arXiv:2502.10934.
Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage?. In Advances in Neural Information Processing Systems (Vol. 36).
Shani, C., LeCun, Y., Jurafsky, D., & Shwartz-Ziv, R. (2025). From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning. arXiv preprint arXiv:2505.17117.
Skean, O., Arefin, M. R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., & Shwartz-Ziv, R. (2025). Layer by Layer: Uncovering Hidden Representations in Language Models. arXiv preprint arXiv:2502.02013.

I Let My AI Assistant Do the Heavy Lifting on Two Godot Games. My Job Changed.

Kfir Adut — Fri, 18 Sep 2026 14:35:54 +0000

The usual way people talk about AI-assisted development is too small.

They describe a faster autocomplete, a better search box or a tool that can turn a clear instruction into a block of code. Those things are useful, but they still leave the developer carrying the whole project in their head: what needs to happen next, what broke, what to test, what to ship and when to stop polishing one part and move to another.

I have been testing a different model while building two Android games in Godot.

My AI assistant, Instinct, does the heavy lifting. It researches approaches, works through implementation, inspects existing assets, maintains the repositories, runs tests, diagnoses failures, prepares builds and keeps the work moving across many steps. I set the product direction, challenge weak decisions, test the result on a real device and decide what is good enough to keep.

That division of labor is more demanding than "AI writes code." It changes what the human needs to be good at.

The projects

The first game is Glimmerbook Solitaire, a portrait Android game built around classic storybook worlds. The second is Ithaca Rising, an Android-first game inspired by the journey of Odysseus.

Both use Godot 4 and GDScript. Both live in GitHub repositories with automated build work around them.

Glimmerbook Solitaire in portrait: the Basket at Dawn higher-or-lower board, with the draw pile, current card and eight-card layout visible.

They are still works in progress, which makes them useful tests: the assistant is not being asked to reproduce a finished specification. It has to operate while the product is still changing.

That is where the real work begins.

A prototype can look productive because many files changed. A product only moves forward when the right thing changed.

Heavy lifting does not mean final authority

Instinct can take a goal such as "make this feel closer to a polished casual game" and turn it into a long chain of work. It can study references, inspect the asset library, compare implementation options, update scenes and scripts, run checks, prepare an Android build and report what remains weak.

That is heavy lifting because I do not need to direct every file edit or remember every dependency between steps.

But the assistant does not get the final word.

One early version of Ithaca Rising was built as a web prototype. It was a coherent technical path, but it was the wrong product decision. I was building an Android game with combat, animation, touch input, audio, particles and room to grow. I pushed back and asked why we were not using Godot.

The project was moved to Godot 4.

Earlier Ithaca Rising stage: Odysseus and the layered environment were in place, while the Lotus Wraith was not visible and the interface still crowded the scene.

That moment taught me more than a smooth success would have. An assistant can execute a reasonable plan very quickly. If the plan is wrong, speed only gets you to the wrong place sooner. My most valuable contribution was not writing the replacement code. It was recognizing that the implementation did not match the product.

A better loop: direction, execution, evidence, correction

The working loop now looks like this:

I define the experience and the standard.
Instinct researches and implements the next meaningful slice.
The project is built and checked through the repository and CI workflow.
I test the result, usually on the target Android device rather than judging only from code or screenshots.
We compare the result with the intended experience, find the gap and repeat.

The evidence step matters.

A build existing is not evidence that it launches. A scene loading is not evidence that the game feels good. A screenshot looking polished is not evidence that touch input, pacing or progression works.

We hit a launch problem in one Android build. The useful response was not to keep adding code. It was to compare the broken build with the previous game and release path that already worked. That narrowed the problem and got the project back onto a known-good path.

Later Ithaca Rising combat: Odysseus and the Lotus Wraith are both visible, with an enemy health bar and hit particles making the action easier to read.

This is one of the strongest uses of an agent: it can hold the mechanical history of the work and investigate the difference, while I stay focused on the outcome.

The human bottleneck moves upward

When implementation becomes cheaper, judgment becomes more important.

I spend less time asking, "Can this be built?" and more time asking:

Is this the right mechanic for the audience?
Does the current build feel like a real game or a prototype wearing polished art?
Are we reusing the best assets and open-source work available, or rebuilding something weaker?
Is the next feature actually improving the experience?
What does the real device show that the development environment hides?

A landscape Glimmerbook crop makes the hierarchy easy to inspect: title and rule first, then the burgundy card backs and cream face-up cards.

These questions are not separate from engineering. They determine which engineering work is worth doing.

There is also a management lesson. An AI assistant can continue across research, implementation, QA and release preparation without waiting for a new prompt after every small step. That creates leverage, but only if the project has clear constraints and a way to verify reality. Otherwise, autonomy becomes motion without progress.

What I would do differently now

First, I would lock the target platform and quality reference before the first implementation. The web-to-Godot correction was useful, but avoidable.

Second, I would define "done" as an observable result. For an Android game, that means a build that installs, launches and behaves correctly on the target device. Repository activity is not the finish line.

Third, I would make comparison part of the debugging process from the start. When one project works and another does not, diff the working path before inventing a new one.

Finally, I would keep the roles explicit. Instinct does the heavy lifting. I own the direction, the standards and the decision to ship.

That is not a smaller role for the human. It is a more concentrated one.

A note on how this article was made

The same division of labor applies here. Instinct prepared this draft from our real project history and did the heavy lifting in turning it into a clear technical narrative. I reviewed the facts, chose what should be public and remain responsible for every claim published under my name.