DEV Community: sakiha6720

I Taught a Tiny Open-Source Language Model to Speak Halloween — Meet the Haunted Frog 🐸🎃

sakiha6720 — Wed, 28 Oct 2026 19:26:47 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

The Haunted Frog is a tiny language model that lives in your terminal. You teach it Halloween, and then it whispers back.

🐸 THE GHOST IS LAUGHING
🐸 THE NIGHT IS HUNGRY
🐸 THE OWL SAYS WHO
🐸 THE GHOST
🐸 THE ZOMBIE IS FLICKERING
🐸 THE FOG IS HUNGRY

Those are six consecutive samples (seeds 1003-1008), not a script and not hand-picked. The frog learned 60 short Halloween phrases ("THE FOG IS ROLLING IN", "THE WITCH IS LAUGHING", "I AM HUNGRY"...) and recombined them. Some outputs reproduce a training line (THE OWL SAYS WHO is one), but "the zombie is flickering" and "the fog is hungry" appear nowhere in its training data.

The "ghost in the machine" here is literal and small. The model is CSE-Frog, an open-source next-symbol model I made earlier this year as a dissection specimen for learning how language models work. It is not a Transformer. It has no attention and no gradient descent, it looks back only about two symbols, and you can read the whole engine in one sitting. For this challenge I built a brand-new app around it that trains it, haunts a terminal with it, and measures honestly what it learned.

Demo

A real, unedited run. Every 🐸 line is model output, and the seed is printed next to it. Run python -m haunted_frog haunt --count 6 --seed 1003 and you get exactly these lines. The dim text ("Something small is breathing inside the machine...") is hand-written narration and is drawn without the frog, so you can always tell which is which.

Code

kagioneko / the-haunted-frog

A tiny open-source language model (CSE-Frog) learns Halloween and haunts your terminal. Hacktoberfest 2026 Week 4.

The Haunted Frog 🐸🎃

A tiny language model learns Halloween.

A tiny frog lives inside your terminal. Teach it Halloween words and it starts whispering strange things back.

The frog is CSE-Frog, a small open-source next-symbol model that you can read end to end It is not an LLM. It has no attention and no gradient descent, and it only looks back about two symbols This project trains it on a 60-line Halloween corpus, lets it speak in a "haunted" terminal, and measures what learning actually changed.

Everything runs locally on a CPU. There are no API keys, no cloud calls and no downloads after pip install.

Recorded with python -m haunted_frog haunt --count 6 --seed 1003 (asciinema → agg, raw cast in docs/media/haunt.cast). Every 🐸 line is model output. Run the same command to get the same lines.

A second real run, as plain text:

…

View on GitHub

git clone https://github.com/kagioneko/the-haunted-frog && cd the-haunted-frog
python -m venv .venv && source .venv/bin/activate && pip install -e .
python -m haunted_frog train      # ~1 second on a CPU
python -m haunted_frog haunt      # Ctrl+C to leave
python -m haunted_frog evaluate   # the experiment below

No GPU, no API key, no network after install.

How I Built It

1. A corpus small enough to read. data/halloween.txt has 60 lines that I wrote for this project, deliberately sharing words (THE ... IS ..., I AM ..., ... IN THE ...) so the frog has joints to recombine at. You can edit it and retrain in a second.

2. Training. CSE-Frog learns by strengthening links between symbols that follow each other (Hebbian-style), plus a memory of the last pair of symbols. Training is one call: Frog(top_k_edges=16).learn(sequences, epochs=5).

The first bug was a good one. I passed the lines as a flat list of strings, and CSE-Frog read the whole list as one sequence of 60 giant symbols. The frog recited the corpus as one run-on sentence:

HAPPY HALLOWEENTRICK OR TREATGHOST IN THE MACHINETHE GHOST IS HEREI AM A FROG

The fix is a list of lists, one sequence per line, so the frog learns where a phrase ends. There is now a test for it.

3. Generation. frog.generate(prefix, n=max_len, seed=s) samples one symbol at a time until the frog picks <END>. My wrapper never edits, filters or re-ranks the output. An empty answer is printed as (silence).

4. The Haunted Terminal. Green frog, orange banner, typewriter effect, a line every few seconds, and a --no-halloween switch for plain output. Each line uses seed base+i, so a whole spooky session can be replayed.

5. An experiment instead of a vibe. python -m haunted_frog evaluate trains 10 frogs and scores each one on the same 100 seeds.

Frog	Halloween words	Unique outputs	Mean length
Untrained `Frog()`	silent (100% empty)	—	0
Control: same-shaped corpus of everyday words	0%	59%	3.6 words
Halloween, word-level, 1 epoch	18%	81%	4.2
Halloween, word-level, 5 epochs	15%	57%	3.7
Halloween, word-level, 20 epochs	19%	38%	3.0
Halloween, 5 epochs, CSE-Frog default `top_k_edges=3`	15%	20%	3.1
Halloween, character-level, 5 epochs	0%	92%	9.4

What worked

Training is what makes it speak at all, and the corpus decides what it says. The control frog never said a Halloween word. The Halloween frog uses one in every 5-7 words.
It recombines. With the settings I ship, 83-94% of word-level outputs are not exact lines from the corpus. Here are the first ten seeds of the 1-epoch frog, unpicked: COLD WIND AT MIDNIGHT · COLD WIND AT MIDNIGHT · A SPIDER IN THE WINDOW · I AM HUNGRY · THE NIGHT IS QUIET · I WANT A WITCH ON A GHOST IN THE WIRES · THE MACHINE · SOMETHING IS HUNGRY · A FROG · THE WITCH IS AWAKE

What failed

It is obsessed with "THE". THE starts 38% of the corpus lines, but 99% of the outputs after 5 or 20 epochs (49% after 1 epoch). Its Halloween-word rate (15-19%) stays below the corpus's own 22.6%, because it over-produces THE and IS.
More training made it duller. Unique outputs fell from 81% (1 epoch) to 38% (20 epochs). Asking the frog for its probabilities showed why: the chance that a line starts with THE goes from 0.435 after 1 epoch to 0.998 after 20. The strongest links keep getting stronger, the temperature stays fixed, and the softmax saturates.
The default settings were too narrow. At CSE-Frog's default top_k_edges=3, only 20% of outputs were unique. I raised it to 16 after looking at seeds 0-39, and then scored everything on seeds 1000-1099 so the choice wasn't graded on the samples that picked it.
The character-level frog almost never formed a Halloween word. In 200 outputs at 5 and 20 epochs it formed none, and at 1 epoch it formed exactly one. It keeps circling the same little loop:

  THE INGHE INGHE IN IN IN WIS INDYS ING     (5 epochs)
  THE IN IN IN ING                           (5 epochs)
  HE S L THE AUNIGIC                         (1 epoch)

My hypothesis is that with only about two characters of context, THE, IN, IS and -ING become attractors it can't escape. I haven't isolated that experimentally yet (that needs a variable context length), but it is the most vivid picture I've seen of why real language models need long context.

Full method, all ten conditions, unedited samples and threats to validity: docs/EXPERIMENT.md.

Why Does Open Innovation Matter?

With a closed model, you can only watch the ghost. With an open one, you can open the machine and find out why it says what it says.

Because the engine is open, small and inspectable, I could look straight at the numbers behind these behaviours. CSE-Frog can print the score breakdown behind any single prediction (frog.show(...)). When the 20-epoch frog got stuck on THE GHOST, I didn't have to guess: scripts/diagnose.py shows THE at 0.998 at the start of a line, and almost all of GHOST's score coming from one direct link. You can run the same script and see the same numbers.

Open also means anyone can repeat this. The corpus is a text file, the seeds are printed on screen, and the model file is plain JSON. python -m haunted_frog evaluate rebuilds the results table in about a minute on a single CPU core, and python scripts/diagnose.py prints the probabilities and score breakdown quoted above. If you think I cherry-picked, run them and check.

And open lets small things exist. A 60-line corpus and a frog-sized model aren't a product. They're a specimen you can dissect. I think more people would understand language models if they could start with one that fits in their head.

The Spirit of It

A ghost in the machine is something that seems alive but turns out to be mechanism. The frog is that idea at a scale you can hold: it says THE FOG IS HUNGRY, and you can look inside and see exactly which links made it say so.

Honest notes

CSE-Frog itself predates this challenge (I published it on PyPI on 2026-10-04, MIT). It is used unmodified as a PyPI dependency. Everything in the-haunted-frog, including the corpus, app, terminal, experiment and tests, was built during Week 4.
I wrote the code with an AI coding assistant (Claude Code). Every number above comes from running the repo.
This is a toy. It does not understand English or Halloween. It recombines short phrases, and the experiment above shows where that breaks.

Context Pointer OS: Reconstructing Memory for Long-Running AI Agents

sakiha6720 — Sat, 23 May 2026 20:49:29 +0000

Context Pointer OS: Reconstructing Memory for Long-Running AI Agents

This is a submission for the Hermes Agent Challenge

What I Built

Modern AI agents are becoming increasingly persistent.

They no longer operate as simple request/response systems.
They maintain memories, tools, workflows, user preferences, execution histories, and long-running task states.

However, most current agent systems still rely on a fundamentally expensive pattern:

Reloading large amounts of context repeatedly.

This creates several problems:

Context window overload
Token explosion
Retrieval duplication
Memory drift
Long-session instability
Difficulty auditing historical reasoning

To explore an alternative approach, I built an experimental architecture concept called:

Context Pointer OS

Instead of forcing agents to continuously carry entire histories, Context Pointer OS introduces a lightweight symbolic routing layer where agents primarily exchange:

references
summaries
retrieval rules
compressed memory pointers

rather than full raw context.

The goal is not “infinite memory.”

The goal is:

reconstructing memory only when necessary.

This project explores whether long-running AI systems may scale more effectively through context reconstruction instead of continuously increasing context size.

⸻

Core Idea

Traditional agent memory systems often work like this:

Agent
└── repeatedly reloads massive context

Context Pointer OS instead proposes:

Agent
└── pointer layer
└── selective retrieval
└── context reconstruction

The system separates:

Active Context
Archived Context
Retrieval Routing
Symbolic References
Reconstruction Rules

This creates a lightweight “operating layer” between the agent and long-term memory systems.

⸻

Example Pointer Structure

A pointer may contain metadata such as:

{
"pointer_id": "mem_042",
"context_type": "security_incident",
"summary": "Suspicious VPS login event",
"priority": 0.92,
"source": "incident_logs",
"retrieval_rule": "only_if_security_context",
"expiration": null
}

Instead of injecting entire logs into every interaction, the agent only retrieves them when required.

⸻

Why This Matters

As agent systems become:

autonomous
collaborative
long-running
multi-session
memory-driven

context management itself becomes an architectural problem.

Simply increasing context windows may not be sufficient.

Large-scale agent ecosystems may eventually require:

memory routing
symbolic compression
retrieval governance
context lifecycle management
auditability layers

This project explores one possible direction toward that future.

⸻

Demo

Architecture Diagram

GitHub Repository

https://github.com/kagioneko/context-pointer-os

⸻

Code

Repository:

https://github.com/kagioneko/context-pointer-os

Example conceptual routing logic:

if pointer.priority > threshold:
reconstructed_context = retrieve(pointer)
agent.inject(reconstructed_context)

⸻

My Tech Stack

This project is primarily architectural and experimental.

Current components include:

Python
JSON-based memory structures
Pointer-based routing concepts
External memory design
Agent runtime experimentation
Context reconstruction workflows

Additional conceptual influences include:

long-running AI agents
external cognition systems
symbolic memory routing
retrieval-aware architectures

⸻

How I Used Hermes Agent

Hermes Agent strongly influenced the direction of this project.

What stood out to me was the idea of agents that:

improve over time
retain operational continuity
search previous interactions
maintain evolving memory structures

This raised an important architectural question:

How should agents manage memory once they become persistent?

Context Pointer OS was developed as an experimental response to that question.

Instead of treating memory as a continuously loaded block of text, the system explores whether agents could operate through:

contextual pointers
symbolic references
selective reconstruction
memory routing layers

Hermes Agent helped frame the importance of long-running agent infrastructure — not only better models, but better memory operating systems around them.

⸻

Final Thoughts

This is not a finished framework.

It is an ongoing experiment exploring how future AI agents may organize memory, context, and continuity.

Maybe future AI systems won’t scale through larger context windows alone.

Maybe they will scale by learning how to point.

Google I/O 2026 Wasn’t About AI Models — It Was About Agent Execution Layers

sakiha6720 — Sat, 23 May 2026 20:45:28 +0000

title: Google I/O 2026 Wasn’t About AI Models — It Was About Agent Execution Layers
published: true
tags: ai, googleio, agents, architecture

This is a submission for the Google I/O Writing Challenge

Google I/O 2026 Wasn’t About AI Models — It Was About Agent Execution Layers

Most discussions around Google I/O 2026 focused on model capabilities.

Gemini got smarter.
AI Studio improved.
Agent workflows became easier.
On-device AI became more practical.

But I think the real shift happened somewhere deeper.

Google I/O 2026 was not just about better AI models.

It was about the emergence of an Agent Execution Layer.

And once you start building multi-agent systems in the real world, you quickly discover something uncomfortable:

The hardest problem is no longer intelligence.

It is state management.

⸻

The Problem Nobody Talks About

When developers first build AI systems, the architecture usually looks simple:

User -> LLM -> Response

But the moment you move into agent workflows, everything changes.

Now you suddenly have:

multiple agents
tool execution
memory systems
long context histories
role switching
state inheritance
retrieval pipelines
security boundaries
autonomous actions

And eventually, the architecture becomes something closer to:

User
↓
Coordinator Agent
↓
Execution Agents
↓
Memory Layer
↓
Tool Runtime
↓
External APIs / Environment

At this point, prompts stop being “messages.”

They become something closer to an operating system.

⸻

Context Is Becoming the New Bottleneck

Most people still think model performance is the primary scaling problem.

I don’t think that’s true anymore.

The bigger problem is this:

Context grows faster than reasoning quality.

The more capable agents become, the more memory, instructions, logs, and coordination data they accumulate.

This creates several failure modes:

context bloat
instruction conflicts
memory drift
role collapse
hidden prompt inheritance
prompt injection propagation
state contamination between agents

Ironically, smarter agents amplify orchestration problems.

This is where I think the next generation of AI infrastructure will emerge.

⸻

From “Prompt Engineering” to “State Engineering”

For the last two years, the industry focused heavily on prompt engineering.

But prompt engineering assumes something important:

That interaction is temporary.

Agent systems break this assumption.

Agents persist.
Agents inherit memory.
Agents maintain roles.
Agents accumulate behavioral state over time.

That means the problem changes from:

"What should the AI say?"

to:

"What state should the AI exist in?"

This is a fundamentally different design philosophy.

⸻

Building Around the Problem

Over the past year, I started building several experimental concepts around this issue while working on multi-agent workflows, memory systems, and autonomous orchestration experiments.

Some examples:

Context Pointer OS

Instead of continuously passing gigantic raw histories into models, agents should reference contextual structures through lightweight pointers.

In other words:

Don't pass the entire world.
Pass references to the world.

This reduces token waste while making long-term coordination more stable.

Project:
https://github.com/kagioneko/context-pointer-os

⸻

AI Instruction Tape (AIT)

Human language is extremely expensive for agent-to-agent communication.

AIT experiments with compressed instruction transfer between AI systems.

Instead of repeatedly sending huge natural language prompts, agents exchange compact operational context.

Project:
https://github.com/kagioneko/ai-instruction-tape

⸻

Esoteric AI Protocol (EAP)

As multi-agent ecosystems grow, natural language alone becomes inefficient as an execution protocol.

EAP explores lightweight structured communication for agent coordination.

Project:
https://github.com/kagioneko/esoteric-ai-protocol

⸻

Google I/O 2026 Confirmed Something Important

What Google showed this year was not just AI tooling.

It was the beginning of infrastructure for persistent AI execution.

The moment agents become:

autonomous
stateful
collaborative
tool-connected
environment-aware

the industry stops being purely about model quality.

It becomes about:

orchestration
memory integrity
state synchronization
execution governance
agent operating systems

In other words:

The future of AI is not just model architecture.
It is runtime architecture.

⸻

The Security Side Is Going to Matter More Than People Think

One thing I learned from real-world VPS incidents and autonomous agent experiments:

The more authority agents gain, the more dangerous context corruption becomes.

A compromised context is effectively a compromised execution environment.

This means future AI systems will likely require:

memory validation
state auditing
execution boundaries
agent isolation
instruction provenance
behavioral monitoring

AI security may gradually evolve into something closer to operating system security.

And honestly, I think we are still very early.

⸻

Final Thoughts

Google I/O 2026 felt like a transition point.

Not because AI suddenly became intelligent.

But because the ecosystem started shifting from:

AI as conversation

to:

AI as infrastructure

And once that happens, developers will need new abstractions.

Not just better prompts.

But:

state layers
memory architectures
execution runtimes
agent protocols
orchestration operating systems

I think that’s where the next major wave of AI development is heading.