I Gave a CAPTCHA Memory Instead of Making the Model Bigger

shiva mani — Tue, 29 Sep 2026 16:10:33 +0000

A CAPTCHA normally gets one chance to decide whether an interaction is human.

That bothered me.

If someone interacts with a CAPTCHA once, the system has some behavioral evidence. But when the same kind of interaction happens again, most CAPTCHA systems don't really remember what happened before. Every attempt starts from zero.

So while working on SwipeCHA, I tried a different approach: keep the existing behavioral classifier, but give the security system memory.

That is where Hindsight came in.

The part that was already working

SwipeCHA is a behavioral CAPTCHA. Instead of asking users to identify objects or type distorted characters, it asks them to slide a handle across a track.

While the slider is being moved, the browser records pointer movement and timing information. Those events are converted into ten features:

avg_mouse_speed
mouse_path_entropy
click_delay
task_completion_time
idle_time
micro_jitter_variance
acceleration_curve
curvature_variance
overshoot_correction_ratio
timing_entropy

Those features go to the existing Random Forest classifier.

That distinction matters.

I didn't start by assuming the model needed to become an LLM agent. The classifier already has a clear job: evaluate the current behavioral signal.

So I kept it.

The problem I wanted to solve was everything around that single prediction.

The problem with starting from zero

The original security path was essentially:

Swipe
↓
Feature extraction
↓
Hard-rule checks
↓
Random Forest
↓
Human / Bot

Every verification was mostly an isolated event.

Suppose the first interaction looks human. The next interaction also looks human. A third one is slightly unusual, but not unusual enough to trigger the hard rules.

Looking at only the third interaction gives the system one piece of evidence.

Looking at all three gives it a pattern.

I wanted the second kind of system.

Why I didn't replace the Random Forest

The obvious architecture would have been:

CAPTCHA → LLM → Human / Bot

I rejected that approach.

The behavioral features are numerical, the existing model is trained specifically for them, and throwing that away would make the system harder to reason about rather than better.

Instead, I split the responsibilities:

Random Forest — evaluates the current behavioral signal.

Hindsight — provides relevant historical security experiences.

Security Agent — interprets the current evidence in that historical context.

Decision Policy — turns those signals into a controlled action.

That separation became the central design of the project.

Memory should store experiences, not mouse movements

There was another decision I had to make before integrating memory.

I could have stored every pointer event.

I didn't want to.

A raw swipe can contain a large number of coordinates and timestamps. Putting all of that into long-term memory would give the memory layer a lot of detail without necessarily giving it useful security context.

Instead, the system retains a distilled security experience.

A simplified example looks like this:

{
"prediction": "Human",
"confidence": 0.98,
"risk_level": "low",
"reason_codes": ["ZERO_HISTORY_BASELINE"],
"recommended_action": "allow"
}

The actual retained information depends on the verification path. The important idea is that the memory represents what happened from a security perspective, rather than becoming a replayable dump of the browser session.

The first interaction is supposed to be boring

On the first interaction there is no history.

So the system should not pretend there is.

The first turn is evaluated using the current behavioral evidence:

Turn 1
Hindsight memories: 0

Random Forest:
Human — 0.98

Reason:
ZERO_HISTORY_BASELINE

Decision:
Allow

Then the experience is retained.

That gives the next interaction something to retrieve.

The interesting part happens on the next turn

On a later verification, Hindsight can recall relevant previous experiences before the Security Agent evaluates the current result.

Now the agent has two kinds of information:

Current interaction
+
Previous security experiences
↓
Contextual assessment

For a consistent human interaction pattern, historical evidence can reinforce the current assessment. The implementation records this kind of context with reason codes such as HISTORICAL_HUMAN_CONSISTENCY.

For suspicious or borderline behavior, previous experiences can provide context that a single swipe cannot.

The important point is subtle:

The Random Forest didn't suddenly become a better classifier. The decision became contextual.

I tested whether the memory was actually persistent

This was one of the first things I wanted to verify.

If the application restarts and all the previous experiences disappear, then the system isn't demonstrating the kind of persistent memory I was trying to build.

So I ran the interaction sequence, restarted the application, and ran it again.

The observed sequence was:

Turn 1 → 0 memories → RETAIN
Restart
Turn 2 → historical memory recalled → RETAIN
Turn 3 → previous experiences recalled

That was a useful checkpoint because it separated two ideas that are easy to confuse: state in a running process and persistent experience memory.

The current implementation uses the official Hindsight client with a local Hindsight-compatible deployment for the development/staging environment. I am deliberately not presenting a live Hindsight Cloud deployment as verified.

The agent doesn't get unlimited authority

I also didn't want the Security Agent to become a black box sitting in front of the CAPTCHA.

It produces structured context:

risk level
historical context
reason codes
recommended action

The final result still passes through a decision policy.

That gives the system controlled actions such as:

Allow
Block
Challenge Again

There is also a deterministic hard-rule path for obvious automation. If the interaction clearly matches that pattern, the system can reject it without making memory retrieval part of every decision.

That keeps the memory layer useful without making it a single point of failure.

The part I underestimated: deciding what to remember

My first instinct was to think that adding memory would be the difficult part.

It wasn't.

The harder question was: what is actually worth remembering?

If I retained everything, the system would have more data but not necessarily more useful context.

That changed the integration.

I started thinking about memory in terms of security experiences rather than raw telemetry:

What happened?
What did the current classifier say?
Was there historical context?
What risk did the agent identify?
What action followed?

That made Hindsight part of the reasoning loop instead of just another storage service.

Failure handling is part of the design

A security check shouldn't stop working because its memory dependency has a bad day.

So the integration has a fallback path.

If Hindsight or the agent layer is unavailable or times out, the system can fall back to the existing Random Forest path.

The integration also includes a circuit breaker so repeated memory-service failures don't keep adding latency to verification.

This wasn't the most visually impressive part of the project, but it was one of the parts I cared about most once the architecture started looking like a real security system rather than a demo.

What the final flow looks like

The resulting pipeline is:

User
↓
SwipeCHA
↓
Behavioral features
├── Hard-rule gate
↓
Random Forest
↓
Hindsight Recall
↓
Security Agent
↓
Decision Policy
├── Allow
├── Block
└── Challenge Again
↓
Hindsight Retain

The Random Forest still evaluates the live interaction.

Hindsight adds historical context.

The Security Agent interprets that context.

The decision policy keeps the final behavior controlled.

That division is more important to me than simply saying “the CAPTCHA now has AI memory.”

What I would not claim

There are a few things I don't want to overstate.

Behavioral signals are not identity.

Historical consistency is not proof that an interaction is legitimate.

Memory doesn't magically make the underlying classifier perfect.

And the current environment demonstrates Hindsight integration and persistent retain/recall using a local Hindsight-compatible deployment; I would not describe that as a verified production Hindsight Cloud deployment.

Being precise about those boundaries makes the system easier to trust.

What actually changed

The most interesting result wasn't a new model accuracy number.

It was a change in the question the system can ask.

Without memory:

“What does this swipe look like?”

With memory:

“What does this swipe look like, and does it fit the security experiences I've already seen?”

That is a small architectural change, but it changes the type of system you can build.

I started with a CAPTCHA that evaluated interactions.

I ended with a security pipeline that can retain experiences, recall relevant history, and use that history as context for later decisions.

The lesson I took away is pretty simple:

Sometimes the next useful improvement isn't a bigger model. It's giving the existing system something worth remembering.

DEV Community: shiva mani

I Gave a CAPTCHA Memory Instead of Making the Model Bigger