DEV Community: Hayato Kamiya

Forgetting is for remembering: my AI chat broke long before the token limit

Hayato Kamiya — Sun, 06 Sep 2026 03:45:54 +0000

I say "I can't remember" a lot. But when I look at what is actually going on inside me, that one phrase turns out to hold several different things.

"I can't remember" is not one state

In my experience there are three.

The first: nothing catches. This is less "I can't remember" than a state where I cannot even feel that anything is there. There is no target to try to remember.

The second: something catches, but it is too heavy to come up. I know there is something related to what is in front of me; it is only the detail I do not have. It is far away, or heavy, and my hand does not reach it. In a sense, this is already remembering.

The third: when I try to pull it up, it gets pushed back. Here the problem is not reach. My hand is on it, and something is working to keep it from coming up. Somewhere in my experience there is something that contradicts that memory, and it silently refuses to let the two surface together. The conscious part of me wants to haul it up, and the part that does not want to is the one that is moving.

The first has no target. The second is out of reach. The third is within reach and is being held down. Lined up they look alike, but they are completely different things.

Seen through how an AI behaves

You send an AI a prompt and something comes back. That much is a given. The interesting part, I think, is how it behaves when what comes back is not quite right. Let me line that up against the three above.

What corresponds to the second, "something catches but it doesn't come up." When there seems to be something related but the substance does not come out, the AI does not stop. It fills the gap with something nearby, or raises the level of abstraction and returns a generality. Some of what gets called hallucination looks to me like this state.

But people do the same thing. There are more than a few people who cannot say "I don't remember" and bluff instead. In filling the gap rather than stopping, people and AI are very much alike.

What corresponds to the third, "pushed back when you try to pull it up." This is a state where a safety lock is engaged, in people and in AI alike. In a person, it was built unconsciously out of experience: something like a trauma, for example, holding the memory down. In an AI, it was put there from outside, during training. The difference is who set the lock: you, or someone outside.

And yet the way it comes off is the same. In both cases it does not come off from the front; it can come off through something like a back door, an unexpected stimulus. In people, as a flashback. In AI, as prompt injection. Not only is the reason for stopping similar; the shape of how it comes off is the same too.

Then what about the first, "nothing catches"? This is the one that does not line up.

Yes, no, null

Let me arrange the same three in different words. Yes, no, null.

"Yes" is the state where something can be pulled up. The second case, heavy and not coming up, is on this side. That there is something related has already come back; only the detail is heavy.

"No" is the state where what exists is the information that it is not there. The third case, being pushed back, is this one. That case is not "it must be there." It is "I would rather it were not." And if the information "it is not there" exists, then, turned around, that is also a trace of something that once was.

"Null" is the state where nothing is defined at all. The first case is on this side.

And there are two kinds of null. One is null from the start: I never experienced it, I do not know it. The other is something that used to be a yes or a no, and became null.

The second kind is forgetting. The second and third cases were about remembering. They were never forgetting.

The awkward part is that from the inside, I cannot tell these two nulls apart. Whether I forgot, or never knew, I cannot say.

"No" and "null" also look alike from the inside. But they are not the same. Holding the information "it is not there" affects what you do before and after. It does not produce the same result as having nothing at all. Even if you cannot tell them apart from inside, the behavior that comes out is different.

So what about AI?

When nothing catches for an AI, either it never had it, or the thread changed and everything was wiped. The former is null from the start. In the latter, the entity itself has been swapped out. Either way, there is no transition in which a yes or a no becomes null inside the same entity.

That is why it did not line up. The AI we have today does not have this function of forgetting.

Before the limit, there was a different limit

If there is no function for forgetting, then the more exchanges you have, the more it should keep carrying. I ended up seeing what actually happens while building something myself.

I build a service called AWAI, which visualizes the flow of thinking from conversations, and it has a chat feature where you can talk with an AI.

When I first built that chat, I did it the way the Gemini API tutorial does. Send the text of every past exchange, every time. Each time, the AI reads the whole history before it responds. In support-center terms, it is as if the staff read the entire case history from the beginning before every single reply to the customer.

When the conversation got long, it broke down.

There was plenty of room under the maximum token count. Even so, once the conversation passed about 20 turns, the instructions in the system prompt started getting vague, its grasp of the conversation got vague, and it started behaving oddly.

This was not about the limit. Separate from the maximum token count, there is an effective amount it can handle while keeping quality, and I think we had crossed that. Put the other way around, as long as you stay inside that range, continuing the exchange is not a problem in itself. A growing history is not the bad thing.

So I reduced the amount of text sent to the API. With that, both the system prompt's instructions and the content of the conversation were grasped properly again.

Reduce it, and it got better.

What I saw from this was that with Gemini 3.1, precision saturates first, at roughly the amount of information in a 20-turn conversation. It is not the limit that caps it; precision does. In that case, select, and stay within that range.

(The turn count is a feel from the implementation at the time; change the model or the length of the prompts and it should move.)

And here a question comes up. So how do the AIs out in the world do their selecting?

Probably everyone is doing this

From here on, this is not something I have verified. It is what I sense from using them. I would honestly rather write it as fact, but I have the impression that this is an area every AI has been built not to answer, whichever one you ask. So I write it as a guess.

My guess is that they are doing the same thing. The reason they do not break down over many turns is that some selection is happening inside.

Early ChatGPT broke down as soon as the conversation got a little long. Was that because this selection was not there yet? Seen that way, it adds up.

If so, the meaning of the growing token counts changes too. That was not the acquisition of forgetting. They became able to select, conversation by conversation, from what had accumulated inside, and so a long conversation now holds. What grew was the ability to choose what to pass along each time, not the ability to forget.

That is the view: AI did not become able to forget. It extended the length over which it can get by without forgetting.

Selection is not forgetting

What is the difference between selection and forgetting?

Selection means choosing again every time. Everything is still there, and you choose the portion to send this time. So what was not chosen has not disappeared, and it may be chosen on the next turn. Turned around, the candidates keep growing. The cost of choosing does not go down.

Forgetting means the candidates themselves get fewer. Once something has come off the list of what can be pulled up, you no longer go looking for it. There is less to search, so the scan finishes sooner. The content is unchanged, and pulling gets lighter.

If you keep everything pullable, then no matter how capable the system, the search space grows until you cannot get a realistic speed out of it. So you need an operation that reduces what can be pulled up in the first place. Forgetting, I think, may be that operation.

Seen this way, what happened earlier means something different too. Reducing what I sent made it grasp things again. That is remembering because I reduced. Selection itself is not forgetting, but the act of reducing led to the result of remembering.

Forgetting is for remembering.

And finally, to be honest about it: AWAI's AI chat has no mechanism for forgetting either.

What it does is this. From the information in past conversation, it pulls out what is close in meaning to what is being discussed now, and hands that to the AI. Alongside that, for each piece of information, it calculates a priority every time, "how much should this be kept," and the high ones get a mark that says "this matters."

So there is a mechanism for choosing. But there is no mechanism that reduces what can be pulled up.

There is a mechanism that decides what should be kept, and only the mechanism for forgetting is missing. That is because I have not been able to think of how forgetting should be built. Everything I have written here about AI is also, as it stands, about my own tool.

What decides what gets forgotten?

I have got as far as forgetting leading to remembering. Then what decides what gets forgotten?

My guess is that it is decided on the side of instinct. Stress, events that left a strong mark, how I feel that day. Somewhere the conscious mind cannot do anything about, what gets forgotten is shifting.

Thinking that way, I notice an odd asymmetry.

Taking things in can be done consciously, to a degree. You memorize for an exam. On a trip, you look at a view thinking, I want to keep this one. What you meant to keep does stay, more or less. (Of course, plenty of things stay on their own, out of habit, without any intention to keep them.)

But forgetting does not run on the conscious mind. Try to forget and you cannot. If anything, you remember it more.

And then: recalling does not actually run on the conscious mind either. I feel as though I am pulling things out, but in practice they come out on their own when the conditions line up. The three cases I started with were all "it did not come out" rather than "I could not pull it," and I think that is why.

Only the way in, where things enter memory, is, just barely, on my side. The way out, and the way things get called up, I cannot operate myself.

That, I suspect, is what makes it hard to put on an AI. If what decides what gets forgotten is the condition of a living creature, things like stress and how you feel that day, then an AI that has none of that has no basis to build the criterion from.

I do have one candidate for something to use instead. Anything that takes too long to retrieve gets judged "did not come back, so it is not there." The second case I started with, "something catches but it is too heavy to come up," looks like exactly that. It does not disappear, though; if a similar stimulus comes along, it comes out again.

If this worked, what would change? Probably you would no longer need to cut threads. You could keep going without breaking it up.

And there is one more thing I think about.

Because you forget, you can remember. And what you remember becomes your next action. I wrote earlier that holding the information "it is not there" affects what you do before and after. If so, then what you forgot comes out directly as a difference in what you do.

The accumulation of that may be what makes a person who they are, or an AI what it is. Not what you remember. What you forgot.

But what decides that, I still do not know.

What I found after writing this

I wrote this without reading any literature, from my own sense of it alone. When I went looking afterwards, several things turned out to be close. I leave them here for anyone who wants to go deeper.

Schacter, The seven sins of memory (American Psychologist, 1999) — Sorts memory's failures into seven kinds and argues they are not design flaws but by-products of otherwise adaptive features. The second case in section 1, "something catches but it is too heavy to come up," is what this taxonomy calls blocking, the tip-of-the-tongue state.
Bjork & Bjork, A new theory of disuse (1992) — Splits the strength of a memory in two: how well it is learned (storage strength) and how accessible it is (retrieval strength). The "yes" in section 3 reads as high storage strength with low retrieval strength.
Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023) — Measures how language model performance drops sharply just from moving the relevant information to a different position in a long context. Close to section 4, "before the limit, there was a different limit."
Chroma, Context Rot: How Increasing Input Tokens Impacts LLM Performance (2025) — Tests 18 models and finds their performance grows unreliable, unevenly, as the input gets longer. Also section 4.
Anthropic, Effective context engineering for AI agents (2025) — The idea of treating context as a finite resource. It describes compaction: summarizing a conversation that is nearing the limit and starting a fresh window from the summary. A published method, aimed at people building agents, in the area I wrote about as a guess in section 5.
Richards & Frankland, The Persistence and Transience of Memory (Neuron, 2017) — Argues that the purpose of memory is not to keep information accurate over time but to make decisions better, and that forgetting is a function serving that. The closest thing to section 6, "forgetting is for remembering."
Borges's story "Funes el memorioso" (1942) and Luria's case study "The Mind of a Mnemonist" (1968) — Both about a person who cannot forget. There is also a paper (Dementia & Neuropsychologia, 2018) that reads the two side by side. This is the "keep everything pullable" side of section 6.
Anderson & Green, Suppressing unwanted memories by executive control (Nature, 2001) — An experiment showing that repeatedly keeping yourself from recalling something makes it harder to recall later. It gets harder to recall; it does not become null. So this looks less like forgetting than like a lock put on a memory after the fact, the safety-lock side of section 2.

What set this off was a service called AWAI. It visualizes the flow of thinking from conversations, and the chat in sections 4 and 6 lives inside it. It still has no mechanism for forgetting.

I measured Gemini Live Translate in 3 languages: same rhythm, 2x the characters

Hayato Kamiya — Fri, 04 Sep 2026 06:46:22 +0000

I build a meeting tool where people who speak different languages talk to each other directly. You speak Japanese, the other person hears English; they speak English, you hear Japanese. The speech translation runs on Google's Gemini 3.5 Live Translate (gemini-3.5-live-translate-preview).

Here is a demo with three people in Japanese, English and Spanish. No signup.

I wanted to add live subtitles to that screen. You listen to the translated audio, and you read what the other person actually said in their own words, before translation. Something like putting original-language subtitles on a dubbed film. Not a transcript that appears after they finish talking: the Pixel Recorder kind, where words show up one or two at a time while they are still speaking.

That gave me a hard requirement I could not design around: if the delay is five seconds, the feature is useless.

Subtitles exist so you can react, in the moment, to what was just said. A subtitle for something said five seconds ago arrives after the conversation has already moved on. Under a second, and you can glance at the panel and keep up. Somewhere between those two numbers is the line between a feature and a decoration.

One word before going further, because it runs through the whole post. The transcript does not come back in one piece: it arrives in small instalments, and each instalment is a chunk. Subtitles are those chunks, joined in the order they turn up.

So the question was simple. How fast does Live Translate actually emit chunks, and how big is each one?

I could not answer it from anything I had.

When I built the prototype, there was no reason to measure this

A few months earlier I had built a prototype on Gemini Live (gemini-3.1-flash-live-preview). The question then was whether you could feed it human speech directly and have the model recognize it as text at all. It could, and it did that well.

Later I settled on Live Translate for making human conversation legible. In principle the two should behave alike, so the prototype code looked reusable — but how much of it carried over was an open question. So I rebuilt on top of it with Live Translate.

When I got to the subtitles, I needed latency numbers. I went back to the prototype for them, and they were not there.

The other side had changed.

Gemini Live is a voice version of an AI chat. You speak, the model answers, one turn each. The AI waits for you to finish. It does not matter how long you take; the conversation does not break. So time was never a variable in the prototype. "Speech goes in, text comes out" was the whole question, and there was no reason to record when each chunk arrived.

Subtitles are not like that. The other side is a person, and people do not wait. If the text does not land while they are still speaking, it is worth nothing — so time itself becomes the requirement.

Same model family, similar-looking API, and the requirements swap out completely depending on whether the thing on the other side is an AI or a human. The prototype answered "does this work." What I needed here was "does this arrive in time." Different job.

Making time measurable

To make the subtitles arrive in time, I first had to know how long they were taking. And to measure "how many seconds after speech begins", I have to know when speech began — on my side of the wire.

So I added VAD (voice activity detection): a step that watches the incoming audio locally and catches the moment a voice starts. When it fires, I record that timestamp and tell the API "starting now".

Gemini Live Translate can also work out where speech begins and ends on its own. But then the start of the clock sits on its side, and there is no way to say what the seconds are counted from. Bringing the time reference over to my side is the whole reason the VAD is there.

The check runs every 32 milliseconds, so the gap to the true onset stays inside that. Against the one second or so that a subtitle is allowed, that is fine enough.

After that it is just logging. The start-of-speech signal, each audio send, each transcript chunk coming back, all with timestamps, on a monotonic clock so nothing shifts under you if the system time is adjusted.

The measuring script pulls the connection settings, the endpoint and the model straight out of the code we actually run, so what goes over the wire is byte-identical to a real session. A measurement of a lookalike would prove nothing about production.

Results

Japanese, English and Arabic. I picked Arabic deliberately: it is written right-to-left, and its script is far from both the Latin alphabet and Japanese.

	Japanese	English	Arabic
Audio length	90s	100s	100s
Speech segments	19	22	21
Transcript chunks	67	73	75
Characters per chunk (median)	6	13	9
Same (mean / p90 / max)	6.0 / 10 / 13	12.0 / 20 / 23	9.0 / 14 / 22
Inter-chunk interval (median / p90)	1.023s / 1.267s	1.049s / 1.399s	1.038s / 1.272s
Speech onset → first chunk (median)	0.64s	0.80s	0.57s
Same (p90 / max)	1.03s / 2.88s	1.24s / 4.40s	1.13s / 2.23s
Arrived during speech / after	63 / 4	67 / 6	71 / 4

The three bold rows are the ones that mattered for building subtitles.

Characters per chunk — how much text arrives at once. This is the only row that differs by 2× (6 / 13 / 9)
Inter-chunk interval — how long until the next piece of text. Near-identical across all three (1.023 / 1.049 / 1.038s)
Speech onset → first chunk — how long before the first character shows up. Also close (0.64 / 0.80 / 0.57s)

So the cadence and the head start are the same in every language, and only the amount of text per delivery changes.

That difference is really about how much a single character carries. Since the interval is almost exactly a second, the per-second count is the same figure: about 6 characters in Japanese, 9 in Arabic, 12 in English. Turned around, that is roughly 170 / 110 / 85 milliseconds of screen time per character. One Japanese character occupies about as long as two English ones.

The chunk counts (67 / 73 / 75) differ because the clips were 90 and 100 seconds long. Per second they come out at 0.74 / 0.73 / 0.75.

For what it is worth, I ran Japanese twice in independent sessions: median onset 0.66s then 0.64s, interval 1.009s then 1.023s. Run-to-run spread is around ±0.05s.

The rhythm does not depend on the language

Three languages, three scripts, and the interval lands on the same second. That does not look like a coincidence.

The server is not emitting per word. It is emitting per unit of time. It appears to flush roughly once a second with whatever it recognized in that second. That means "one or two words at a time" is a property you get in any language. What changes is how many characters fit in a second.

Onset latency is language-independent too: 0.57s to 0.80s at the median, 1.03s to 1.24s at p90. The outliers (2.2s to 4.4s) were all the first utterance of a session. Warm-up. Every subsequent utterance landed in the 0.4–1.2s band. It recurs once per reconnection, which is worth knowing if your session drops.

So why does that number — characters per second — differ by 2×?

This part is speculation. I suspect the rate at which people exchange information by voice settles in roughly the same place whoever is speaking. The processing happens on the human side, and that side does not change with the language. If so, however differently a language chooses to express things, the amount it can carry per second stays put: a script that packs more into each character needs fewer of them, one that packs less needs more, and the same content goes across either way.

There is research that measured something close to this. A 2019 study of 17 languages across 9 families, with 170 native speakers, found that speech rate (syllables per second) varies a lot between languages, but multiply it by the information density of a syllable and every language lands around 39 bits per second (SD 5.10). Languages with less dense syllables talk faster to make up for it.

That study does not include Arabic, and I measured characters on a screen rather than bits or syllables, so the two do not stack directly. Still, "the interval and the head start line up, and only the characters per second differ" fits that shape of explanation.

Two more things I checked, because they change how the receiving side has to be built:

The stream is append-only. I brought up the Pixel Recorder at the top, and one thing it does is rewrite what it already showed you: as recognition catches up, it goes back and corrects an earlier guess.

If Live Translate did the same, subtitles would have to be built on the assumption that text already on screen can move. That changes the whole display, so it was worth checking first.

I looked for rewrites with a deliberately strict test: does any chunk start with three or more characters matching the tail of what I already have? Two hits across all runs, one in Japanese and one in English. Both turned out to be at the same point in the audio, where the speaker restarted a phrase, and the model transcribed the restart faithfully. Not a protocol resend: a real repetition, reproducing at the same position across four independent sessions. Zero corrections. What arrives simply stacks up in order; nothing behind reaches forward to change what is already there. You join the pieces as they come and never look back.

Chunks arrive while the person is still talking. This one decides whether the feature is possible at all.

If the text only landed once an utterance was over, subtitles could only ever be finished sentences appearing in blocks — the opposite of what I wanted. And in this audio the longest single utterance ran 14 seconds. Wait for the end and the panel sits empty for those 14 seconds, by which point the conversation has moved on.

It does not wait. Over 90% of the text was on screen while the person was still speaking. The rest turns up just after they finish.

Only the on-screen text varies by language

And it was never only the character count. How many chunks the same speech breaks into, and where those breaks fall, both change with the language.

Here is what actually arrived:

English   'Anyway, the true'  /  ' ability of the'  /  ' lovely ghostwriter'
Arabic    'طيب، عشان كده يا جماعه'  /  ' قدرات الكاتب'
Japanese  'ラブリーゴースト'  /  'ライターの真の'

English and Arabic broke at word boundaries, and every continuation carried a leading space (look at the gap in front of ' ability of the'). Join them and that space is your word separator.

Japanese did not. The single word "ラブリーゴーストライター" (lovely ghostwriter) is split into 'ラブリーゴースト' and 'ライターの真の' — cut in the middle of the word, with nothing to signal that it is mid-word.

Two consequences for the client:

Concatenate verbatim. Do not trim, do not insert separators. Trimming the leading space welds English and Arabic words together. The Japanese mid-word split means a word will sit visibly incomplete for about a second, and there is nothing you can do about that.
Switch text direction per bubble. Subtitles go out the way a messaging app looks: one utterance, one bubble. What comes back is the speaker's own language, so in a multilingual meeting a right-to-left bubble and a left-to-right one sit in the same panel. Pick a direction for the whole panel and one of them breaks, so the UI has to decide bubble by bubble (in HTML, dir="auto" on each one).

So the UI limit cannot be a character count

Here is the rule I was about to write: cap the subtitle panel at N characters, drop the oldest.

With a 2× spread in characters per second, that cap buys a different amount of time in every language. Cap at 60 characters and Japanese keeps 10 seconds on screen while English keeps 5. Tune N until Japanese feels right and English scrolls away before you can read it.

Cap by rendered lines instead, and the difference cancels out:

	Characters per second	Characters per line (mobile)	Characters in 3 lines	3 lines ≈
Japanese	6.0 (full-width)	~20	60	~10s
English	12.0	~40	120	~10s
Arabic	9.0	~35	105	~12s

Capping by lines is, of course, still capping by "characters per line × lines". The difference is that you are not the one choosing the characters per line.

Chunks arrive with no line breaks in them. Where the text wraps is decided by the width of the panel and the font — and the panel is the same width in every language, because it is the width of the screen. Fill it with full-width characters and a line holds about 20; fill it with half-width ones and it holds about 40. The character count per line moves by exactly as much as the characters are wide.

That turned out to move in step with the characters-per-second figure. So three lines hold 60 characters in one language and 120 in another, and in time they both come out at 10–12 seconds. The line count was doing the normalization for me.

If the horizontal side follows the font, so should the vertical. Reserve the height of those three lines in font-relative units, not pixels. Arabic, Devanagari and Thai have taller line boxes, and a px-fixed box clips them.

(Three lines rather than three messages, incidentally, because that 14-second utterance came to about 84 characters, which overflows three lines on its own.)

The three-line panel is running here if you want to see whether the reasoning holds up in practice. Turn subtitles on in the toolbar; they are off by default.

What this does not cover

Of the three recordings I measured, only the Japanese one is an actual person speaking.

For English and Arabic I had no audio to work with, so I took the translated speech the model hands back during a Japanese run and fed that in as the input. Which means those two languages are the model talking to itself: clean, no accent, no disfluency, no one talking fast. Real speakers will chunk differently, and the model card itself notes that accents degrade language detection.

Separately from this measurement, I have confirmed that English speech from a real person is transcribed properly. But that is as far as it goes — I did not measure chunk granularity or latency on it. For every other language I have not even checked that much.

I have not measured Chinese, Korean, Thai, or Hindi. The one-second period looks structural rather than tuned, so I would expect it to hold, but that is a guess and I would rather label it as one.

And this is one speaker at a time. What happens when several people hold separate sessions concurrently is a different measurement.

Where this ended up

The tool is called AWAI: people in different languages sit in the same meeting and talk. The subtitles I measured here are on that screen.

And the demo I have linked a few times is not a mock-up made for this post. The code drawing those subtitles is exactly the production code; the only difference is that the data comes from a recorded replay instead of a live channel. Open a real meeting and the same thing runs.

Subtitles, though, only exist during the meeting. What AWAI is actually for is what is left afterwards: the conversation sorted by topic, with who said what on which subject. If subtitles are "what did they just say", that is "what did we actually talk about". The 0.64 seconds I measured here is one instant at the front of it.

Building for more than one language usually gets framed as absorbing the differences between them. Measured, the differences and the sameness fell into two clean piles.

The text on screen differs — the character count, where it breaks, how tall a line is. The rhythm it arrives in does not. And that seems to be less about Gemini than about people exchanging information by voice at roughly the same rate whatever language they use. The writing system varies; the speaking and listening underneath it does not.

That is the part that stuck with me while building a multilingual meeting tool. The thing that matters when two people talk to each other appears to survive the change of language.

And I only saw it because I wired the translation up myself and measured what came back, one piece at a time. Using it would not have shown me that. If you are building on a streaming speech API, measure it for your own use rather than carrying over what you checked before — what you verified with an AI on the other end goes back up for verification the moment you put it between two people.