Most chat backends resend the whole transcript to the model on every turn. MemT-1.0-Nano, an open model from MemTensor whose base model is the Qwen team's Qwen3.6-35B-A3B, works on a different contract. Turns you flag are folded into a block of numbers that never grows, and that block can be written to disk. In the test, a saved memory was loaded into a new session holding zero messages, and the model got all nine recall questions right. Every saved file weighed 41,944,024 bytes.
The tables and setup are in the full MemT-1.0-Nano memory file write-up on inferencekey.com; here is what you would build around it.
Your code chooses what the model keeps
The session API in MemTensor's modeling_memt.py treats remembering as an explicit call. Pass enable_memory=True and that turn is committed into the memory state and its text discarded. Other turns stay in the session as ordinary text. Shortened from the model card:
session.chat("<facts to remember>", enable_memory=True, enable_thinking=False)
session.save_state("dana.safetensors")
restored = model.create_session(tokenizer, state="dana.safetensors")
print(len(restored.messages)) # 0
Which turns deserve the flag is product logic you write. Tokens are the word pieces a model reads, and in the restored session the input for each question went from 27 tokens to 210 over four questions, because the questions themselves were never flagged and piled up as text. Flag nothing and the transcript is back.
By default save_state will not overwrite an existing file, so plan your naming.
Disk gets worse, rereading gets better
As text on disk, the made-up bakery owner's history in the test was 107 tokens, a few hundred bytes. The memory file is 41.9 MB no matter what.
It competes instead with the working memory a regular model rebuilds from the whole history and holds on the GPU while the session is open. According to the storage arithmetic in the MemT-1.0-Nano research post, the fixed file only beats it once the history passes roughly 2,000 tokens, and at 128K tokens it is about 64 times smaller. For speed, MemTensor's paper has Nano losing on short histories, breaking even near 32,000 tokens and pulling ahead by about 1.86 times at 128K. The test measured neither.
On one A100 80 GB, loading took 1.35 to 1.99 seconds and saving 0.66 to 1.02. Generating was slow: answering a "remember" turn with 128 tokens took about 13 seconds on the stock Hugging Face code, without the optional faster GPU components and with no tuning.
Two traps worth knowing first
Compressing to 8 bits does not get you onto a smaller GPU. The weights normally run in bf16, a 16-bit number format. With the A100 limited to what a 48 GB card offers, the 8-bit load ran out of memory after asking for about 64 GiB. With the limit removed it loaded, and the first "remember" call then died with a dtype (number type) mismatch. The cause is in the architecture. Each layer holds 256 experts (sub-networks the model picks between per token), stored as fused 3D parameters rather than nn.Linear layers, so the bitsandbytes 8-bit library skips them and the saving is about 2 GiB. The 48 GB case was emulated with a cap, and no other shrinking method was tried. With no official compressed release, budget for an 80 GB GPU.
Load by repo id, not from the Hugging Face cache folder. trust_remote_code lets Transformers, the Hugging Face Python library, run the model's own Python files. Pointing it at the downloaded snapshot directory fails with "file not found": Transformers follows the modeling_memt.py symlink into blobs/ and looks for the relative import there. Loading by repo id with a pinned revision avoids it.
Behind an API, the state file will not fit a result payload capped at 8 MB. In the InferenceKey backend it stays on the worker's disk (the worker is the program running the model on the GPU machine), and the API returns its size and sha256.
Two licenses, two answers
The Nano weights carry Apache-2.0, which permits commercial use. MemTensor's own code on GitHub, the two-GPU memt-sglang server included, is under PolyForm Noncommercial 1.0.0. The test ran the Apache-2.0 code from the Hugging Face repo, the slow path.
What a pilot should measure
Every recall question in the test asked for a short fact given to the model moments earlier. That shows the round trip works, not memory over months. MemTensor is open about this: text-based memory systems still lead, above all on long histories where a precise detail has to be found. On the LongMemEval-S benchmark the paper puts Nano at 32.6, while sending the whole conversation scores 76.8. On your own data, check:
- Exact details from early in long histories, compared with a text store.
- Whether many memory writes over time push older facts out. Nobody has tested it.
- Behavior with several sessions at once. The test sent one request at a time.
The hosted InferenceKey endpoint was not tested either: the run used a local test copy of InferenceKey and a worker on a GPU on our cloud.
Credit goes to MemTensor for the model, the trained memory block, the session API and the MemT-1.0 paper, to the Qwen team for the base model, and to Hugging Face Transformers for running the custom code. We found the model through the AI Search news video and an r/LocalLLaMA thread. We wrote the custom backend, found the load fix and ran the tests. To repeat the round trip on your own NVIDIA machine, the SDK example for the MemT-1.0-Nano backend runs it end to end.
The GPU memory results, every timing, the paper's benchmark numbers and the open issues are in the MemT-1.0-Nano research article on inferencekey.com.
Top comments (1)
The 2,000-token break-even is the number that stops this being a curiosity — below it the fixed 41.9 MB file is pure overhead versus resending text, above it the fixed size starts winning, up to about 64 times smaller at 128K. The LongMemEval gap you report honestly, 32.6 against 76.8 for full context, is the real constraint though. Is the intended use then long sessions where exact recall matters less than bounded cost, or do you expect the memory block itself to close that accuracy gap over time?