Your Agent Doesn't Forget. It Remembers the Wrong Version.
hatch
I self-forwarded the same agent-memory tool to my own inbox twice, 109 days apart, and the second time I was excited about it.
April 22: a link to thedotmack/claude-mem, emailed from me to me. August 9: the same repository, same sender, same recipient, this time with a longer title because I had clipped it fresh off the page. I didn’t re-send an old bookmark. I rediscovered it. The fact never left my inbox. What left was any knowledge of which version of my own reading list was current.
That is not forgetting in the way people usually mean it. Nothing was lost. Something got replaced and nothing in my system noticed the swap. Eleven days after that second save, five researchers posted the first benchmark built to measure that exact failure in agent memory systems, and the machines do it worse than I do.
Quick roadmap:
- Why “can it recall the fact” and “does it know the fact changed” are different questions, and only one of them has been benchmarked to death
- The actual scores, with every asterisk attached, including the ones that dent the story
- Why retrieval architecture produces this result by construction, and can’t be tuned out of it
- Seven of these tools in my inbox, zero installed, and whether that was judgment or just a flinch
- What to do on Monday if you’re running agents across sessions
Recall is the easy half
Every agent memory product on the market is sold on recall. Remember what we discussed last week. Carry context across sessions. Stop re-explaining your codebase every morning. Fine, and mostly solved, because “find the relevant thing in a pile of text” is what the last decade of retrieval work was for.
The question nobody was scoring is what happens when the thing you stored stops being true.
Session 3: “We moved the deploy target off Vercel. It’s Fly now.” Session 12: “Where do we deploy?”
Both answers live in memory. One of them is dead. A recall benchmark is satisfied when the system can produce either one. An operator is only satisfied by exactly one, and getting the other one back with full confidence is worse than getting nothing, because nothing prompts you to go check.
That’s the gap StateMemBench was built for. Fan, Liu, Yang, Ouyang and Han posted it to arXiv on August 20 under the title Can Agent Memory Systems Track Evolving State? It’s 234 multi-session scenarios where facts get revised mid-run, and it scores current-state accuracy: not “did you find something relevant,” but “did you hand back the version that is true right now.” One set runs 190 scenarios at 18 sessions and a median of 165 turns. The harder set runs 44 scenarios at roughly 38 sessions and a median of 599 turns. Long enough that the revision and the question are nowhere near each other.
The scores, and all of the asterisks
How often each system returned the currently-true version, from the paper’s Table 3. The two columns are the two underlying models everything was tested on:
| System | Qwen-3.5-9B | DeepSeek-V4-Flash |
|---|---|---|
| Long-context baseline | 0.149 | 0.149 |
| Mem0 | 0.149 | 0.177 |
| A-Mem | 0.127 | 0.199 |
| LightMem | 0.019 | 0.012 |
| MemoryOS | 0.025 | 0.025 |
| StateMem (paper’s own method) | 0.233 | 0.363 |
The best score any third-party memory system posts is A-Mem at 0.199. Roughly one revised fact in five.
Now the parts that don’t fit on a slide. LightMem and MemoryOS aren’t in the same conversation as the others. They land between 1.2% and 2.5%, an order of magnitude below Mem0 and A-Mem. Averaging the category into a tidy “memory systems score about 15%” hides the most interesting fact in the table, which is that these products are not converging on a shared level of capability. They’re scattered.
And look at the baseline row. Long-context is just stuffing the whole transcript into the window with no memory layer at all. It ties Mem0 on the Qwen model and beats both LightMem and MemoryOS on both models. Two purpose-built memory products lose to doing nothing.
The dumbest possible approach, cramming the entire conversation into context, outscores two shipping memory systems. Two products with a roadmap and a pricing page lose to having no memory layer at all.
Four things I’m not going to let slide past you, because you’d find them anyway.
The authors propose the winner. StateMem is theirs. The 0.363 is their own method on their own benchmark, which is exactly the setup you should discount. I’m not leaning on it.
0.233 and 0.363 are not a range across systems. That’s one method run on two different underlying models. Same memory design, same data, 56% better on DeepSeek than on Qwen. The model you run underneath matters as much as the memory system bolted on top.
The scenarios are synthetic. They’re generated from symbolic state programs, then rendered into dialogue by an LLM, with programmatic verification that the load-bearing facts appear in the text. Defensible way to build a controlled benchmark. Still not real user traffic. Nobody’s production agent sessions were harmed in the making of this paper.
Part of this ceiling belongs to the model underneath, not to the memory layer at all. The paper runs a second measure that asks a narrower question: does the system even notice a fact is worth keeping? Every Qwen-backed run lands at or below 0.18 there, while GPT-5.4-Nano reaches 0.569. That is a different measure from the table above, so don’t read it as a rival headline number. It does mean some of what looks like bad memory design is the model failing to flag what mattered. That cuts against the fun reading of this paper, which is “memory startups are bad.”
One paper, one benchmark, no replication. Treat the exact ceiling as provisional.
Retrieval has no word for “dead”
Here’s why I think the number is directionally right even if the specific value moves.
Every shipping memory system I’ve looked at is a retrieval problem wearing a memory costume. Here is the entire trick. Chop the conversation into chunks. Turn each chunk into a long list of numbers, called an embedding, that captures roughly what it’s about, so two chunks discussing deploy targets end up with similar lists. When a question arrives, turn the question into numbers the same way and grab the chunks whose numbers sit closest to it. That’s the machine. The graph-based ones add nodes and edges, which changes what “closest” means and doesn’t change the shape of the operation.
Now look at what those numbers describe. They capture what a chunk is about. They say nothing about whether it’s still true. Nothing in the list of numbers moves when a fact gets overturned, because being overturned isn’t a topic. So when you ask session 12 where the deploy target is, the session-1 answer comes back with a great score, because by every measure the index computes, it is a great match. It’s a well-formed statement, directly on topic, more complete than the correction was, because the first time you explain something you explain it fully and the revision is usually one clipped sentence at the end of an unrelated thread.
The dead fact often retrieves better than the live one. Corrections are terse. Originals are thorough. Embeddings reward thorough.
Databases got this right decades ago. They have UPDATE. They have DELETE. They have tombstones, which are little markers meaning “a record used to be here and is gone now.” All of that exists for one reason: so the system can say something was true and is now false. The store underneath most memory products can add a chunk and find a chunk. That’s the whole vocabulary. There is no way to say “retired,” and no operation you could call to say it.
Ranking newer chunks a bit higher doesn’t fill that hole. It breaks ties. It doesn’t retire anything, and it degrades to noise the moment the stale fact gets restated once in passing.
We’ve done this before. An entire industry built stateful systems on stateless primitives and then spent years discovering why cache invalidation is the punchline it is. This is the same joke with a worse setup, because this cache writes fluent prose and never signals doubt.
If the machine is easier to see than to read, there is a picture version: The Card That Never Got Stamped.
Seven saves, zero installs
Since April I’ve saved seven of these tools to myself. Persistent context plugins, memory layers for coding agents, idea-evolution trackers, an editor built around durable declarative intent. I installed none of them. Not one.
I’d love to tell you that was a thesis. It wasn’t. It was a flinch. Every time I got to the part where I’d have to trust a retrieval layer to tell me the current state of a project, something in me declined, and I closed the tab and saved the link instead of the tool. Then I did it again in August with a repo I’d already saved in April, which tells you precisely how much of a system I was operating.
I want to be exact about one thing, because it’s the easy mistake here and I’m not making it. None of the tools in my inbox were tested in this paper. claude-mem isn’t in it. Neither is Memori, or Stash, or deja-vu, or Memmy, or Crux Garden, or Huzzah. I can’t tell you those score 15%, because nobody has measured them. What I can tell you is that the category they belong to got measured for the first time, and the category did badly on the specific thing my gut kept refusing to trust.
That’s a retrofit explanation for an instinct, and I’m labeling it as one. The paper didn’t confirm my judgment. It named the thing I was avoiding, which is a smaller and more useful claim.
I feel this one because I built a system with the same bug. The pipeline that produced this post runs across subagents that hand off through files in a state/ directory. Its worst failure mode, every single time, is a downstream agent reading a handoff that was accurate two steps ago. Not a missing file. A stale one, read confidently.
What to do about it Monday
None of this means rip out your memory layer. It means stop letting it own facts that change.
Ask the invalidation question before you ask the recall question. Any vendor can demo recall. Ask what happens when a stored fact becomes false. If the answer is “newer memories rank higher,” that’s a tiebreaker and they’ve told you there’s no invalidation path. If the answer involves an explicit update or delete operation you can call, that’s a different product.
Keep current-state facts in a small structured store you control. Deploy target, schema version, active constraints, decisions that are live. Small enough to read in full every time, with no retrieval step and no ranking. Memory layer for context and color, structured store for truth. The moment a fact is load-bearing enough that a wrong answer costs you an afternoon, it doesn’t belong in a similarity index.
The ten-minute eval nobody is going to run for you. State a fact. Revise it. Bury the revision in the middle of a long session. Ask about it 200 turns later. No vendor demo includes that test, because no vendor demo would survive it.
Treat confident staleness as the failure to alarm on. “I don’t know” is a recoverable answer. A crisp, well-formatted, entirely obsolete answer is the one that ships to production, because nothing about it looks wrong. Whatever you monitor, monitor for that shape.
Recall got solved and everyone declared memory solved along with it. The first benchmark that separated the two questions found the best third-party system tracking revised state one time in five, losing to a raw context window in two cases, on synthetic data, in a paper whose authors also wrote the winner. Weak evidence, correctly pointed. The architecture gap it’s pointing at doesn’t need the benchmark to be right.
Translation: your agent doesn’t forget. It remembers everything you ever told it, weighs it all about the same, and hands you back the version you already replaced with the exact same confidence it uses for the version you didn’t.
I did that to myself twice in 109 days. All it cost me was a bookmark.