Preprint · not peer-reviewed
Retrieval Is Not Memory
Everyone ships a vector database and calls it memory. But memory is not a retrieval function over embeddings — it is a governance function over experience. This is the part of memory that is not retrieval.
The barrier to shipping a system marketed as "AI memory" has collapsed to a weekend: embed some text, store the vectors, return the top-k nearest neighbors of a query. We argue this near-universal pattern is a category error. A vector database with top-k cosine retrieval answers exactly one question — what stored text is embedding-similar to my query? — and none of the questions a memory system actually has to answer. We define memory not as a retrieval function over embeddings but as a governance function over experience: the discipline of deciding what becomes durable, what decays, what reactivates, how contradictions resolve, how differently-typed knowledge is stored and updated, what consolidates, what is forgotten, what is relevant despite surface dissimilarity, and what must be suppressed even when retrieved. Retrieval is one operation inside this function, not the function itself: retrieval ⊂ memory. A system that only performs nearest-neighbor lookup has storage, not memory. We formalize nine governance decisions as the research questions the field has skipped, give a typed schema (fact / episode / procedure / working-state), and offer a constructive existence proof at PROTOTYPE/PILOT maturity — then connect memory governance to execution governance: a retrieved memory that shapes behavior is an admission decision.
1 · Retrieval is not memory
Something quiet happened to the word memory. Two years ago, giving a language model persistent memory was a hard, open problem. Today it is a tutorial: chunk your documents, embed them, put the vectors in a store, and at query time return the k chunks whose embeddings are closest to the query's. This is a genuinely useful pattern — retrieval-augmented generation (RAG) [1] — and its accessibility is a real achievement. But its accessibility is also how the word got redefined downward. Because the mechanism is easy, "memory" came to mean "whatever a vector database does." The field now routinely labels a similarity index a memory system.
This is a category error, and it is worth stating precisely rather than as a complaint. Nearest-neighbor retrieval answers one question: which stored items are embedding-similar to the current query? Memory has to answer a larger set: which experiences should persist and which should vanish; when an old memory should fade, and whether fading is deletion; when a faded memory should surge back because a situation recurred; how two memories that disagree reconcile; whether an item is a timeless fact, a timestamped event, a reinforced procedure, or ephemeral scratch state — because those demand different treatment; what many raw observations distill into; what ought to be forgotten by design; which memory is relevant despite being surface-dissimilar; and, critically, when a retrieved memory should be prevented from influencing behavior at all.
None of these are similarity questions. A similarity index is silent on every one. It is, precisely, storage with fuzzy keys — and storage is not memory. Storage remembers everything and knows nothing: it has no opinion about what mattered, no mechanism to let the past correct itself, and no gate between recalling something and acting on it.
2 · The category error
Retrieval over embeddings returns the top-k items by similarity to a query. It is a pure function of the current query and the current store — no state of its own, no notion of time beyond what the items encode, no output but a ranked list.
Memory as governance over experience is a set of policies over the life-cycle of experience: admission (what enters durable store, and how strongly), decay, reactivation, reconciliation, typing, consolidation, forgetting, relevance weighting, and suppression. Retrieval is the read primitive these policies wrap; it is necessary and radically insufficient.
A vector store's contribution over a hash map is that its keys are fuzzy — approximate meaning instead of exact string. That is valuable, and it is all it adds. Fuzzy-keyed storage still cannot tell you an item is stale, was superseded last week, belongs to a different tenant, is a one-off episode rather than a standing fact, or that recalling it is not permission to use it. Cosine distance measures textual-semantic proximity; it does not measure recency, authority, provenance, applicability, or trust. Treating proximity as a proxy for all of these is the error the rest of this paper anatomizes.
3 · The nine decisions a memory system must make
Each decision is a question the default architecture cannot answer. Table 1 summarizes; Table 2 gives the typed schema.
Q1 · Durability and differential encoding
Not all experience deserves equal persistence, yet flat vector stores encode every chunk with equal permanence and weight. A memory system must make a differential encoding decision at admission, driven by signals well studied outside AI: salience (how much it matters to current goals), surprise / prediction-error (it violated an expectation — the classic learning signal), consequence or reward, repetition, and explicit tagging. Weight admission by these and the consequential is remembered while noise washes out. Uniform-weight admission is not neutral — it is a decision to treat all experience as equally worth keeping, which no similarity score can correct after the fact.
Q2 · Decay
Human retention falls with time absent reinforcement [3]. An engineered memory needs decay driven by age, disuse, and supersession. The critical point: decay ≠ deletion. Decay should be graceful and reversible — a diminishing weight that lowers default influence and retrievability without destroying the item, so reactivation (Q3) stays possible. A store that only appends has no decay; a store that hard-deletes on a timer conflates decay with forgetting (Q7) and forecloses reactivation.
Q3 · Reactivation and reconsolidation
This is the decision pure similarity misses worst. A decayed memory is, by construction, ranked low by a similarity index — its low weight is exactly what top-k discards. But the signal that should restore it is not "the query resembles this item"; it is "the situation this memory belongs to has recurred." Context-triggered reactivation keys off entity, task, temporal, or environmental cues — this project, this counterparty, this error, this time of month — and surfaces the dormant memory because the context matches even when the query text does not. Biological memory reconsolidates on reactivation: a recalled memory becomes labile and is re-stored, updated by current context [4]. A similarity index re-ranks text, not situations. The most valuable reactivations are the ones a similarity query suppresses.
Q4 · Contradiction
When two items disagree, a memory system must choose among recency, authority, provenance, supersession, and coexistence — because they may not actually conflict. This is where typing (Q5) becomes load-bearing: a contradicted fact is a genuine conflict that must resolve (a person cannot both live in Boston and not), but two differing episodes are not a contradiction at all (a meeting Monday and another Tuesday). A flat vector store cannot tell these apart; it returns both contradictory facts side by side and lets the generator average them — which is how confident contradictions reach the user. Detecting a factual contradiction requires knowing a relation is functional (holds at most one value) — a symbolic property, not a metric one. And "lives in Boston" and "lives in Seattle" are near each other in embedding space, which is exactly backwards for detecting that they conflict.
Q5 · Typing
Four types, each with different persistence, decay, retrieval, and update semantics (Table 2). The distinction between the timeless fact and the timestamped episode is Tulving's semantic/episodic memory [8]; to these we add the reinforced procedure and ephemeral working state. The root error of the flat vector store is putting all four in one namespace with one persistence policy: a fact and a scratch note get identical treatment; an episode gets overwritten as if it were a fact; working state leaks into the durable corpus and poisons future retrieval with the residue of finished tasks. Typing is not an optimization — it is the precondition for every other decision being correct rather than uniform. Naive continual overwriting is also how connectionist memory forgets catastrophically [9].
Q6 · Consolidation
Raw experience is verbose and redundant; durable memory should be compact and general. Consolidation distills many low-level items into fewer high-level ones: episodes → facts or procedures (twenty observations of "deploy failed after config change" become the fact "config changes are high-risk" and the procedure "validate config before deploy"). The natural implementation is an offline digest pass — periodically, away from the request path, read the raw episodic buffer and write consolidated, higher-typed memory. This is the engineering analogue of sleep-dependent consolidation, where the hippocampal record is replayed and integrated into neocortical structure (the complementary-learning-systems account) [5]. A store that only appends at ingestion never consolidates; it accumulates.
Q7 · Forgetting
Forgetting is not failure; it is function. A memory system should deliberately discard working state after its task, superseded facts once resolution picked a winner, secrets and PII past their retention window, and noise consolidation judged not worth distilling. Two regimes matter: hygienic forgetting (keeping the store clean and relevant) and governance forgetting (honoring deletion requests, retention limits, the right to be forgotten) — the second a hard requirement in regulated domains and impossible in an append-only store that treats deletion as an afterthought. A memory system that cannot forget cannot be compliant, cannot stay relevant, and cannot stop finished-task residue from degrading future retrieval.
Q8 · Relevance beyond similarity
Similarity is a weak proxy for relevance. The memory that most needs to fire is often surface-dissimilar to the query. Relevance has dimensions embedding distance does not capture: causal relevance (this earlier event caused the condition you are now in), procedural applicability (this how-to applies even though its text shares few tokens with your query), entity and temporal links, and the blunt but vital "this bit me before" signal. None is monotonic in cosine distance; some are anti-correlated with it. Ranking by proximity alone systematically discards the causally and procedurally relevant in favor of the merely lexically alike — so retrieval@k by embedding has a relevance ceiling below what the store contains.
Q9 · Suppression and gating
This most sharply separates memory from storage. Retrieval finding an item is not permission to use it. A retrieved memory must pass a gate that can deny: the item is stale or superseded; out of scope; wrong tenant; unverified or low-trust; security-sensitive; or contradicted by a more authoritative item.
A pure retrieval system has no gate — whatever it returns flows straight into the prompt and thence into behavior — which is why "the model confidently used a stale/leaked/wrong-tenant fact" is a memory-architecture failure, not a model failure.
Summary tables
Table 1. The nine governance decisions.
| # | Decision | Question | What similarity alone cannot do |
|---|---|---|---|
| Q1 | Durability / differential encoding | What becomes durable? | weight admission by importance |
| Q2 | Decay | When does it fade? | lower influence without deletion |
| Q3 | Reactivation / reconsolidation | When does it return? | surface a low-ranked item because the situation recurred |
| Q4 | Contradiction | How do conflicts resolve? | know a relation is functional; veto a conflicting write |
| Q5 | Typing | Fact / episode / procedure / working-state? | apply four incompatible policies in one namespace |
| Q6 | Consolidation | What distills into durable form? | turn many episodes into a fact/procedure |
| Q7 | Forgetting | What must leave? | remove by design and prove removal |
| Q8 | Relevance beyond similarity | What matters despite dissimilarity? | rank by relevance, not proximity |
| Q9 | Suppression / gating | What must not act, even when recalled? | deny influence; fail closed under ambiguity |
Table 2. Typed memory schema — distinct semantics per type.
| Type | Persistence | Decay | Retrieval key | Update semantics |
|---|---|---|---|---|
| Fact | long; timeless | by supersession only | entity / relation | replacement under contradiction rules |
| Episode | bounded by relevance window | by age | time / context | append-only; never overwritten |
| Procedure | grows with successful reuse | by disuse; supersession by better how-to | applicability to task | reinforcement |
| Working state | task-scoped only | at task end (discard) | within-task only | mutated freely; must not leak to durable store |
4 · Why the field misses this
The category error is a response to an incentive gradient, not a failure of intelligence. RAG ships in a weekend — embedding APIs, vector stores, and top-k retrieval are commoditized; governance is real engineering with no turnkey library, so the path of least resistance produces storage, and storage demos well enough to be called memory. Benchmarks reward retrieval@k, not governance — recall@k, MRR, nDCG measure whether the relevant chunk was fetched, not whether the system decided correctly what to keep, forget, resolve, or suppress. This is Goodhart's law on memory research [6]: when retrieval@k becomes the target it stops being a measure of memory; several decisions (forgetting, suppression) are actively penalized by metrics that reward returning more, not withholding correctly. The embedding monoculture became the default ontology: if the only operation is nearest-neighbor over a flat namespace, contradiction becomes "return both," typing becomes "one collection," decay becomes "nothing," and suppression becomes "trust the top-k." And underneath it all, the missing typed schema is the structural cause — without types there is nowhere to hang differential persistence, no way to say "this is an episode, never overwrite it," and no basis for type-specific rules.
5 · A memory that governs, not just retrieves — reference architecture
The claim is narrow and we hold to it: memory-as-governance is buildable, because a working stack already enforces several of the §3 properties. We describe the systems at the property level only — enforced behaviors, not internal designs — and label each by maturity. None is enterprise-hardened.
- LIST — time-bounded, typed observation indexing (Q3, Q5, Q8). An observation index whose read interface is time-bounded and typed by construction, not a flat similarity blob: a read takes an explicit time lower bound and a per-project scope and returns bounded, typed, timestamped records. Relevance is therefore temporal and scoped, not only semantic — the context-recurrence signal similarity discards. And the interface is declared read-only and non-executing: recalling an observation cannot, by construction, cause anything to happen — the suppression stance of Q9 built into the read primitive. (PILOT.)
- Offline consolidation layer — digests over raw observations (Q6, Q7). An away-from-the-request-path pass reads many low-level observations and produces durable, higher-level summaries, with a narrative layer over the consolidated record rather than the raw stream — episodes→generalization, dropping the noise it judges not worth keeping. (PROTOTYPE.)
- jeeves-floor symbolic layer — typed relations, functional hard-vetoes, contradiction trichotomy (Q4, Q9). A typed symbolic store carrying relations, some marked functional (at most one value, the OWL FunctionalProperty sense) [7]. Its contradiction check exposes an explicit consistent / contradiction / unknown trichotomy, hard-vetoing a claim that violates a functional relation. Two properties matter: it resolves factual conflict symbolically (detecting that "lives in X" and "lives in Y" conflict, which no cosine distance encodes); and — the honesty that makes it memory rather than a guesser — the third verdict is unknown: with no checkable relation in the graph, it abstains rather than fabricating a ruling. We confirmed this live: an ungrounded claim returned unknown / veto=false with the reason "no checkable relation found." (PROTOTYPE/PILOT.)
- Evolution memory — lessons vs obstacles, informs but never authorizes (Q1, Q5, Q9). Two durable, typed records of past self-improvement attempts — lessons (proven) and obstacles (rejected) — stored distinctly. The load-bearing property is what it is forbidden to do: it is retrieval-only and is never a verdict authority. A lesson can inform a decision; it can never, by itself, authorize one — that authority lives in a separate, out-of-loop gate. (PROTOTYPE.)
Table 3. Coverage of the nine decisions. The stack answers Q4 (contradiction) and Q9 (suppression) substantially, several partially, and leaves graded reversible decay (Q2), automatic reconsolidation-on-recall (Q3), a unified typed schema (Q5), and governance-grade forgetting (Q7) open. That is the correct shape for an existence proof: the properties are buildable, some are built, and the problem is not pretended solved.
6 · Memory governance is execution governance
The suppression decision (Q9) is the whole theory viewed from one angle. Our companion paper The Orchestration Gap [2] argues that in a multi-model runtime, control must attach to the execution chain via four invariants: out-of-loop verification, signed provenance, fail-closed admission, and contradiction checking. A retrieved memory that influences behavior is an admission decision — when a recalled item flows into a prompt and shapes what the system does next, it has been admitted to the execution context exactly as a tool call is admitted to execution. The mapping is one-to-one: provenance (a memory carries where it came from, so the gate can weigh authority), fail-closed admission (under ambiguity, withhold), contradiction checking (veto a memory that conflicts with a more authoritative one), and out-of-loop authority (memory informs; a distinct authority decides).
7 · Open problems
Restating the decisions as the agenda the field must take up: a principled salience/surprise/consequence model for admission (Q1, partial); graded reversible decay (Q2, open); reconsolidation-on-recall (Q3, open); contradiction resolution generalized beyond modeled relations (Q4, substantial); a unified fact/episode/procedure/working-state schema (Q5, open); auditable consolidation (Q6, partial); governance-grade, provable forgetting (Q7, open and important); causal/procedural relevance combined with — not subordinate to — similarity (Q8, partial); and a unified, auditable memory-admission gate (Q9, the natural next build). Cross-cutting all of them: evaluation. The field needs benchmarks that score governance — correct forgetting, suppression, contradiction resolution, reactivation — not just retrieval@k. Until they exist, Goodhart keeps steering effort toward retrieval and away from memory.
8 · Conclusion
The barrier to entry collapsed, and in collapsing it quietly redefined memory down to whatever a vector database does. That is a category error with a precise shape: it mistakes storage — persistence plus fuzzy-keyed recall — for memory, which is the governance around persistence and recall. The nine decisions are the part of memory that is not retrieval, and they are, individually and demonstrably, buildable. The correction is not a better embedding model; it is a typed schema and a set of governance policies wrapped around retrieval, and a gate between recalling a memory and letting it act.
References
- P. Lewis, E. Perez, A. Piktus, et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS, 2020. arXiv:2005.11401
- Perslis Research. "The Orchestration Gap: Why Model-Level Alignment Cannot Survive Multi-Model Runtimes." Preprint, 2026. research.perslis.com/orchestration-gap.html
- H. Ebbinghaus. Memory: A Contribution to Experimental Psychology. 1885 (Eng. trans. 1913). The forgetting curve.
- K. Nader, G. E. Schafe, & J. E. LeDoux. "Fear memories require protein synthesis in the amygdala for reconsolidation after retrieval." Nature 406:722–726, 2000.
- J. L. McClelland, B. L. McNaughton, & R. C. O'Reilly. "Why there are complementary learning systems in the hippocampus and neocortex." Psychological Review 102(3):419–457, 1995.
- C. A. E. Goodhart. "Problems of Monetary Management: The U.K. Experience." 1975. Popularized by M. Strathern (1997): "when a measure becomes a target, it ceases to be a good measure."
- W3C. "OWL 2 Web Ontology Language: Structural Specification and Functional-Style Syntax (Second Edition)." W3C Recommendation. owl:FunctionalProperty
- E. Tulving. "Episodic and Semantic Memory." In Organization of Memory, pp. 381–403. Academic Press, 1972.
- R. M. French. "Catastrophic forgetting in connectionist networks." Trends in Cognitive Sciences 3(4):128–135, 1999.
- A. d'Avila Garcez & L. C. Lamb. "Neurosymbolic AI: the 3rd wave." Artificial Intelligence Review 56(11):12387–12406, 2023.
How to cite
Perslis Research. "Retrieval Is Not Memory: Memory as a Governance Function over Experience." Preprint, 2026. https://research.perslis.com/memory.html
@techreport{perslis_retrieval_is_not_memory_2026,
title = {Retrieval Is Not Memory: Memory as a Governance
Function over Experience},
author = {{Perslis Research}},
institution = {Perslis Research},
type = {Preprint},
year = {2026},
url = {https://research.perslis.com/memory.html},
note = {Preprint, not peer-reviewed}
}