Memory Is a Compiler
tl;dr: Memory preserves the past in a form meant to be useful in the future, then recompiles what that past means as new evidence and new qu...
📝 摘要
本文提出"记忆是编译器"的核心比喻，批判了当前 AI 领域将向量数据库、知识图谱、上下文窗口、RAG 等数据结构等同于"记忆"的混淆。作者认为，这些只是记忆系统的编译产物（IR），而非记忆本身。真正的记忆是一个持久化系统：它保存证据、接收新证据，并持续重新编译证据对当前任务的意义。文章以个人融资经历为例，说明同一件事的意义会随新证据而改变——这不是记忆失败，而是记忆的核心工作方式。
核心概念： 记忆不是表示，而是编译。它保存原始证据（ledger），并基于新信息持续重新编译出 State、Retrieval、Context 三种产物。任何可从 ledger 重建的都是编译视图，不可重建的才是真正的记忆。
关键技术栈 / 关键要点：
- 记忆的两个核心能力 — 保存过去的本质 + 基于新信息重新解释，缺一不可
- 记忆编译三种产物 — State（当前状态，fold 产生）、Retrieval（检索候选，不做判断）、Context（单次调用的有界视图）
- 每种表示都是 IR — 向量索引擅长相似性、图擅长关系推理、列式表擅长聚合，不应将任何一种作为 canonical 形式
- LLM 是编译器的一个 pass — 负责从非结构化证据提取类型化断言，但整个编译管线必须是类型化、可重放的
- AOT/JIT/PGO 记忆 — 常用结构提前编译(AOT)、意外问题即时编译(JIT)、反复出现的 JIT 路径提升为 AOT(PGO)
- 保留原始字节 — 永远不要丢弃重新解释的能力，raw artifacts 是证据而非真理
- 记忆可以研究自身 — 后台编译派生声明（如 PageRank 中心性）并写回 ledger，即"做梦"
关键结论 / 实践建议： 设计 AI 记忆系统时，核心是维护一个不可变的只追加 ledger（证据账本），所有上层结构（图、索引、状态投影、上下文）都是可重建的编译视图。保留原始数据、跟踪依赖关系、支持局部失效和重编译，才是记忆系统的关键能力。
tl;dr: Memory preserves the past in a form meant to be useful in the future, then recompiles what that past means as new evidence and new questions arrive. What we often call “memory”, RAG, vector DB, context windows, context graph, knowledge graph, etc, they are not memory at all. Those are temporary representations compiled from it.
Several years ago, when I was fundraising for an idea, I walked out of a partner meeting at a venture firm convinced that one particular partner had just killed our round. I remembered the meeting vividly: the dude circling back to the same slide, the answers I gave, and the pass email that arrived a day later. For months after that the whole experience had a simple meaning in my head. This was the firm that didn't get it, and the partner who'd made sure of it. Fast forward about a year, and I learned something I hadn't known when I pitched. The risk that partner kept circling back to was all too real, and his suggestion was the one that ultimately kept the idea from dying outright.
What's strange is that I hadn't misremembered anything. My recall of the questions, and their order, was accurate, the pass email was still in my inbox, and when I went back to re-read it the sentence was exactly as it had always been. But my memory of the meeting, my understanding of it, changed completely. The partner who'd sunk the round was now the person who'd diagnosed us first. Skepticism became warning, the pass became a diagnosis I didn’t listen to and their suggestion became the fix. I was remembering the same meeting and relating it to everything else differently.
The past hadn't changed, but what it meant had. And, I bet you have experienced things like this too, cos’ this happens all the time. A decision that looked foolish turns out to be reasonable once you learn something you didn't know when it was made. That, I think, is the hallmark of memory, the kind we have and the kind most intelligent systems should have. People define memory in a lot of ways, and it certainly does more than two things, but it does at least these two: it holds on to the essence of something from the past, and it’s able to be reinterpreted based on new information. For this piece, those two are all I'm going to worry about. I'm using "memory" broadly here, for a person, an agent, a team, or any system that accumulates knowledge and experience and later acts on it. The implementation might differ but the problem doesn't. The past accumulates, new evidence arrives, and what the past means for the present keeps changing.
What do we mean by memory anyway?
From here on I'll stick to AI systems. It's where the word gets used most loosely, and it's where the design is still up for grabs. And, depending on who you ask, an AI system's "memory" might be a vector database, a knowledge graph, a context window, a conversation transcript, a key-value cache, or weights trained into a neural network.
That list should already make you a little suspicious. Most of those are data structures, each specialized in its own way: vectors are good at similarity, graphs at relationships, tables at aggregation. A context window is closer to a scratchpad for a single LLM pass, and weights compress statistical regularities. None of them can even begin to do what we described at the start. So why are we calling all of them memory?
We've taken representations that are useful for particular computations and confused them with the memory system itself. My memory of the meeting didn't permanently contain the single graph edge
Partner ──KILLED──► Round
because later experience made this one far more useful:
Partner ──IDENTIFIED──► Risk ──THREATENED──► Company
Memory isn't a representation; rather, it's a persistent system that keeps evidence, takes in new evidence, and keeps recompiling what that evidence means for the task at hand. Put another way, it stores the present because it expects to need it later, and it organizes what it stored around that expectation. The catch is that the expectation is formed before the question exists, so when the question arrives it's often not the one the memory was organized for, and the memory has to reorganize. The ledger persists; memory compiles.
Three things memory compiles
Once you treat memory as the persistent system, three things that usually get confused with it fall into place, and it helps to be concrete about what the system is compiling from. Underneath there isn't much: an append-only ordered-list of things that happened, and the raw artifacts they came from where you can afford to keep them. Each entry is one small fact that carries its source, its time, its confidence, and maybe who is allowed to see it if it's a shared system. Producing those entries is the first thing compilation does, and it's extraction in the plain sense, from a pass email to a fact about which partner, which risk, and which date, pointing back at the sentence it came from. Nothing in the list is an answer; everything that turns it into answers is compilation, and it produces the three things below.
- State is what currently holds, and it's produced by folding the list in order: take each event, apply it to the state that came before, keep going. When a new event lands the state changes once, whether or not anyone asks, which is what makes it state rather than a pile of evidence and a hope. And because the fold can stop anywhere in the list, what the system believed in March is a query rather than a forensic project.
- Retrieval finds evidence that might be relevant, and it's an instrument the compiler uses rather than a thing the memory is. It runs over whatever indexes have been built from the list, by keyword, by similarity, by entity, by time, and hands back candidates. It doesn't decide what's true; if it returns two records that contradict each other, retrieval has done its job, and choosing between them is somebody else's.
- Context is the bounded view assembled for one invocation, meaning this question, this asker, this moment, built from the folded state and the retrieved evidence this asker is allowed to see. It's used once and released, because keeping it would turn a view into a claim about the world, and views go stale the moment the list grows.
None of these is memory, rather they’re what memory compiles, and each of them can be thrown away since they can be rebuilt from the list, which is the only thing that can't be.
Every representation is an IR
Compilers routinely lower one source program into different intermediate representations (IRs) for different jobs. LLVM doesn't force your source code to permanently look like the form that's best for loop optimization and then make every other pass work on that. Memory shouldn't permanently look like the form that's best for any one operation either. Whichever structure you pick as canonical, you've baked in a set of questions. A vector index makes "what's similar to this?" cheap and "who depends on whom?" nearly impossible, a columnar table makes "how many, grouped by what?" cheap and "what's similar?" hopeless. Each one is the right shape for a job and the wrong shape for the others.
Operation Useful compiled representation (IR) Similarity search Vector index Relationship reasoning Graph / adjacency matrix Aggregation and filtering Columnar table Model invocation Bounded context window
The representation usually follows the operation. Graphs deserve their own mention, though, because "knowledge graph" has become the most common answer to "what's your memory?" It's also the easiest place to make the canonical-representation mistake, because a graph looks like meaning: an edge reads like a fact. But a graph is a poor canonical representation for the same reason it's a good IR. Every graph privileges some particular set of entities, relationships, and questions, the ones it was drawn to answer. That doesn't get rid of graphs, it moves them from truth to IR. Want PageRank? Compile an expertise graph. Want shortest paths or communities? Compile the corresponding graph and run the ordinary algorithm. If memory can compile the graph you need, [, then an ordinary graph computation [ is just [.
That's what happened with the meeting. New evidence produced a different relational view of the same event.
The compiler is not an LLM. The LLM is one compiler pass.
So what does the compiling? Almost all of the evidence is language: transcripts, emails, diffs, tickets, the pass email in my inbox. Until recently nothing could read that and produce structure from it without a person in the loop. LLMs can. Give one a transcript and it can tell you who was there, what was claimed, what was decided, and with what confidence. Give it an old interpretation and a new fact and it can tell you what the fact changes. That's the front-end job a compiler needs done on messy source: decompose it into typed pieces whose meaning can be manipulated. The obvious move is to hand the whole job to the LLM, and that's the mistake.
Compilers are normally deterministic but LLMs clearly aren't. Run the same transcript through the same model twice and you can get two different sets of assertions, run it through next year's model and you'll get a third. That's fine for a proposal step and fatal for a system of record, because everything downstream (what we believe, what depends on it, what to invalidate when it changes) has to be rebuildable from a record that doesn't move. So the semantic model has to sit inside a larger compilation pipeline that is typed and replayable, and the LLM is one pass in it.
The model proposes typed assertions from messy evidence. Deterministic validators check schema, ontology, provenance, and policy before anything is committed. Once committed, structural projections can be deterministic relative to the ledger and the projector version.
Ahead-of-time, just-in-time, and profile-guided memory
Of course, remembering doesn't mean replaying your entire life every time someone asks where you left the keys.
AOT = structures we already know we need JIT = the view this unexpected question needs PGO = learning which JIT work should become AOT tomorrow
In practice, common structures are maintained AOT, unusual questions assemble JIT views out of them, and JIT paths that keep recurring get promoted to AOT through PGO.
Every view carries a ledger watermark, so staleness is explicit: if a graph store is at ledger_epoch: 1042 while the ledger is at 1048, you patch it, tolerate the gap within an SLO, or fail closed.
Keep the bytes (and why memory is not a model)
If you take reinterpretation seriously there's another consequence: never throw away your ability to change your mind. Keep the bytes.
A 2028 semantic model should be able to reinterpret a 2026 transcript rather than inherit its parser forever. Raw artifacts are evidence, not truth, and preserving them lets a better compiler revisit an old interpretation. That pass email is the small version of this. It sat unchanged in my inbox for a year while what it meant was rebuilt around it.
This is also why memory isn't a neural network model. Parametric learning asks how the model's function should change after new experience. Memory asks a different and usually more surgical question: what does this evidence change, what depends on it, and what can stay untouched?
new evidence ──► append ──► invalidate affected interpretations ──► recompile dependents
The system changes locally because the evidence changed locally, with no global retraining cycle. A model can perform a compiler pass, even the most sophisticated one, but persistent evidence, dependency tracking, invalidation, and recompilation belong to the memory system.
A memory that studies itself
So far the memory has only changed when the world changed or someone asked it a question. But there's no reason to wait. A memory can study itself.
At 3 a.m., nobody is asking questions. The system compiles a broad interaction graph (who consults whom, who reviews whose work, who gets pulled into incidents) and runs PageRank over it.
The result isn't the canonical claim "Elena is important." It's a derived claim:
claim: Elena centrality = 0.0873
type: DERIVED
graph: expertise-network-v7
ledger_epoch: 1048
That discovery goes back into the ledger as a derived claim with lineage. The same machinery can find latent clusters, dependency cycles, or shifts over time. This is why I call the slow path dreaming: the system isn't recalling its past so much as reorganizing it.
No weights change, yet tomorrow's compilations can use structure that wasn't available yesterday. Experience changed what the system can compute.
Here's a useful test. If something can be deleted and faithfully rebuilt, whether it's a graph, an index, a state projection, or a context, it's a compiled view. Evidence and semantic history are different. Lose them and something irrecoverable is gone.
We started with a meeting whose meaning changed when new evidence arrived. A memory that could never do that wouldn't be more faithful. It would just be unable to learn.
The meeting stayed the same, but my view of it changed. That wasn't memory failing, rather that was exactly how memory works.
----
Note: I am CEO of Sentra and deal with some of the issues that I wrote about here. At Sentra, where we are building a unified foundational layer that is a shared "memory" (as defined in the article) in service of enabling a company brain: a shared, living model of the entire org. It absorbs all data, structured and unstructured in an org and services humans and agents alike.