How Compaction Works in Pi
Why compaction is needed for large language models and how Pi implements it
📝 摘要
本文详细解释了编码智能体（如 Pi）中 Compaction（上下文压缩）的工作原理。当 LLM 对话历史超出上下文窗口限制时，compaction 通过 LLM 摘要替换旧的对话历史，保留近期消息不变，从而为新消息腾出空间。文章从 LLM 对话结构出发，对比了两种上下文溢出处理策略（重置 vs 压缩），并深入解析了 Pi 的具体实现——使用独立的摘要请求、专门的 system prompt 和结构化摘要格式。最后讨论了 compaction 对 prompt caching 的影响：压缩会打破精确前缀匹配，导致缓存失效，但后续请求可重新建立缓存。
核心概念： Compaction 是编码智能体处理上下文溢出的核心机制，本质是用 LLM 生成的历史摘要替换完整对话历史，在保留关键上下文的同时释放上下文窗口空间。
关键技术栈 / 关键要点：
- 上下文窗口限制 — Transformer 架构限制了 LLM 可处理的输入长度，编码智能体的对话历史持续增长终将溢出
- Compaction 触发机制 — Pi 在 turn 结束后自动检测上下文接近上限时触发，也可通过
/compact手动触发，甚至在遭遇溢出错误时 mid-turn 触发 - Pi 的摘要策略 — 使用独立的摘要请求（非复用对话历史），采用不同的 system prompt（"context summarization assistant"），可使用不同 LLM 模型而不增加成本
- Token Budget 保留机制 — Pi 默认保留最近 20k tokens 的消息不参与压缩（约 5-20 turns），仅压缩更早的历史
- Prompt Cache 失效 — Compaction 改变了前缀，导致 retained turns 的缓存状态不可复用，首次 changed token 之后的所有内容必须重新计算
- 可扩展性 — Pi 支持自定义 compaction 扩展，用户可替换摘要 prompt 实现不同的压缩策略
关键结论 / 实践建议： Compaction 是不可避免的权衡——牺牲精确前缀缓存以换取对话连续性。理解缓存失效的成本有助于在长会话中合理安排 compaction 时机；Pi 的独立摘要请求设计使得可以使用更便宜的模型完成压缩，降低总体成本。
Date:
From:Earendil Engineering < rfc@earendil.com >
To:You
Subject:
If you have ever had a long coding session in a coding agent like Pi, Claude Code, or Codex, you will have triggered a compaction. In this post we explain how compaction works and when Pi needs to compact.
An LLM conversation
Large language models (LLMs) have limited context windows. The context window is what the model can "see" while producing a response. The transformer architecture used by LLMs limits how much input they can process. The input for a coding agent session includes all the previous messages and tool calls, and this keeps growing as you work. Once it exceeds the context window, the LLM rejects the request.
When working interactively with a coding agent like Pi, the agent sends requests to an LLM and receives responses. Each request includes a system prompt, loaded files such as AGENTS.md, tool definitions, and the conversation history.
A coding agent's first LLM request contains this initial context, along with a first user message.
request 1:
[system][tools][user]
This starts a turn. The LLM may first return an assistant message containing tool calls. The agent program executes them and sends a new request to the LLM containing the complete conversation, now including the tool results. We get back another assistant message. The turn is finished when the assistant has completed generating output.
after request 1:
[system][tools][user][assistant: tool call][tool result][assistant]
<-------------------> ^ <--------->
returned by LLM | returned by LLM
|
produced by the agent
We continue working, and send another message.
request 2:
[system][tools][user][assistant: tool call][tool result][assistant][user]
^
new user message
Each turn expands the conversation. Eventually, the history exceeds the context limit. The next request then returns an error such as Request exceeds the maximum size.
[system][tools][user][assistant][....][tool result][user]
^
exceeds context window
Handling context overflow
When we cannot continue with the existing conversation as-is, we have two choices.
- We can start a new, empty conversation without the accumulated context. This discards the history, including prior decisions and unresolved work. It might still be a good idea to do, because the performance of LLM outputs decrease as the context size grows.
- We can create a smaller representation of the conversation context, since we want to keep this conversation going. That is what compaction does.
Compaction
In theory, there are many ways to implement compaction. For example, we can write a deterministic function which keeps some of what is in the conversation and discards the rest. In practice, though, implementations of compaction use an LLM request to summarize the conversation history.
Compaction replaces part of the history with a compressed representation, leaving room for additional messages and tool calls.
[system][tools][compaction result][user]
^
new message
Pi's implementation
Let's look more closely at how Pi specifically implements compaction.
When conversations grow too long, Pi uses compaction to summarize older content while preserving recent work. Compaction is triggered when the context limit is nearing the total size of the context window. It can also be manually triggered using the /compact command.
Pi checks for auto-compaction after a turn ends. Until then, each request extends the existing prompt and can reuse its cached prefix. Pi may also compact mid-turn, if it encounters a context overflow error.
When compacting, Pi retains some number of recent messages unchanged.
before compaction:
[system + tools][older turns][recent retained messages]
The number of retained messages varies because Pi uses a configurable token budget. Pi's current default of 20 thousand tokens comes out to roughly 5 to 20 turns. All the messages before this cut point are extracted and serialized, and will be summarized.
Pi's compaction prompt
The ideal outcome of a good summarization for a coding agent is like a handoff briefing from one shift to the next. Pi's compaction prompt focuses on the fact that there is a lot in the existing context that is no longer relevant. We should only keep around what is still important context for the next LLM request.
Pi therefore sends a different request for compaction than for regular conversation.
- The system prompt used in the standalone compaction request is different. Instead of telling the LLM "you are an expert coding assistant", we tell the LLM "you are a context summarization assistant."
- The user message in the compaction request is also different. It requests "a structured summary of this conversation branch for context when returning later." The prompt specifies sections for goal, progress and key decisions.
- It's a standalone request that doesn't use any of the existing conversation history, which means it can use a different LLM model without incurring any unnecessary cost.
The result of the compaction is appended to the Pi session as a compaction entry, and the session can now continue. After the compaction request, the context has been compressed.
after compaction:
[system][tools][summary][recent turns][new user message]
There is now room in the conversation context for many more messages.
Pi stores the compaction summary as plain text in the session. This keeps the compacted context readable and portable, since we can switch models in Pi and continue using the summary.
Compaction and prompt caching
Prompt caching is used by LLM providers to make repeated requests in the same conversation less expensive. In an active coding session, we pay less for the context that has already been generated by the model. This caching requires an exact prefix match, so compacting a session will break the prompt cache.
cached before compaction:
[system][tools][older history][recent retained turns]
<-------------------- cached prefix -------------------->
first request after compaction:
[system][tools][summary][recent retained turns][new user message]
<-- reusable -->^
|
first changed token
|
+-- everything after this point must be recomputed
The retained turns contain the same tokens, but they now follow a different prefix. Their previous cached state therefore cannot be reused.
New requests after compaction will benefit from prompt caching again.
Experiment
Since Pi is extensible and malleable, you can replace its compaction with your own. To test a different compaction mechanism, ask Pi to create an extension with a custom compaction prompt.