Designing an Always-On Agent

📝 摘要 #

本文讨论「always-on agent」（常驻智能体）的架构设计：与需要用户主动发起、生命周期短暂的 chatbot 不同，always-on agent 具有 single-threaded（单线程）、proactive（主动）、situationally aware（情境感知） 三大特征。作者认为，人们对于是否已实现 AGI 的分歧，更多源于 form factor（形态）而非模型能力。文章从上下文管理、心跳唤醒、消息工具、通知订阅几个层面给出可落地的架构干预，并指出真正的瓶颈在于模型的 discernment（判断力）。

核心概念： 常驻智能体的难点不在「让它能做事」，而在「让它知道何时该做事、何时该保持沉默」；架构可以廉价地搭出来，判断力才是范式转变的核心障碍。

关键技术栈 / 关键要点：

1. Single-threaded（单线程） — 把多会话收敛为一个了解用户的 AI 线程；核心难题是上下文窗口管理和多任务并行，需要 compaction 与 dispatch 两种手段。
2. Agent 自主压缩 — 传统第三方 compaction 会「切断 agent 的动量」；Rubric 改为让 agent 每轮感知 token 用量，并自行调用 foldContext（将一段消息折叠为自写摘要）和 dropContext（丢弃一段消息）。
3. Dispatch / 子智能体 — 主线程只保留高层上下文，把订餐、预约等重活派发给 sub-agent，实现边聊天边并行推进。
4. Heartbeat 心跳 — 向聊天历史注入一条 "heartbeat" 消息以唤醒睡眠中的 agent，可按固定节奏运行，也可由 agent 自行安排唤醒时间。
5. Messaging as a tool — 主动场景下，agent 只有调用发消息工具才会触达用户；它醒来做完事可以判断「没什么值得说」而保持安静，最后一条消息只作为给自己看的记录。
6. 情境感知 — 通过广泛访问（邮件、浏览器、文件系统 + computer use）和 notification 系统把外部事件即时注入上下文；用小型模型（如 Jev）给通知分级，重要通知才唤醒 agent，其余批量堆积。
7. Discernment（判断力） — 今天的模型在 RL arena 中只学会了「攻克任务」，在模糊环境中要么过度做无用功（如回答不属于自己的 Slack 消息），要么错过真正的触发点。

关键结论 / 实践建议： 架构层面（单线程、心跳、收件箱）已是已知可建的部分；应让 agent 从「害羞」开始，随记忆和正反馈逐步建立主动行动的自信，并在 system prompt 中明确「什么都不做完全没问题」、不确定时先询问、行动前检查用户历史反应。作者认为新的、以判断力和记忆使用为奖励目标的 RL 环境是最大的机会。

---

Before ChatGPT was released, I used to imagine AI would be something like Jarvis from Iron Man. I imagined it would always be running, it would have a personality, and it would have the same awareness of my environment that I do.

But today's chatbots and agents aren't like this. We spin them up for tasks and projects, but if we don't send them a message, nothing happens. They're dry and robotic, and they only know the context we give them.

I have a hunch that the reason people disagree about whether we've reached AGI is more about form factor than it is about model capabilities.

To (over-)simplify, I would define the always-on, Jarvis-like form factor as:

• single-threaded
• proactive
• situationally aware

Whereas chatbots and agents are:

• ephemeral (short-lived)
• reactive (they only work when explicitly started)
• context-dependent (heavily influenced by whatever happens to be in front of them)

Chatbot1 thread

Always-on agent1 thread

A chatbot and an always-on agent

In this post, we'll talk about how we're designing architectures for always-on agents, and the limitations of today's models that make this hard.

Single-threaded #

Instead of managing a bunch of different threads, and setting each one up with the right context for our work, we just want to talk to one "AI" that knows us and handles everything.

Let's say we woke up sick. We want one agent to be able to cancel our meetings, order soup, look into our symptoms, and book a doctor's appointment if necessary.

At first, this sounds simple: just unify everything into one thread and call that our single-threaded agent. The hard part is context window management, since passing everything through a single agent will quickly fill up its context window.

We also often want to do several things at once. Booking the doctor's appointment or ordering food might take a while, because they're heavier browser-use tasks, and we want to keep chatting about our symptoms without waiting for everything else to get done.

Compaction #

Compaction is the de facto standard for managing context windows, but we've found that compaction lobotomizes the agent's momentum, especially when it's done by a third-party process.

It makes more sense to give the agent tools to compact and awareness of how full its context window is, and then just let it manage its own tokens, so it can keep its momentum while avoiding overflow.

Tangibly, we give our agents:

• awareness of their token usage at every turn
• a foldContext tool, which replaces a range of messages with a summary the agent writes
• a dropContext tool, which removes a range of messages from context

In practice, our agents tend to wait for a lull, like the end of a task, before they compact. Then they fold away the reasoning and tool calls they won't need again, and keep everything that's still in play.

Context window+2K14K / 128K

1I woke up sick. Cancel my meetings?

An agent folding and dropping its own context

Dispatch #

Even with good compaction, one thread fills up fast if the agent does all the heavy work in it. So we also give the main agent sub-agents it can dispatch work to.

This kills two birds with one stone: it frees up the main thread to hold more, higher-level context about everything, and it lets the agent multitask on deeper work, like the food order and the doctor's appointment, while it keeps talking to you.

Agent

I woke up sick. Can you get

me soup and a doctor?

On it. How are you feeling?

A sick day in one thread, with two sub-agents

Proactive #

Proactive agents can do work without being prompted, and they can send unsolicited messages. For example, if the agent knows you're flying to Austin on Wednesday, it can find the check-in details for your Airbnb, work out the best way there from the airport, and send it all to you as you land.

To unlock this behaviour architecturally, we first need to get away from the request-based nature of the agent. Typical systems act a lot like synchronous functions: we send a request and wait for the agent to return.

For proactivity, the agent needs to be awake whenever it wants, and to message us regardless of whether we sent it a request.

Heartbeat #

We give the agent a heartbeat. A heartbeat is just a simple message, "heartbeat", that we push into the chat history. It wakes the agent if it's asleep.

We can run the heartbeat on a cadence (for example, every hour), or have the agent manage its own awakeness by scheduling its own heartbeats.

Messaging as a tool #

We also give the agent a tool for sending messages to the user. In a normal chat, the agent's last message in a turn is its reply to you. Here, it only reaches you when it calls the tool, so it can wake up, do some work, and decide there's nothing worth saying. Its last message stays in the thread as a note to itself about what happened.

In the Austin example, the agent wakes up on a routine heartbeat and discovers the flight. It dispatches a sub-agent to pull together the directions and check-in details, then schedules a heartbeat for around the time the flight lands, so it can message you at the right moment.

Agent

Wednesday, October 287:00

AA 1432Departs 9:40 AMORDAUS

The Austin trip, from heartbeat to message

Newsletter

The Grid

What we’re learning about building with AI. Two short emails a month.

Situational awareness #

The prerequisite to proactivity is knowing what you're doing, and when. We want our agents to be able to act on incomplete information from us, and, like in the Austin example, even start working on triggers from the world, not just on direct instructions from us.

Broad access #

At a basic level, we can enable situational awareness with broad access: give the agent full access to our email, browser and file system so it can discover flights and accommodations we booked, search past orders, and find patterns in our behaviour that surface our preferences.

The one simple trick here is just giving agents computer use tooling and access to our computers. Coding agents got great with one simple tool for running terminal commands on our computers, and general-purpose agents can do the same with computer use.

But agents don't know what they don't know, and being technically capable of discovering context often isn't enough to have a real-time understanding of what's going on.

Notifications #

So we give agents a notifications system, where information is injected into the agent's context window as soon as it's available. By subscribing to events in the world, agents can discover important context sooner and step in when they're needed.

For example, if your flight to Austin is cancelled, the agent can immediately start looking for alternative flights while messaging you to coordinate, instead of waking up an hour after the problem is solved to see what happened.

To implement notifications, we give the agent an inbox, and documentation on how to adapt webhooks and polling jobs to push events into that inbox.

One problem quickly emerges: the inbox fills up with noise. Spam offers in email, Slack messages that aren't addressed to the agent, GitHub events from repos that aren't in focus. If we wake the agent for every notification it receives, we could be doing so every minute, burning tokens and clouding its context window.

One way to cut the noise is to classify notifications as important or unimportant with a smaller model, such as a system one model like Jev. Important notifications wake the agent, and unimportant ones pile up so the agent can read them in bulk when it wants to.

Jev

Wake the agent?

Yes

No

AgentAsleep

Later0

Notifications sorted by a small model

Discernment #

Architecturally, everything above is a fairly simple series of interventions. Make the agent single-threaded, and give it tooling for compaction and dispatch. Give it a heartbeat, and separate when it's awake from when it messages you. Give it broad access and notifications so it can discover what it needs to be helpful on its own.

But for the models available to us today, this re-architecture is a paradigm shift in expectations. Before, we would give our agents a task and the context to complete it, and they would work until they finished and returned to us. Now, we're giving them a raw, unfiltered stream of noise and asking them to figure out what to do, and how to do it, while they keep processing more noise.

And today's models are very very bad at this, because they lack discernment.

Everything in their training (specifically their RL arenas) has taught them how to crush tasks. So these agents demo really well when the signal to kick off work is very clear, but they struggle to figure out how to be helpful in ambiguous environments.

For example, agents processing messages on Slack often decide to act on messages that aren't addressed to them. Then, after discovering they lack the mandate, access and context to complete the task, they settle on writing a design doc and sharing it. This is useless and embarrassing.

An agent answering a message that wasn’t for it

On the flip side, agents often miss real triggers to do work.

Agents are either too eager to do busy work or missing opportunities to be helpful. Calibrating them to the right balance is more art than science.

Designing for discernment #

The behaviour you want to design is for the agent to start out shy, and over time build the confidence to act proactively as it builds itself a lane where it can make an impact.

This hinges on good system prompting and good use of memory. Concretely, we want:

• a system prompt that's really clear about what the agent is responsible for, and that doing nothing is totally fine
• the agent to ask when it isn't sure whether something is its job, instead of guessing
• a memory of how we reacted when it did act (did we thank it, correct it, or ignore it?), which it checks before it acts again

I also think there are huge opportunities in new RL environments that reward discernment and memory use for this new paradigm.

Back to Jarvis #

There's still a long way to go before agents feel like Jarvis. Some of the gap is architecture we already know how to build: one thread, a heartbeat, and an inbox. Discernment is the hard part, and it's the part I'm most excited to work on.

We're building always-on agents at Rubric right now. If you are too, we'd love to compare notes.