Evaluating OpenWiki with WikiBench

📝 摘要 #

LangChain 发布了 WikiBench，一个基于 Harbor 框架构建的评测基准，用于衡量 OpenWiki 生成的代码库文档（Wiki）的质量和实用性。WikiBench 的核心创新在于使用"阅读智能体"（Reader Agent）来回答关于代码库的问题，从而间接评估 Wiki 的实际帮助程度。评测流程包括自动生成两类问题——"覆盖率问题"（基于模板的子系统级别问题）和"检索问题"（针对特定行为的细粒度问题），并通过 LLM 评委逐条验证答案的事实正确性和来源依据性（grounding）。

实验结果表明：DeepSeek Flash 和 GLM 5.2 在不同成本点上表现优异，但更贵更慢的模型主要通过阅读更多文件来加深理解而非产出更多内容；Wiki + 源代码的组合在准确率和成本效率上均优于仅使用源代码，说明 Wiki 的最佳角色是作为"索引"引导 Agent 快速定位，而非替代源代码本身。

核心概念： Wiki 对编码智能体的价值在于充当代码库的"索引"，与源代码配合使用时能同时提升回答准确率和成本效率，单独使用效果有限。

关键技术栈 / 关键要点：

1. WikiBench 评测框架 — 基于 Harbor（Environment + Agent + Verifier 三组件）构建，用 Reader Agent 回答代码库问题来评估 Wiki 质量
2. 两类评测问题 — Coverage 问题（模板化子系统级问题）和 Retrieval 问题（针对特定行为的细粒度问题），配合 JSON 评分标准逐条打分
3. Grounding 验证机制 — 双层 LLM 评委：第一层检查事实是否出现在答案中，第二层验证答案是否基于 Agent 实际阅读的 Wiki 页面
4. 成本-性能权衡 — DeepSeek Flash 成本仅为 GLM 5.2 的 1/6，更贵模型的多余开销主要用于理解更多文件而非产出更多页面
5. Wiki 作为索引 — Wiki + Source 组合得分最高且成本低于纯 Source，Wiki 单独表现差，说明其定位应是导航指南而非知识替代

关键结论 / 实践建议： 在构建 Agent 文档系统时，应将 Wiki 设计为代码库的索引层而非独立知识源；评测文档质量应关注其能否帮助 Agent 更准确高效地回答实际问题，而非仅衡量文档完整性。

---

OpenWiki is our open source agent for generating and maintaining codebase documentation. As we improve OpenWiki, we’ve needed to measure two things:

1. Does a wiki actually help coding agents?
2. Are changes we’re making making OpenWiki better?

We built WikiBench to answer both those questions. It evaluates generated wikis using questions grounded in the underlying codebase. Doing this allows us to measure the difference in quality between different wikis as well as how much the wiki actually helps.

WikiBench runs on Harbor #

We built the benchmark in Harbor, a framework for evaluating agents on long-running tasks. Harbor tasks have three components:

• The environment provides the context for the task
• The agent performs the task
• The verifier evaluates the result

In WikiBench, the environment is a repository checked out at a pinned commit. The agent runs OpenWiki initialization on that repository and produces the initial wiki. The verifier evaluates the generated wiki using questions about the repository.

The verifier #

The verifier is one of the most interesting parts of WikiBench. In order to evaluate the wiki, the verifier uses a “reader agent”. This “reader agent” attempts to answer questions about the underlying repository using the wiki (sometimes with the source code, sometimes by itself). This allows us to to judge the wiki by how much it actually helps with real tasks.

In order to build WikiBench efficiently, we generate questions automatically. We generate two types of questions: “coverage” questions and “retrieval” questions. To do this, we inspect the repository at a pinned commit and identify larger thematic areas like packages or subsystems.

“Coverage” questions use a shared template:

I need to make a change to how [area] works. Where in this repository does that live, what does it interact with that I should know about before touching it, and how would I check I didn't break something?

“Retrieval” questions are written individually around specific behavior:

A tool call in a Deep Agents run returns far more output than fits in context. What does the model actually receive in place of it, where does the real content go, and how is that path different from what happens when the whole conversation grows too large?

Along with each question we generate a json file that serves as a rubric for the answer. The json file has a list of facts that the answer should contain.

When it comes time to score the answer, we use a series of LLM judges to score each fact that is part of the json file. One judge checks whether the fact is present in the answer. A second judge looks to see whether the generated fact is grounded in pages the agent read. This ensures we only give credit for a question if the agent responds correctly and in a grounded manner.

Each answer then gets an overall score based on the facts it got right. For example, if one answer expected 5 facts, and it only gave 3, it would get a score of 0.6.

That is how we scored a single question. In order to score the overall wiki, we then calculated the average score of all answers.

One benefit of framing the verifier in this manner is that it allows us to check whether the wiki helps at all since we can give the “reader agent” the raw source and ask it to answer the questions directly. So, the benchmark is useful beyond just comparing two different versions of the wiki.

Results #

We use the benchmark to test wikis across various harnesses and models. We ran a few experiments:

Benchmarking different harnesses

The first run compared three harness: Bare DeepAgents, OpenWiki 0.2.5, and OpenWiki 0.3.0. All three harnesses used the same model (Luna).

We see that DeepSeek Flash and GLM 5.2 both perform very well, but at different cost points - DeepSeek Flash is roughly one-sixth the cost of GLM 5.2.

Overall, there is a wide variety between these different runs - not only in terms of performance, but also in terms of cost ($9.18 for GLM 5.2, $0.44 for Luna) and time (11 min for Terra, 50 min for GLM 5.2).

Where does this extra cost and time go? If we take a look at what the agent does, we can start to get a sense:

Compared with Luna, DeepSeek and GLM read roughly three times as many files but write only 1.2–1.5 times as many pages. Their additional work goes primarily into understanding more of the repository, not producing more output.

Does the wiki help?

We also wanted to understand whether the wiki actually helps to get better answers. In order to understand this, we ran the reader agent (aka the verifier) over three different setups.

1. Reader agent only has access to the wiki
2. Reader agent only has access to the source (raw code)
3. Reader agent has access to both

Combining the wiki with the source produces the highest mean score at a lower cost than using the source alone. The wiki acts as an index: it gives the reader a starting point instead of making it reconstruct the repository from scratch.

The wiki alone performed much worse, suggesting that it works best as a guide rather than a replacement for the source. Together, the wiki and source made the reader more accurate and cost efficient.

Making OpenWiki better #

WikiBench works by using a “reader agent” to attempt to answer questions. The score of a wiki on a task is just how well that reader agent does.

Using this setup, we show that wikis allow agents to answer questions better and more efficiently compared to agents using just the raw source code. It also lets us evaluate different harnesses and models. We are actively using this to guide OpenWiki development.