KV Cache Engineering for LLM Serving, clearly explained
Everything you need to understand why the KV cache grows, the 12 ways models and serving engines reduce it, what each technique actually sav...
Topic
#prefix-caching
10 篇笔记
SemiAnalysis 发布 AgentX InferenceXv3 研报，Mooncake 成为关键基础设施
SemiAnalysis 发布 AgentX InferenceXv3，用大规模真实 Agentic workload 重新审视推理系统性能。随着长上下文、多轮交互和 KV Cache 复用成为常态，Agentic Inference 正从“拼算力”走向系统级优化，而 Mooncake 也正在其中扮演越来越重要的基础设施角色。
Agent 终章（Harness 成本篇）：一次百炼账单降低 88% 实战
关于优化 token，本文或许能够提供一些经验和参考
5 design patterns for long-horizon agent harness
Any agent can look brilliant on a one-shot task. Give it a week of real work and it starts to fall apart."Long horizon" means the agent keep...
How Compaction Works in Pi
Why compaction is needed for large language models and how Pi implements it
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across N
Your KV Caching Is Broken
A practitioner's guide to KV cache management. How a modern caching architecture cuts input token costs by 90% and speeds up LLM inference b...
Optimizing distributed AI inference: Advanced deployment patterns
Learn about the three optimization levers for distributed AI inference: prefill/decode disaggregation, KV cache strategy, and speculative decoding.
Session-Aware Agentic Routing: Continuity-Aware Model Selection for Long-Horizon LLM Agents
How Session-Aware Agentic Routing in vLLM Semantic Router preserves long-horizon agent continuity with session memory, safe model-switch boundaries, prefix-cach
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale.