KV Cache Engineering for LLM Serving, clearly explained
Everything you need to understand why the KV cache grows, the 12 ways models and serving engines reduce it, what each technique actually sav...
Topic
#long-context
3 篇笔记
Efficient Decode Context Parallelism with vLLM for Long Context Workloads
Decode Context Parallelism (DCP) in vLLM shards KV cache across GPUs by sequence dimension, enabling 3× higher throughput on long-context agentic workloads comp
Editing Files at 1000 Tokens per Second
A new model and inference method for high-accuracy full-file edits at 1000 tokens/s.