Agentic Kernels in Production
TL;DR: We’ve built an agentic kernel development framework that identifies model-level optimization opportunities, generates improved kernel...
Topic
#model-optimization
13 篇笔记
KV Cache Engineering for LLM Serving, clearly explained
Everything you need to understand why the KV cache grows, the 12 ways models and serving engines reduce it, what each technique actually sav...
AI Chip Architectures
A look at AI Chip Architectures. NVIDIA, AMD, TPUs, Trainium, Groq, Cerebras.
Optimizing for cost and intelligence
Balance cost and intelligence on the Claude Platform, with measured results for prompt caching, effort, model choice, budgets, and multi-model strategies.
Profiling LLM Inference with SGLang and Torch Profiler
This blog is my attempt to learn how to profile an llm served using sglang and its in built torch profiler integration that comes out of box...
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across N
Your KV Caching Is Broken
A practitioner's guide to KV cache management. How a modern caching architecture cuts input token costs by 90% and speeds up LLM inference b...
Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction
📝 摘要 Google Research 发布了一种将 Multi-Token Prediction (MTP) 增层改造（retrofit）到已冻结的 Gemini Nano v3 生产模型上的方法，在不修改基座模型权重的前提下，大幅加速 Pixel 设备上的端侧 LLM 推理。该方法已在 Pixel 9 和 10 系列实机部署，使 AI Notification Summaries、Proo…
Optimizing distributed AI inference: Advanced deployment patterns
Learn about the three optimization levers for distributed AI inference: prefill/decode disaggregation, KV cache strategy, and speculative decoding.
vLLM + LMCache: A Starter Guide, No GPU Required
Get started easily: a single MacBook is all you need to develop vLLM + LMCacheFor New Contributors · Covering Frontend / L1 Eviction / L2 Storage / Observability If you ever skipped LMCache because you didn’t have a GPU on hand, this guide was written for you. LMCache’s multi-platform framework has already decoupled the GPU […]
Designing distributed AI inference: Core concepts and scaling dimensions
Learn about the five-dimensional design space in modern LLM serving, including tensor, pipeline, expert, data, and context parallelism.
Editing Files at 1000 Tokens per Second
A new model and inference method for high-accuracy full-file edits at 1000 tokens/s.