Agentic Kernels in Production
TL;DR: We’ve built an agentic kernel development framework that identifies model-level optimization opportunities, generates improved kernel...
Topic
#quantization
10 篇笔记
KV Cache Engineering for LLM Serving, clearly explained
Everything you need to understand why the KV cache grows, the 12 ways models and serving engines reduce it, what each technique actually sav...
AI Chip Architectures
A look at AI Chip Architectures. NVIDIA, AMD, TPUs, Trainium, Groq, Cerebras.
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
LLM Inference: Past, Now, and Future — Where Does the Value Move?
LLM Inference — Past, Now, and Future: Where Does the Value Move?For the past two years, LLM inference has been one of the most contested la...
Kimi K3 Is Here: Efficient Day-0 Support on vLLM
vLLM delivers day-0 Kimi K3 serving with hybrid KDA prefix caching, DSpark speculative decoding, production-scale disaggregation, and optimized kernels across N
Optimizing distributed AI inference: Advanced deployment patterns
Learn about the three optimization levers for distributed AI inference: prefill/decode disaggregation, KV cache strategy, and speculative decoding.
Designing distributed AI inference: Core concepts and scaling dimensions
Learn about the five-dimensional design space in modern LLM serving, including tensor, pipeline, expert, data, and context parallelism.
从 305 GB 到 7.4 GB：大模型 KVCache 架构演进全景
五一本打算更新一年前写的开源引擎 KVCache 实现文档，但梳理下来发现，光是 KVCache 演进的理论部分就已足够撑起一篇独立文章。在 Agent 改变搜索习惯、重塑软件行业的今天，多轮交互带来的长上下文让 KVCache 持…
本地大模型：从跑起来到跑得好
0. 太长不读很多人玩本地大模型，卡住的地方不是模型太复杂，而是不知道自己卡在哪一层。你以为是在跑一个模型，其实是在同时做五个决定：硬件、内存、Runtime、模型选择、量化。每一层都有坑，每一层都可以单独优化，但它们又互相约束。后面所有问题，基本都能归到这五层里。两条读法：如果...