DEV Community

Felipe 0liveira
Felipe 0liveira

Posted on AI-assisted

Why KV-Cache Memory Tricks — Not New Math — Drove This Week's Biggest Inference Wins

This digest covers LLM inference and serving developments from 2026-10-28 to 2026-11-04. The pattern this week is hard to miss: almost every serious throughput or cost win came from rethinking how the KV cache and expert weights are managed in memory, not from new attention math. Speculative decoding also got a round of re-tuning specifically for mixture-of-experts and hybrid-attention models, which behave differently from the dense models it was originally designed around.

🔥 Highlights

  1. Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading — up to 19x throughput by merging KV cache and expert-weight memory into one pool.

  2. Day 0 support for Mistral Large 4 on vLLM — speculative decoding, NVFP4, and KV offload shipped for a 1T-param model on release day.
    Day 0 support for Mistral Large 4 on vLLM

  3. HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models — speculative decoding re-tuned for hybrid MoE architectures.

  4. Sliding window beats linear attention, but only on short answers — quantifies the real KV-cache tradeoff for long-context serving.
    Sliding window beats linear attention, but only on short answers

  5. "Weight Space Programming" cuts Autoresearch out of the RSI loop — a 1M-token-context model squeezed onto a 16GB consumer GPU.
    LLMs can read & write neural network weights: "Weight Space Programming" cuts Autoresearch out of the RSI loop

arXiv (cs.LG, cs.DC)

  • HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models (2026-11-02) — Instead of a fixed verification length, HybridSpec picks it dynamically per step using draft-acceptance estimates and measured target latency, because hybrid Gated-DeltaNet+MoE models see verification cost explode with length far faster than dense models do (71% growth vs. 4-6% from length 4 to 16). It also adds block-parallel GDN kernels that reconstruct recurrent state in one launch instead of per token. Reports 1.75-1.88x over plain autoregressive decoding and 1.3-1.33x over the prior best speculative method — directly relevant if you're serving Qwen3.6-class hybrid-attention MoE models and tuning speculative-decoding parameters.

  • Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models (2026-10-28) — An empirical study across 10 sparse MoE models (30B-1T params) finds that fine-grained MoE breaks the textbook rule that speculative-decoding speedup peaks at mid-batch-size; it instead peaks at low batch sizes, traced to expert-routing overlap across draft tokens plus H100 memory-bandwidth and L2-cache reuse effects. The authors' max-entropy model predicts unique-expert counts within 4% error. Useful if your production speculative-decoding speedup curve doesn't match what the dense-model literature predicts — this explains why, and gives a way to model it for capacity planning.

  • PDScope: Breaking the Prefill-Decode Seesaw for Memory-Bound MoE Inference (2026-10-29) — For CPU-offloaded MoE inference, PDScope decouples prefill (compute-bound, wants aggressive expert prefetch) from decode (I/O-bound, wants conservative PCIe transfer) using phase-specific expert-activation prediction and a compiled, zero-copy async execution pipeline. Reports 2.5-10.3x prefill throughput and 2.1-6.1x decode throughput gains over existing systems — a concrete recipe for running large MoE models without keeping every expert GPU-resident.

  • Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading (2026-10-29) — Built on top of vLLM, Rimuru replaces fixed memory boundaries between model weights/experts and KV cache with a shared page pool, per-tensor-type "memory views," and a page-table-aware fused MoE kernel that reads non-contiguous expert weights directly. Reports up to 19.03x throughput and a 98.1% average TTFT reduction versus baseline — a serving-framework-level change applicable to any vLLM-based MoE deployment under fluctuating load.

  • ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference (2026-10-30) — Identifies that KV-cache transfers in prefill/decode-disaggregated serving act as long-lived "elephant flows" that stateless ECMP routing can collide onto the same network spine, stretching transfer time by seconds. ARK is a sender-coordinated path-reservation scheme that spreads concurrent transfers across spines with no switch changes. Cuts mean/P95 flow-completion time by up to 27.3%/34.0% and mean TTFT by 12.9% in RDMA simulation — a network-layer fix easy to miss when optimizing disaggregated serving purely at the scheduler level.

  • ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration (2026-11-02) — A benchmarking tool that predicts serving performance from a handful of "anchor" measurements (one per distinct kernel set) and analytically scales to other node counts and configurations, instead of exhaustive trial-and-error. On AMD MI355X with vLLM/SGLang/ATOM, 11 anchors (9.1 GPU-hours) replaced testing 226 configurations (204 GPU-hours) with 8.1% median latency-prediction error, landing a top-5 shortlist within 12% of the best feasible config — a direct cut to the cost of deployment config search.

vLLM blog

  • Day 0 support for Mistral Large 4 on vLLM (2026-10-29) — Covers vLLM's stack for serving Mistral Large 4 (1T params, ~49B active, MoE with latent attention and cross-layer KV sharing): FP8/NVFP4 quantized checkpoints with FP8 latent KV cache (NVFP4 ~1.1-1.2x faster decode at TP8, within 2 points of FP8 on evals), a built-in DSpark speculative decoder (~2x throughput at TP8), decode/prefill context parallelism (~7.9x KV capacity, up to 1.6x lower TTFT), NIXL disaggregated prefill/decode on GB300, and HiSparse host-memory KV offload (4.5-5.7x token capacity). Gives concrete, measured levers for anyone sizing a Mistral Large 4 deployment on day one.

Hugging Face blog

  • Welcome Le Chonk: Mistral Large 4 on Hugging Face (2026-10-29) — HF's companion piece to the vLLM post, with vLLM serve flags (tensor/context/expert parallelism, batching limits, prefix caching), GGUF quantization levels (Q2_K through BF16) with llama.cpp memory/throughput figures on Apple Silicon and multi-GPU, and Inference Endpoints benchmarks on 8xH200. A concrete menu of deployment paths with hardware-specific cost/performance figures for teams deciding how to run this model.

  • A Low-Cost Decision Model on CPU: Optimizing DiffusionGemma with vLLM - Part I (2026-10-29) — Serves a 26B-A4B decision model on CPU (Intel AMX, BF16) via vLLM instead of GPU, including a new single_pass_reads flag (upstream PR in review) that merges prompt and answer-scoring into one forward pass, cutting latency 18.8% at concurrency 1 and raising throughput 5.5-15% at concurrency 2-20. A 24-vCPU node (~$996/mo) cuts always-on cost 79% versus an A100 replica (~$4,754/mo) with 5-13% lower cost-per-1000-decisions under moderate load — useful for teams deciding CPU vs. GPU for latency-tolerant classification/decision workloads.

  • Sliding window beats linear attention, but only on short answers (2026-11-02) — Compares sliding-window attention against a hybrid linear-attention mixer ("Bifocal") at equal KV-cache memory budget, distilled from Qwen3-4B. At matched cache size, sliding-window wins on short answers but Bifocal pulls ahead on long math answers (15.5% vs 13.8% pass@1 on AIME/HMMT) thanks to retained long-range state; on H100, both hold ~195 tok/s decode from 1K-128K context while full attention drops to 92 tok/s at 128K. Concrete guidance for choosing attention/KV-cache architecture for long-context serving — sliding window is a cheap strong default, but the hybrid can buy better long-generation quality at a small prefill-speed cost and no extra decode cost.

Together AI blog

Nothing new in this window — the latest post predates it and was already covered.

Modal blog

Nothing new in this window — checked the blog index and sitemap directly; the most recent post predates the window.

Latent Space

  • "LLMs can read & write neural network weights": "Weight Space Programming" cuts Autoresearch out of the RSI loop (2026-11-03) — Buried in the daily roundup: a patched Strata inference engine runs DeepSeek V4.1 Flash with 1M-token context on a consumer RX 9070 XT (16GB VRAM) via KV-cache splicing of stable prompt segments plus resident-only expert routing, dropping cold-start from ~100s to ~5s and SSD reads from 58GB to 1.3GB per prompt. Also notes a 763B MoE quantized to ~1.05 bits/param (72.1GB) losing only 1-4 points on LiveCodeBench/SWE-bench. Concrete evidence that aggressive KV-cache tricks can make huge-context inference viable on consumer hardware, and a data point on how far sub-2-bit quantization can go before quality degrades.

  • "Google admits AI Mode hurts Search revenue; -15% $500B drop worst since 2004 IPO. AI Bear Market coming?" (2026-10-31) — Inference-relevant items inside the roundup: prompt caching cuts a typical agent task's cost 3.6x ($27 to $7.44), with each cache invalidation adding ~5% to the bill; Nemotron 3.5 Super hits 4,856 output tok/s/GPU on GB200 at NVFP4 while keeping 99.62% of BF16 accuracy; and turbopuffer's switch from FIFO to LFU chunk caching roughly doubled byte hit ratio and halved p99 latency. Hard numbers for justifying prompt-caching investment and NVFP4 as a near-free accuracy/throughput trade.

  • "Building AIE CODE: The top Coding Agent Conference" (2026-10-30) — Inference-relevant items inside the roundup: vLLM's day-0 Mistral Large 4 support (see above) with speculative decoding yielding 1.92-2.23x output throughput; SGLang kernel fusion on GB300 NVL72 nearing 1,000 tok/s/user; a vLLM tuning pass nearly doubling Nemotron 3.5 Super VL throughput (6,058 to 11,840 tok/s on one GB300); and SkyPilot's in-rack spare-node scheme recovering ~27% extra usable capacity on a 128-rack cluster. Multiple concrete throughput-tuning levers with before/after numbers for serving cost decisions.

  • "Gemini 4 Carbon tops leaderboards, launches for cyber defenders." (2026-10-29) — Inference-relevant items inside the roundup: the AKV serving engine's cache "drop/repos" operations cut agent cost 27.7% and improve P95 latency 1.51x; SGLang's HiCache CPU-DRAM KV offload adds 20-40% more cacheable tokens under concurrent agent load on B200; MLX kernel updates make batched decoding 2-3x faster; and Modular open-sourced its full MAX serving stack (compiler, runtime, hardware backends) royalty-free. KV-cache offloading and engine-level cache ops are now showing double-digit cost/latency wins, and a major serving stack going fully open-source changes the framework-selection calculus.

Through-line

Three independent threads converge this week: the KV cache has become the primary lever for both cost and latency (elastic expert/cache memory pooling in Rimuru, network-aware transfer scheduling in ARK, host-memory offload in the Mistral Large 4 launches, cache "drop/repos" ops and HiCache offload surfacing in Latent Space's roundups, and consumer-GPU KV splicing for million-token context); speculative decoding is being actively re-derived for MoE and hybrid-attention models rather than assumed to behave like it does on dense transformers (HybridSpec, the sparse-MoE speedup study, DSpark on Mistral Large 4); and quantization keeps pushing further than expected, from NVFP4 holding near-BF16 accuracy at scale to sub-2-bit MoE checkpoints still clearing benchmarks within a few points. None of these are algorithmic breakthroughs in attention itself — they're memory-and-scheduling engineering, and that's where the throughput is coming from right now.

What's catching your eye this week — the KV-cache offloading trend, or the MoE-specific speculative decoding retuning? Drop a comment below.

Top comments (1)

Collapse
 
max_quimby profile image
Max Quimby •

The "it's memory management, not new math" framing matches what we keep seeing in practice — the attention kernel is rarely the bottleneck once you're past a toy batch size; it's how much KV you can keep resident before you start paging or evicting. The Rimuru result is interesting because merging KV and expert weights into one pool only pays off if your traffic is bursty enough that the two don't peak together — on steady high-concurrency MoE serving we've found expert-weight residency and KV pressure do collide, and the unified pool just moves the eviction fight somewhere less visible. The batch-size inversion for speculative decoding on fine-grained MoE is the detail I'd flag for anyone tuning this: the dense-model intuition that speedup peaks at mid-batch is actively misleading, and if you tuned your draft length there you left throughput on the table. Did the digest say anything about how HybridSpec's dynamic verification length behaves under mixed prompt lengths? That's usually where adaptive schemes get fragile.