DEV Community

Cover image for Basalt walkthrough is up: why 776 tok/s needs the whole card to itself
InApp
InApp

Posted on Originally published at imapp.blogspot.com

Basalt walkthrough is up: why 776 tok/s needs the whole card to itself

I kept getting the same question in DMs after the benchmark posts: "How is Basalt actually pulling 776 tokens a second?" Not the number itself — the mechanism. People wanted to see why it behaves differently from vLLM or llama.cpp on the same hardware, so I recorded a full walkthrough instead of pasting another config snippet.

The detail that surprised me while editing the footage: Basalt doesn't just stream faster, it pins the entire KV cache to a single GPU's VRAM and refuses to shard across cards. That's why "only on one GPU" in the title isn't a footnote — it's the core design. On an RTX 4090 it holds a full 32k context window in roughly 18 GB, which sounds fine until you realize that leaves almost nothing for a second model running concurrently.

The real cost to the operator is exclusivity. You can't co-host anything else on that card while Basalt is serving. If you want full-speed inference plus a separate service, you're either buying another GPU or scheduling Basalt alone. I walk through the memory map, the queueing behavior, and exactly where the bottleneck sits in the video: https://www.youtube.com/watch?v=sMvxrrUq9M0

Top comments (0)