DEV Community

#llamacpp

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
llama.cpp on a Cloud TPU: An Unofficial ggml Backend Through XLA

llama.cpp on a Cloud TPU: An Unofficial ggml Backend Through XLA

10
Comments
10 min read
How much does running multiple streams raise aggregate tokens/s on one GPU?

How much does running multiple streams raise aggregate tokens/s on one GPU?

Comments
5 min read
Neykosa phone benchmark: 14.8 tok/s from a 0.6B model on a 4 GB Galaxy A05s, and what that number leaves out

Neykosa phone benchmark: 14.8 tok/s from a 0.6B model on a 4 GB Galaxy A05s, and what that number leaves out

Comments
2 min read
Local Inference: Running Models on Your Own Machine

Local Inference: Running Models on Your Own Machine

Comments
13 min read
llama.cpp 1.0.0 Stabilizes the Local Inference API — Dual-Track WASAPI Capture Still Owns the Latency Budget

llama.cpp 1.0.0 Stabilizes the Local Inference API — Dual-Track WASAPI Capture Still Owns the Latency Budget

Comments
4 min read
Can a small language model run on a budget Android phone with 4 GB RAM?

Can a small language model run on a budget Android phone with 4 GB RAM?

Comments
4 min read
111GB 모델 파일이 램 78.8GB로 도는 이유: POCKET GGUF를 비트 수·KV 캐시·토큰당 읽는 가중치로 분해한 램 예산표

111GB 모델 파일이 램 78.8GB로 도는 이유: POCKET GGUF를 비트 수·KV 캐시·토큰당 읽는 가중치로 분해한 램 예산표

Comments
5 min read
I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests

I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests

Comments 4
2 min read
Jan open source ChatGPT alternative: offline local AI

Jan open source ChatGPT alternative: offline local AI

Comments
7 min read
How to Use a Remote Ollama or llama.cpp Server From a Desktop AI App on Windows in 2026

How to Use a Remote Ollama or llama.cpp Server From a Desktop AI App on Windows in 2026

Comments
6 min read
I Cut Dedicated GPU Memory to 4GB, and a Local LLM Still Ran at 90% Speed

I Cut Dedicated GPU Memory to 4GB, and a Local LLM Still Ran at 90% Speed

Comments
16 min read
Extending llama.cpp to Support Reflection Beam-1.0-501B-A23B: Handling Complex MoE and Attention Architectures

Extending llama.cpp to Support Reflection Beam-1.0-501B-A23B: Handling Complex MoE and Attention Architectures

Comments
3 min read
Running MoE LLMs Larger Than Your RAM: A Practical Guide to mmap, Expert Offloading and SSD Streaming with llama.cpp

Running MoE LLMs Larger Than Your RAM: A Practical Guide to mmap, Expert Offloading and SSD Streaming with llama.cpp

Comments
8 min read
One Word, Two GPUs: Why Gemma 4 Answers the Same Question Differently on a Laptop and a Tesla T4

Hardware differences subtly alter benchmark grades

One Word, Two GPUs: Why Gemma 4 Answers the Same Question Differently on a Laptop and a Tesla T4

30
Comments 7
12 min read
Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

24
Comments 1
12 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.