DEV Community

Pingredsai
Pingredsai

Posted on

Why Your Phone Runs LLMs 80x Slower Than It Should (And What I Found)

Why Your Phone Runs LLMs 80x Slower Than It Should (A Debugging Log)

Tags: on-device inference / llama.cpp / Android / performance
Status: draft


I ran Qwen2.5-1.5B-Instruct Q4_K_M fully offline on a Google Pixel 4 (Snapdragon 855, 2019).

Metric Measured
Generation 0.5 tok/s
First token 1.9 s
Peak RSS 1.3-2.0 GB
CPU 402% (4 threads saturated)

Theoretical limits say this device should do 30-60 tok/s.
It is 60-120x slower. And the CPU is already maxed out.

Here is the full debugging log — including three mistakes I made.


1. Do the math first

Memory bandwidth bound:

model size      1.06 GB
SD855 bandwidth ~34 GB/s
one token reads all weights once
-> theoretical ceiling ~32 tok/s
Enter fullscreen mode Exit fullscreen mode

Compute bound:

1.5B params x 2 FLOPs = 3 GFLOPs per token
SD855 CPU peak ~182 GFLOPS
-> theoretical ceiling ~60 tok/s
Enter fullscreen mode Exit fullscreen mode

Both ceilings land in the 30-60 tok/s range. Measured: 0.5.

Knowing the gap (60-120x) tells you what to suspect.


2. Suspect 1: thread count? (ruled out)

adb shell top -b -n 1 -o PID,%CPU,%MEM,ARGS
Enter fullscreen mode Exit fullscreen mode
PID    %CPU   %MEM   ARGS
17834  402%   24.3%  com.example.localai
Enter fullscreen mode Exit fullscreen mode

402% — four threads fully saturated. Not a threading problem. Ruled out.


3. Suspect 2: missing ARM optimization? (HIT)

This is the valuable one.

llama.cpp does its heavy lifting in ggml. From ggml/src/ggml-cpu/CMakeLists.txt:

if (GGML_NATIVE)
    # probe host CPU, add -mcpu=native
    ...
else()
    if (GGML_CPU_ARM_ARCH)
        list(APPEND ARCH_FLAGS -march=${GGML_CPU_ARM_ARCH})   # <-- key
    elseif(GGML_CPU_ALL_VARIANTS)
        ...
    # neither set -> NO -march at all
endif()
Enter fullscreen mode Exit fullscreen mode

When cross-compiling for Android:

  • GGML_NATIVE is OFF (can't probe the target)
  • GGML_CPU_ARM_ARCH defaults to empty
  • Result: no -march is passed at all

Compiler falls back to the baseline ARMv8-A — no dotprod, no fp16.

Quantized matrix multiplication (99% of LLM compute) degrades to a slow path.

Fix:

arguments += listOf(
    "-DUSE_LLAMA_CPP=ON",
    "-DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16"
)
Enter fullscreen mode Exit fullscreen mode

Result: prefill went from 114 s to 45 s — 2.5x faster.

Better. But not enough.

Verify the optimization actually landed

Don't trust the flag. Disassemble:

llvm-objdump -d libggml-cpu.so | grep sdot
Enter fullscreen mode Exit fullscreen mode
18f558: 4e829420    sdot   v0.4s, v1.16b, v2.16b
Enter fullscreen mode Exit fullscreen mode

sdot is there. dotprod is active. Ruled out.

Still 0.5 tok/s.


4. Suspect 3: mmap pages getting evicted? (ruled out)

Android reclaims mmap'ed pages under memory pressure. Then every weight read falls back to flash:

flash read ~1.5 GB/s
1.06 GB / 1.5 GB/s ~ 700 ms/token
Enter fullscreen mode Exit fullscreen mode

Right order of magnitude for the 2 s/token we saw.

So I forced the model to stay resident:

mparams.load_mode = LLAMA_LOAD_MODE_MLOCK;
Enter fullscreen mode Exit fullscreen mode

Result: still 0.5 tok/s. Ruled out.


5. Three mistakes I made (worth avoiding)

Mistake 1: assuming llama_batch_get_one pins position to 0

I claimed: "llama_batch_get_one() sets pos to 0, so every token recomputes the whole sequence."

Then I hand-rolled llama_batch_init() with explicit positions.

What actually happened:

  1. Reading the header: llama_batch_get_one's pos is nullptr — llama.cpp assigns positions automatically. Not 0.
  2. My hand-rolled version hung the prefill (43 tokens, never returned).
  3. Reverting to llama_batch_get_one fixed it.

Lesson: for perf issues, suspect compiler flags and memory access first — not a detail you "remember".

Mistake 2: using a removed API

llama_kv_cache_clear(g_ctx);   // removed in current llama.cpp
Enter fullscreen mode Exit fullscreen mode

Now:

llama_memory_clear(llama_get_memory(g_ctx), true);
Enter fullscreen mode Exit fullscreen mode

Mistake 3: guessing struct field names

mparams.use_mmap  = false;   // fields no longer exist
mparams.use_mlock = true;
Enter fullscreen mode Exit fullscreen mode

Merged into one enum:

mparams.load_mode = LLAMA_LOAD_MODE_MLOCK;
Enter fullscreen mode Exit fullscreen mode

Common lesson: llama.cpp's API moves fast. Always read include/llama.h. Never write from memory.


6. Where it landed

After ruling out three suspects:

3 GFLOPs per token
2 s per token measured
-> ~0.8% of peak CPU utilization
Enter fullscreen mode Exit fullscreen mode

And it is not a bandwidth wall (~500 MB/s measured vs ~34 GB/s available).

So compute efficiency itself is the problem — beyond "wrong config". Could be Android CPU scheduling, OpenMP synchronization overhead, or platform-specific behavior in llama.cpp.

That is not a one-day fix. And it should not block shipping.


7. What I did about it

  1. Ship it and state the number honestly — README says "Pixel 4: 0.5 tok/s"
  2. Explain the limitation — old SoC + large model
  3. Ask the community for more data — Snapdragon 8 Gen 2/3 results welcome
  4. Move performance work to v2 instead of blocking v1

8. Transferable lessons

Lesson Why
Check compiler flags first -march / dotprod issues are the #1 cause of slow on-device inference
CPU saturated ≠ efficient 402% usage can mean you're running the slowest implementation
Compute theoretical limits The gap tells you what to suspect
Verify with disassembly Flags set ≠ instructions generated
Read the header, don't recall it llama.cpp API churns fast
Set a stop-loss for the project Perf work must not block shipping forever

Appendix: environment

Item Value
Device Google Pixel 4 (Snapdragon 855, 6GB, Android 13)
Model Qwen2.5-1.5B-Instruct-Q4_K_M.gguf (1065 MB)
llama.cpp commit 8a1a9b5 (2026-10-09)
NDK 30.0.16248370
CMake 4.1.2
Build flags -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16, -DUSE_LLAMA_CPP=ON
Runtime n_threads=4, n_ctx=2048, load_mode=MLOCK

Source code

Full implementation (Kotlin + JNI + llama.cpp + Jetpack Compose), MIT licensed:

https://github.com/Pingredsai/local-ai-android


Running the same model on newer hardware? Send me your numbers — that is exactly what this project needs.

Top comments (8)

Collapse
 
alfred_odong_322108a5cc3d profile image
Alfred Odong •

Great debugging log, and the "do the math first" step is the part most people skip. Two related traps from the desktop side that look identical to a slow model and aren't:

  1. Scheduling, not compute. A llama.cpp server I started from a tool that runs at nice 5 got its child processes parked on the efficiency cores under load, and a 3-second transcription took 37 seconds. Same binary, same flags; the only difference was who launched it.

  2. Storage, not compute. Running off a USB stick, the whole model is read at every start, so on a 34 MB/s drive a 2.6 GB model took over a minute before the first token, and seconds on a USB 3 drive. Tokens per second was fine both times; the stopwatch said otherwise.

Your -march finding is the same shape: the number that looks like "the model is slow" is almost never the model.

Collapse
 
pingredsai profile image
Pingredsai •

Thanks — I went and tested both of your hypotheses. Results:

Scheduling: not it (on this device).

I read field 39 (current CPU) of /proc//task//stat while inference
was running. All four inference threads stayed on cpu4-7 (the A76 mid/big
cores) and never touched cpu0-3 (A55). OpenMP is working too — three
openmp_worker threads plus the main thread, matching n_threads=4.

Frequency management: real, but small.

I did find heavy throttling though. Under schedutil, cpu4/cpu6 oscillate
between 710 and 1920 MHz (max 2419), and cpu7 never exceeded 2016 (max 2841).
Battery at 33.9 °C, so it's not thermal — it's schedutil responding badly to
bursty load.

So I rooted the device, chmod'd the sysfs nodes, and pinned all 8 cores to
performance (1785 / 2419 / 2841 MHz):

schedutil 0.5 tok/s first token 1863 ms
performance 0.6 tok/s first token 1619 ms

Only +20%. Real, but not the dominant factor.

Where that leaves me:

CPU at full clock, threads on big cores, dotprod instructions confirmed present
— and utilization is still ~1% (3 GFLOPs / 1.67 s ~= 1.8 GFLOPS against a
~182 GFLOPS peak). Memory bandwidth isn't the wall either (~635 MB/s vs
17-34 GB/s available).

So the bottleneck looks like it sits inside ggml's quantized kernels on this
platform — memory access patterns, cache behaviour, or OpenMP barrier overhead
on short work units.

Your storage point I couldn't reproduce: switching to LLAMA_LOAD_MODE_MLOCK
made no difference, so pages weren't being evicted in my case.

Appreciate the pointers — the scheduling check alone was worth doing, even
coming back negative. It's now a bounded question instead of a vague "it's slow".

Code: github.com/Pingredsai/local-ai-and...

Collapse
 
alfred_odong_322108a5cc3d profile image
Alfred Odong •

Thanks for running both and writing up the negative results; that is rarer than it should be. With scheduling, frequency, dotprod and bandwidth all ruled out and utilisation at 1%, I would stop looking at the hardware and bisect the software in two cuts.

  1. Is it ggml or is it the app? Build llama-bench with the exact same CMake flags, adb push it, and run it on the same GGUF: ./llama-bench -m qwen2.5-1.5b-q4_k_m.gguf -t 4 -p 64 -n 32. If llama-bench does 15-30 tok/s, the kernels are fine and the 0.5 lives in the JNI/app layer. If llama-bench is also at 0.5, it is the build.

  2. Build type. A Gradle debug variant hands CMake CMAKE_BUILD_TYPE=Debug unless you override it, and ggml at -O0 with asserts on is 20-50x slower than release. sdot showing up in the disassembly does not tell you the optimisation level. Add -DCMAKE_BUILD_TYPE=Release to that arguments list, or benchmark the release variant.

If it lands in the app layer, the two things I have seen produce exactly "threads busy, 1% useful work": a decode loop that clears or recreates the KV cache every token, so each token is a full prefill (watch n_past; 2 s/token is suspiciously close to your prefill cost per token), and OpenMP spin barriers across the 3+1 A76 clusters. For the second, -DGGML_OPENMP=OFF to use ggml's own thread pool and n_threads=3 so every thread sits on the same mid cores is a cheap test.

Curious which cut it lands in. I would bet the build type.

Thread Thread
 
pingredsai profile image
Pingredsai •

You called it — it was the build type.

I checked the previous build.ninja: zero -O flags anywhere. Not -O0
explicitly, just none. CMake's Debug config with no optimisation. And the
project was building the debug variant, so AGP was handing
CMAKE_BUILD_TYPE=Debug to CMake.

Fix: force -O3 in CMakeLists (before add_subdirectory(llama.cpp)) plus
-DCMAKE_BUILD_TYPE=Release in the Gradle arguments.

Results on the same Pixel 4, same GGUF, same model:

before   prefill 43 tok = 42 s      0.5 tok/s    first token 1863 ms
after    prefill 43 tok = 0.556 s   18.5 tok/s   first token 52 ms
Enter fullscreen mode Exit fullscreen mode

37x on generation, 75x on prefill.

Also: the .so sizes were a useful confirmation. libggml-cpu.so went from
3.92 MB to 0.95 MB, which is a decent smell test for "did optimisation
actually happen".

Your point about sdot in the disassembly not telling you the optimisation
level was exactly right — that's what sent me down the wrong path for days.
I never checked the most basic thing.

llama-bench is still on my list as the next cut, but this one landed first.

Thanks — genuinely. This saved me a lot of time.

Collapse
 
pingredsai profile image
Pingredsai •

Thanks, this is a great pair of examples — both hit close to home.

The scheduling one is especially relevant. Android's big.LITTLE scheduler is
aggressive about migrating long-running threads, and I measured 402% CPU
(4 threads busy) but never checked which cores they actually landed on.
That's my next test — reading per-thread affinity from /proc, or pinning
threads with sched_setaffinity and comparing.

The storage one matches an experiment I ran: I switched to
LLAMA_LOAD_MODE_MLOCK to force the model resident instead of mmap'd,
expecting a big win. It made no difference (0.5 tok/s either way), which told
me the pages weren't being evicted. But your point makes me want to measure
the steady-state read rate separately from startup cost.

Agreed on the shape of the bug: the number that looks like "the model is slow"
is almost never the model. I'll report back on the affinity test.

Code: github.com/Pingredsai/local-ai-and...

Collapse
 
abhi_research profile image
Abhi •

Great debugging work! Did you test the same model on newer Android devices to compare performance? I'm curious how much improvement comes from hardware versus runtime optimizations.

Collapse
 
pingredsai profile image
Pingredsai •

Thanks for asking! The 37× speedup was on the same Pixel 4 with the same model. Enabling compiler optimizations took it from 0.5 to 18.5 tokens/s. I don’t have comparable numbers from a newer phone to share yet.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.