← Back to Custom Model Training
2026-06-09Model TrainingNPUPrefillBenchmarks

Still Need to Fix the Prefill Operation on NPU

A short local-inference field note on why the benchmark table is interesting but hardware-specific: the selected NPU path is still carrying CPU prompt-prefill work.

Source context: This short field note first appeared on Eric's Advisory Hour Substack. View the original note or subscribe to Advisory Hour.

The field note

Still need to fix the prefill operation on NPU. I felt someone out on the internet would find my performance benchmark numbers more interesting than useful. You would need my exact some hardware profile (snapdragon elite) for it to be useful.
Terminal performance summary comparing GPU and selected NPU browser benchmark runs with rows, pass counts, average total time, browser time, and CPU, NPU, and GPU token counts.
Benchmark output from the local GPU/NPU comparison, with the caveat that the selected NPU rows still show CPU tokens because prompt prefill remains CPU-bound.

Why it matters

The important signal is not that the NPU wins a neat leaderboard. It does not, at least not yet. The benchmark is useful because it makes the routing problem visible: selected NPU runs can still carry CPU token work when prompt prefill is not actually landing on the accelerator.

That turns the next fix into a systems problem instead of a vibe check. If the goal is a genuinely useful local model harness on Snapdragon-class hardware, the path has to separate browser overhead, prompt prefill, decode, graph partitioning, and device selection instead of treating “NPU selected” as proof that the whole workload moved.

What this note points toward

  • Hardware-specific receipts: the numbers matter most for the exact Snapdragon Elite profile that produced them.
  • Prefill as bottleneck: the next measurement target is whether prompt prefill can stop falling back to CPU.
  • Harness-first debugging: useful edge inference needs instrumentation that names where each token and each browser operation actually ran.

Connection to the SLM series

This sits beside Tinkering with a NPU, NPU and Gemma4, and Snapdragon Hexagon: Learning about QNN. The pattern is consistent: small local models only become operationally useful when the harness can prove where the work ran, how long it took, and which fallback path quietly stole the loop.

Original source

This local copy preserves the note text, source link, and inline media from Eric's Advisory Hour. Original Substack URL: https://substack.com/@advisoryhour/note/c-267302619.