Local Inference Puzzle

I wrote about how I was tracking benchmarks to figure out my cost savings. Along the way I discovered something. Clocking benchmarks of local inference to make sense of the total tokens a second by target architecture-am I running something on NPU, GPU or CPU? What came back surprised me and revealed I have a lot more work to do. Something is… off.

Consider the result of a cold/hot pass, shown in the image below. Cold means loading the gemma model up from scratch then then tossing an inference task to it. Hot means keeping the model loaded up. What doesn’t make any sense to me is why the CPU is winning in my setup: hence the reason why I have more work to do.

After all, if I can effectively just use CPU bound inference to get work done then why bother with a GPU? Is there a problem in my benchmarking tools? Or could it be something else? Here are my current theories:

1. It’s a problem in my benchmarks.
2. It’s a problem in how I’m loading gemma up cold vs. hot.
3. CPU nodes are getting loaded in the ONNX graph even when I’m not expecting them to. (I need to build a visualization tool, similar to what I learned with the debug tool for 1999 EQTS)
4. The ONNX model may be using CPU kernels not GPU or NPU ones.
5. Something I’m doing wrong with DirectML?

While it may not look great-it’s a good sign in my eyes. It means I have a lot of room to find even more performance! And now I have some benchmarks to show whether I’m improving or not improving the overall system.

Shareable Lessons

There’s three lessons from this for me. I’m going to share them with you here. The idea of a problem being opportunity, the importance of visualization of work being done, and then the importance of measurement.

First, and let’s not waste time and get into the lessons: just because something isn’t working isn’t cause for a concern-it might be cause for celebration. The directional capability of Gemma for vision tasks is already proven several times over for me. The fact I may have more opportunities to sneak out faster runtimes is significant. It’s like driving to a restaurant and finding out the door is closed, but then you take a chance on a taco truck and your mind is opened to new experience.

Until writing this article it didn’t even occur to me that I had a second learning to share, let alone a third. However, building a visualization tool for what the pipeline is doing is architecturally important: you need to see where workload shifts off the target platform. After all, as a mere mortal I cannot read thousands of logfiles one grep at a time and hope to make any sensible progress. And really, why fight thru it? Instead, it’ll make far more sense to just have Codex or Claude Code build a fun threejs visualization tool so I can watch what happened. I can’t wait to share what that ends up looking like!

Third, these insights and opportunities wouldn’t have even come together if I hadn’t taken two additional steps. First, I built an evaluation measurement harness to make sense of the performance characteristics of my harness. It started because I wanted to learn about the NPU, and how it was different from a GPU or CPU. And then I used the benchmarks from that evaluation to write this essay. Had I not done so, I wouldn’t have generated the insights around those measurements.

Too many corporate boardrooms and meetings pull up dashboards and people sit and stare at the charts, the numbers, without really processing them. It’s the same phenomena that I wrote about in the problems of letting AI do your writing. If you want to start generating solutions: one must engage the brain.

Thanks for reading! Subscribe for free to receive new posts and support my work.