Local vision tools: Eyeballing early build of Independence Town

I’ve written before on the use of Local AI for vision tasks and why this is such a great unlock for your agentic loops. It can be explained simply with a casual algorithm example.

1. Your agent needs to build a thing.
2. You have a picture of a thing.
3. The agent gets the token description of the thing.
4. The agent then builds a thing.
5. The agent then reviews what was built to compare it to the thing.
6. Additional clarifications are needed, so the agent returns back to step 3.

This works great because you save in a few different ways:

1. The inference compute
2. The data transmit
3. The “agent restarted it’s tool call chain

So instead of those tokens going right to the credit card, they go to a battery powered device that hums along softly in my living room.

Overall I’ve been impressed-enough that I’ve become a reasonable fan. I’m fan of these local AI vision for quality engineering tasks right now-I say right now because who knows what AI in a year will bring us. It’s hard to imagine free inference going out of style.

Here’s some logs from my local web UI tooling. This will give you a sense of the kinds of things you can start thinking of when you use local vision, and why it’s so darn useful.

I run each image thru multiple segmentation approaches, and then each of those I again re-process. Normally when you get back an image, say from a vision model from a public API you’ll get back one response. Imagine though if you got back 14-32? Fine grained detail on every little slice of the image to get a bearing on what’s working or what isn’t. And as segmentation approaches get better and the vision model i can use locally get better? The text I have off the images improves, too.

A significant amount of Independence Town worked in that vision loop. I’d like to think this is, in part, why the visual quality is as high as it is for such a fast two-day build. If you’re not familiar with independence town, check the submission here.

Local vs Paid

As with all things, there are tradeoffs with local ai-be it quality, speed, or accuracy. Since a lot of the work I’m doing isn’t necessarily time restricted, I’m ok with things taking a little longer if it means the cost of some actions get absolved.

I tossed my swagger API docs for the vision model over into gpt-image-2 to generate an explainer graphic. None of my local image models are capable of this task. This is, perhaps, a good example of the capability gap. While local ai can solve many tasks, it doesn’t do all of them-but really, which ones do you need?

These are the tasks I have setup on the jetson, which I’ll share more about the special API I built for that tomorrow. Turns out these tasks will get you pretty far down the road!

I mentioned earlier in this article that tomorrow I’d be writing on the Jetson API. It’s so that if you want to build something similar to this, you can point your coding agent at the article to have the structural architecture to build the same sort of thing. Until then!