Local Vision Model: Examples and thoughts

This is a handful of shots from a local vision model doing process work on the microverse project John’s World:1995. Life can be like this. We stare at a rectangle in the evening, either in our hands or on the wall. At work, we’re often interacting with more rectangles. It might be a computer on your desk, a shopping kiosk, or even a clock display. Rectangles layer in our own life, and we’re oblivious to them unless one day you blink twice and take note of just how many rectangles are around you.

The vision model service I’m running locally has a toolkit of segmentation tools. The thought I went with to explore is that building a toolkit, toolbox or buffet of vision options for an agentic coding model to use would be useful to agentic coding tools in helping me to build out virtual world projects. What I didn’t expect is that these same tools are useful reflections for me.

The UI you see below is native to the project. It’s a throw-away debugging tool. I do these more regularly after the write-up I shared about how useful it can be to just ad-hoc build out various tools around your project. It’s not enough to drive AI into a project. The idea isn’t to just prompt an AI to build your idea. Use the AI to help build an ecosystem of tooling around whatever project you’re working on. The additional perspective is incredibly useful.

The image above here, it’s part of a series of captured frames for another captured moment from the ancestor simulation I built to explore a different multi-verse. The rectangular points of interest are bounding boxes that is automated by the system. I am not yet using OpenCV5; it’s on the roadmap for an upcoming weekend.

On the left-side of the navigation I have different segmentation options, including splits to handle SAM and Vision. The reason why this is possible is due to how my local ai and platforming system is setup: it’s a broader platform with API capabilities, and with every model release from the frontier I revisit what can be done: a little faster, a few more capabilities, and an itch that just had to be scratched.

Multi-layered Segmentation

One of the features is SAM-lite. It’s a port of Facebook’s Segment Anything technology. It layers in with my other systems so I can slice up an image in regions of interest, or via specific regional fill zones. This grid is a highlight of what I mean. each image is a slice that’s processed by the image model. Each processed segment has text in a json format that I can then use in a broader pipeline or handoff. It allows me to handoff to the model quite a bit for review.

The perk of this running on my hardware is the cost: it’s free. So even in the imperfections, I’m content with the operating cost.

And since it’s free, I’m seeing a few hundred vision runs per day. The caveat here is that I need to be working on something that needs vision. Vision tokens from third party API can chew up budgets in some loops, and so I’m ok with trading the imperfect slower for the perfectly free. I’m just more inclined for the outcome. That said, new models like Fable are teasing me with the possibility that a new class of performance is possible.

Danny in 1995

If you’re dropping in to this and follow me on using local AI tools for the vision part of the story, but aren’t sure how to make sense of John’s World: 1995 then let me take this moment to share a little bit of that project. John’s World started as an extension of another project, where I have a 4.3 inch display in my kitchen that generates infinite stories in an anime style universe. I had some interest in building out the quality of those episodes. They are interesting, and they’re fun to watch-but I got to wondering what it would mean to have a tool that I could use to navigate around the world.

While walking the dog, I took a picture of a house and used an AI tool to simulate navigation. I was trying to figure out a puzzle: can I simulate direction in a vision or video model? How does that work? I’ve solved that problem, and several more now. The weirdest thing happened while solving that puzzle: a nameless man in a hat emerged. If you go back over my substack history you’ll see prior posts explaining in more detail about this. I called him John, for John Doe. This gave birth to John’s World.

History of John Doe, per Google:

“John Doe” originated as a medieval English legal fiction and has evolved into a widely used placeholder name for unidentified or anonymous individuals.

The name “John Doe” first appeared in medieval English law during the reign of Edward III (1327–1377) as part of a legal workaround called an action of ejectment. Landowners who wanted to remove squatters or tenants faced complex procedural hurdles in property disputes. To simplify the process, lawyers invented fictional plaintiffs and defendants: John Doe as the plaintiff and Richard Roe as the defendant. These names allowed courts to rule on property ownership without directly suing the actual tenant, effectively creating a legal fiction that became standard practice in English common law

At first, John’s World had one art style and one character. However, the world was empty and sad. John needed at the very least a Jane Doe. Once I had two characters in the world, I began to dream bigger. Could I do two different universes? And I could, over time and with a lot of experimentation, trial and error.

Danny lives in universe 1995, which is the year it takes place in. There are others, but the images above are for a kind of historical fiction of an alternative universe. What life might have been like in the 1900s. I find the technology interesting and it’s a fun hobby of mine, on this rectangle that follows me around.

Funny, isn’t it. The humble rectangle shows up everywhere when you start looking for them.

Subscribe now