Model Training · Source archive

Experiments in training Pico LLMs

Originally published 2026-04-07 on Advisory Hour.

Originally published as an Advisory Hour Substack post on 2026-04-07. This first-party site copy preserves the text, localizes public supporting media, and connects the record to its project trail.

Original post

Experiments in training Pico LLMs image from the original Advisory Hour post.
Experiments in training Pico LLMs image from the original Advisory Hour post.

Elon made a comment recently about how it’s likely we’re all missing the boat on how much intelligence can be squeezed per gigabyte and this sent me down a rabbit hole of training models-and this isn’t exactly earth shattering work here, but it’s fun. Turns out optimizing intelligence per megabyte is an interesting puzzle.

Optimization problems are fun puzzles-they remind me of games. And I like games as a way of learning about how to do things. So, I’ve taken a step into small model building by building an RPG game engine inside a very small LLM.

Let’s take one of my newest models into consideration.

This is one of the trained models on Saturday. It’s using a basket of new techniques organically bolted on from prior runs. The thing that makes this model different from a lot of other models is that it’s super tiny. So tiny that it can run surprisingly fast on a cpu (the model is all of 64 meg). The crazy thing? It works pretty well!

The model is trained on a template setup-well, it has a few templates that it supports-note: a 64 meg model can support multiple templates. This is one of those templates:

Experiments in training Pico LLMs image from the original Advisory Hour post.
Experiments in training Pico LLMs image from the original Advisory Hour post.

The first line is the task. The second is what turn in the sequence we’re in. I’ll describe the parameter list, but if you look closely it’s an RPG list. The task? What happens if P1 uses poison strike and P2 uses ignite, but P1 heals? Answer in a moment.

The structure of the template is straightforward and has a mini-DSL like syntax. DSL if you’ve stumbled into this technical exotic lore just stands for domain specific language that allows you to program concepts with far more ease and flexibility.

Experiments in training Pico LLMs image from the original Advisory Hour post.
Experiments in training Pico LLMs image from the original Advisory Hour post.

This won’t mean a whole lot to you unless you’ve been messing around with RPG systems, so let me try to expand on what these all mean.

Field meanings

  1. sh = shield

  2. psn = poison

  3. brn = burn

  4. reg = regen

  5. th = thorns

  6. stun = stun

  7. frag = fragile

  8. cd0, cd1 = cooldown slots

  9. alive = 1 or 0

Action syntax

The main action forms it was trained on are:

  • atk:x

  • heal:x

  • guard:x

  • cleanse

  • poison_strike:x:y

  • ignite:x:y

  • brace

  • wait

Expected input format

You give it a prompt like this: 

TASK resolve_final_state
TURN=1 ACTIVE=p1
P1 hp=<int> max=<int> sh=<int> psn=<int> brn=<int> reg=<int> th=<int> stun=<int> frag=<int> cd0=<int> cd1=<int> alive=<0|1>
P2 hp=<int> max=<int> sh=<int> psn=<int> brn=<int> reg=<int> th=<int> stun=<int> frag=<int> cd0=<int> cd1=<int> alive=<0|1>
SCRIPT <actor:action | actor:action | ...>
ANSWER

The tiny model then generates the canonical final state block:

TURN=<int> ACTIVE=<p1|p2>
P1 hp=<int> max=<int> sh=<int> psn=<int> brn=<int> reg=<int> th=<int> stun=<int> frag=<int> cd0=<int> cd1=<int> alive=<0|1>
P2 hp=<int> max=<int> sh=<int> psn=<int> brn=<int> reg=<int> th=<int> stun=<int> frag=<int> cd0=<int> cd1=<int> alive=<0|1>
WINNER=<none|p1|p2>

The idea I had here is that the pico sized Small Language Model could function as a fuzzy state machine for game logic. Fuzzy because I think games are more fun when a rule is bent, broken or an error allowed. It’s the “wait, what just happened” experience when the boss has 0.0001 hp and then comes charging at you for the killing blow.

Inferred decision logic

The training data does not include IF-THEN style game logic. Instead, the training data simply presents all the cases and the model has to learn what to apply. The idea I was going with here was building up the game engine rules by just showing the log output. It’s pretty snazzy for the 1 person in the entire world who needs something like this.

Still, I have an entire game engine inside 64 meg. I find that fascinating. Some of these tiny models are in the 25-30 meg range. What if instead of just having a single 1 gigabyte model that tried to do everything, you had a thousand 64 meg models that could be loaded or paged from disk at any moment? It’s an interesting thought.

Let’s explore where the model falls apart.

Weird Failure Conditions

A 64 meg model is not going to wow you with all it’s capabilities. It’s been trained on a very narrow domain problem. The failure modes are just as fun. The first is playing around with the concept of fragility.

What happens if you set frag = fragile?

Turns out the token stream totally collapses and the boundaries completely fall apart. I like to imagine the formless void around non-defined areas of a latent space of a LLM as the white noise you’d see on the old television sets trying to pickup signal. It’s amusing to me that if you set fragile to fragile, the model spirals out and breaks.

Certainly… fragile.

Experiments in training Pico LLMs image from the original Advisory Hour post.
Experiments in training Pico LLMs image from the original Advisory Hour post.

When regen fails

This is a fun example of setting reg=regen as the prompt. That’s it.

Experiments in training Pico LLMs image from the original Advisory Hour post.
Experiments in training Pico LLMs image from the original Advisory Hour post.

What I like about this one is that while it did hit a failure mode of putting player 2 in, it stopped on regen and completely lost track of what was going on. There’s this idea of “label collapse” at play here. Label collapse is, as I understand it, when the model stops distinguishing entities and reuses one label everywhere.

I doubt very much that there’s anything like an interior world space in this 64meg model. But this label collapse is a consistent problem. You can see it here, with poison, too.

Experiments in training Pico LLMs image from the original Advisory Hour post.
Experiments in training Pico LLMs image from the original Advisory Hour post.

So in this poison example you see the entire output prediction just completely falls apart. P1 doesn’t exist because, in the story I’m telling here, P1 disappeared after poisoning P2. What’s interesting to me is that P2 does show a -1 poison damage counter on their HP. Is it broken or… creative?

Experiments in training Pico LLMs image from the original Advisory Hour post.
Experiments in training Pico LLMs image from the original Advisory Hour post.

But Eric, why not just write the damn code?

You might laugh, and yet I’ve had this conversation with many people. Here’s the thing. Yes, determinism has a place.

I’ve written many toy game engines over the years-more than is healthy to count. There’s a thousand, perhaps more, ways to build a game engine with code. But… growing or educating a small mind to be a game engine? That’s a new one. I’m doing that now because it’s neat and oddly fun to grow digital minds. Plus, I have this idea of hosting one of these models on the raspberry pi. I think that’d be neat.

What’s next?

I’m running a confirmation test of a research direction that has done rather well, and so will be leaning in that direction. I’m curious how much of an interior world state I can pack into a tiny model. How much of a universe can you squeeze into one of these? But I’ve gotta wait, my one GPU is busy with wall clock time.

Experiments in training Pico LLMs image from the original Advisory Hour post.
Experiments in training Pico LLMs image from the original Advisory Hour post.

So for now? I’ll be taking the dog for a walk.

Thanks for reading! Subscribe for free to receive new posts and support my work.