Originally published as an Advisory Hour Substack post on 2026-07-27. This first-party site copy preserves the text, localizes public supporting media, and connects the record to its project trail.
Original post
There’s a looming crisis on the understanding of AI produced software and evals are not going to be enough to contain this threat. Software testing gets you to reliable, but not to understood. “That’s where you’re wrong, Eric. We’ve got Evals!”
The term “eval” is used like a magic incantation from Harry Potter as a salve to these woes. An eval at best is a fence. There are different types of fenceline. They all fall over, eventually. Further, speak to enough devs across companies and you’ll learn they don’t understand their own evals-they haven’t had the time. It’s brutal keeping up.
This article is an exploration of learning rates, forgetting curves, and the coming crisis to understanding of software. It’s a wickedly hard problem. The director or manager of a big firm may already be aware of their simmering problem: “the throughput is lying to us”. This is not an article about understanding every line of code. It’s deeper than that.
AI is turning software production from a construction problem into a metabolic problem. The winning organization will not be the one that generates the most. It will be the one that can digest, govern, discard, and relearn what it generates.
Glimpses of Addiction
The dopamine registers of my brain are fully hijacked by the endless capability of AI. And I’m going to take you into the background lore to situate for you what brought this topic forward. It didn’t just emerge out of the air. Substack is a place of words, after all-and images, too.
This story begins with a contemplation of my website. It’s a vast ecosystem. I’m approaching one-thousand distinct experiments and projects. That’s such an absurdly large number for one person. I built an atlas to even be able to explore it all.
The atlas grew out of all the things I’ve shared online. The directory is a reminder, by the way, that I know I need more hobbies that don’t involve computers. Projects or experiments that reached a reasonable point of done all appear in the inventory of the atlas. Articles I’ve written on topics, like this one, appear there.
Around two hundred or so projects an AI system intuited a topology of what I work on for hobbies. It was a little unexpected to see the consistency. Moment by moment I just compound on the most recent project, and build the next thing that looks interesting to explore.
What might be interesting for you to see, and a little unnerving to me, is that the content reveals a hidden structure to my interests-at least the ones I’m permitted to publicly write about. My coworkers endure extremely long communications from me on topics that aren’t listed in the topology view. Regardless, the view fascinates me. There’s a strange thing about life: you are your own mysterious puzzle.
If you note the topology in the graph view, you’ll see a focal around “Applied AI” simply because the reach and extent of my projects is enabled by AI. I’ve written before that I now view “Applied AI” like “I know how to write email”. And like email, I’m still working to perfect my skill at that, too. A lot of that “Applied AI” is a result of articles like this one-where I explore the ramifications of AI in a given setting.
I do not claim to have it all figured out-it seems like every time I do then a new model releases that has me restart the learning process. What I am claiming is that “Applied AI” has given me leverage of scope. I’m able to tackle projects that I just wouldn’t be able to before. I’ll be posting soon about a new movie in the works, an Ancient Egyptian pyramid exhibit, a WW2 dogfighter, and a robot repairing an island after a storm-all enabled because of AI. Plus, a few others that would make it apparent that I need more time outside than inside.
Applied AI has enabled me to create massive amounts of software. And for a time, life is good in this sun-I pull the slot machine of the next AI generation, and I might get anything from the solution to a math conjecture to a highly detailed 3D model. It’s just too fun, frankly. These are things I’ve always wanted to do, and never could. And now I can pour my endless firehose of ideas into a magic AI machine that just makes it happen.
Long time subscribers will remember the competition I was running over the end of June and early July. The Summer into AI 2026 competition ended a couple weeks ago. Over the course of the competition, there were 91 apps shipped out to the distribution channel of the world wide web. Pause: 91.
You could do everything from play games to watch the viral plague path of bats proceed from the east to west coast of the United States. Ninety-one apps over four weeks-of apps deployed to the internet (I’m not counting those kept private) is an impossibly large number. These were not simple apps without UI or features. These were often fully polished apps, with threaded images, audio and tutorials. The collective shipped more software in that time frame than some development teams deliver in a year. Needless to say, I’ve been giving this a lot of thought.
Let’s remember …
Brains-human ones, are curious puzzle boxes.
I write from time to time on little cognitive hacks I’ve found to improve memory, retention or learning rate. For example, one of my favorites is taking advantage of spatial cues-being aware that your brain will automatically produce a map to something you’re interested in remains a profound unlock for driving new locations and remember weird stuff. The visual system of the brain is magic and we humans haven’t even really begun to unravel what it truly could do.
Perhaps you’ve considered the human avatar’s unusual relationship to reality. We’re all born into this world without a user manual on an F1 key press, let alone a heads-up display. We get vibes and sometimes indigestion as our feedback loop. This life is like setting difficulty up to “nightmare” and seeing what happens. If you’re reading this, then just know you’re doing pretty good. The game goes on.
Still, there’s a sad reality. Even if there was a manual, there’s a good chance you’d forget some parts of it. You’re not just tossed into the world without clear instruction-you have to be very sure you write down what you learn, otherwise it’ll be forgotten.
The Forgetting Curve.
It’s an expected part of being human. This is not a shocking claim: humans forget things-with very rare exception. It is unlikely you recall the fifth word said by the person at the grocery store from four months ago. Human brains just aren’t wired for recall in this way. We forget, and in fact scientists have plotted this all out over repeated experiments. The chart looks like the following.
The chart demonstrates how over time humans forget things, but with spaced repetition you can gradually improve your recall of the material. This is easily testable yourself with a flashcard system like Anki. This curve is at the root of all learning: it doesn’t matter the topic. You may have the spark of insight glistening inside your mind about the problem of AI and this curve, and we’ll get to that. Just a moment more.
The Subscription Capability
Have you ever wondered what all is conceptually possible with the free, $20 or $200 a month AI subscription? That is to say, if you could use every single token then when might be the upper bound of how many things you could reasonably produce? The answer to this question depends on initial variables, but the theoretical max is largely understood as the following.
These apps are not MMOG with thirty years of content, let’s be clear. However, it’s an app that solves a problem. And so if you have the right focus, and determination, and perhaps the right kind of setup… then you, too can optimize a slopfactory and produce a torrent of apps, books or blog posts online.
The sheer scale of this raises all sorts of questions. I wouldn’t blame you for thinking these numbers seem a little extreme or perhaps even wrong. Let’s just use the listed numbers then as directional. The AI tools allow you to ship a lot of software, books or posts. The problem is you won’t remember much of it-let alone understand it.
I didn’t include a spaced repetition in the chart. I did this because, sadly, most people are not learner types. The competitive edge of a learner is very apparent, however. If the above represents the traditional “I show up, do my job, then forget about it”, then the below is the person that takes an active, learning based approach to their job. If mastery of a profession is related to what you’ve learned about it, then you can see here this can help-but this is still one person’s output.
That’s to say, you might remember what you do.
This isn’t an argument that you, the individual programmer, understand every line of code. The difference here is production rate. We used to accumulate misunderstood code slowly enough that incidents, maintenance, and employee turnover exposed the gaps. Some firms even require absences of key team members to expose these gaps (or the fraud). Now one engineer can generate an organization’s previous annual output in a month. A linear governance process cannot manage an exponential intake-the firm is broken. Let me prove it with an exploration of team dynamics.
Team Dynamics
I regret to say that large enterprise teams make this problem worse-if you’re allowed to use a more capable model, that is. If you’re on a team with reduced capability, then it becomes clear very quickly that it’s like you are a middle school basketball team competing about Michael Jordan’s Chicago Bulls. Still, it’s not all roses if everyone in your firm has access to the most advanced of AI. In fact, it compresses the timeline of this problem becoming visible.
Let’s take my first example of the complexity added over a 52 week period that’s constrained by evals or human testing. The simplified chart below shows exactly this-and why I declared that Evals are not enough at the start of this essay.
Observe that you reach a point where the system itself overcomes what your human team is capable of doing. This isn’t terribly surprising. It’s why companies that shed their most senior technology members often shoot themselves in the foot in such a terrible way-whatever chance they had at understanding their own internals is gone forever.
Some of these effects can be mitigated with a team-based learning.
There’s books and books written about management, people and cultivating learning at the team level. It can even mitigate the effects of one individual whose employment propensity is summed up as “at least I showed up”. And there’s a fair argument that not all organizations need learning types-or that there’s even much to learn. I’d challenge that argument, but my focus today is on the intersection of those companies that sit in this knowledge work structure. If you’re reading this and your head is nodding, then you are who I am writing for.
Let’s take a moment to understand this next chart.
It plots out the impact at the team level of human memory, learning and production rate of these AI systems over time. This is a 30 day window. In the top we can see the impact of personal memory versus the shared team memory-assume the team is putting the work in.
The bottom of this chart shown is a useful tokens impact and how it relates to enterprise output versus what is understood. The Free, Plus and Pro are different AI subscription output capabilities so you can judget their impact-recall the chart a little earlier in this essay where I showed the output rates of these different subscriptions. Observe that you can, at scale, very quickly reach a point of no return. Once crossed, you’re not going back without a shock or change to the system. This raises a necessary question: to what extent do you need to understand how your system works?
Now let’s expand this at scale over the next 12-18 months. The top of the chart shows the dramatic (and if I could play background music as you read this, I would) of the impact of learning versus not learning. The middle chart shows the effective vibe of these AI tools over time: a linear path up and to the right. Then over time, the bottom graph, is the Domain of the Unknown.
The enterprise instinct-and I say this as someone who has worked in a great many kinds of them- is to buy another tool to try and make sense of the code or systems. We have a wiki! We have a LMS! However, it’s ironic: the tooling needs to be learned, and it has to be adopted by the team. Further, it has to use human learning approaches-which none of the software discovery tools do.
Instead, such tools too often focus on search and interpretability. Is it better than nothing? It might be worse-the humans aren’t forced to learn. Using such a system might even be dangerous if implemented poorly. Why?
The firm just handed two of the human brain’s powerhouse features to the machine.
What should be understood?
A firm ought to be intentional on what amount of cognitive debt is acceptable. Too many firms never entertain the question. Consider the spread: 0% would be total comprehension of every aspect of the system. 100% would be the complete opposite. If you had to rebuild the firm tomorrow due to some disaster, could you? Not the physical components-the stuff in-between.
In all seriousness, a firm may need to even entertain the most dangerous of questions: should we block a PR if it isn’t learned and understood by at least two humans? Of course not. You’ll never ship a single line of code again if you embrace this. And because that’s the choice, it exposes the formula:
Cognitive debt=deployed behavioral complexity−verified human control
It doesn’t make it easy-choices have consequences.
A firm that maximizes on output and leaves comprehension to a secondary category may soon discover the market will revolt. Success in the market has two platinum rules, perhaps you’ve heard of them. Know thyself; know thy customer. My claim here on a market revolution against the firm is easy to understand with a thought experiment.
Let’s say you own a restaurant. The economics of the business have been challenging and payroll is your highest expense. In this telling, you fire the entire staff and replace them with new hires. You weren’t overstaffed before, if anything, you were under-staffed. But now you have new hires, paid less, and more of them.
You tell the hires, “It’s just cooking and customers, you’ll figure it out. We need to get people in and out as fast as possible.” Your long-term customers come in at the open expecting their usual meal.
How do you suppose this story concludes?
The market might tolerate the friction: if the pricing is low, the customers dependent, or if you’re fortunate enough to be in a protected class of software-a regulatory moat, then you can embrace the mighty power of the regulator’s pen.
The restaurant example is provided here to highlight that this isn’t a technical problem, but a brain powered one. The cookware still functions. The plates still hold food. The tables still could be sat at. What changed in the story is the cognitive aspect.
In a knowledge company, engineering leaders may reach for “technical debt” as the colloquial phrase to deploy. This is the wrong term. The restaurant’s technology was just fine.
Cognitive debt is the right phrase. Technical debt implicitly assumes that someone understands the system enough to fix it-consider that for a moment. Often, technical debt is used by people who don’t understand why a system works the way it does, why the sloppy human written hack is the best solution, and misreads their lack of comprehension as a technology based one. And when I say often, I mean, too often.
If you’re a list-type person, here’s what I’d suggest writing down as your whiteboarding assignment. You can take the cognitive debt aspect and map it down further into three types of understanding.
Understanding what the software is intended to do.
Understanding how its major mechanisms work.
Understanding every implementation detail.
Working with this list, if you take even 10 minutes to try the exercise, will open your mind in a way that may even shock you.
Understanding is not a Markdown File
Software organizations keep treating understanding as though it lives in documentation. It lives in people who know which documented rules to ignore, when to become suspicious, and what “normal” feels like. I’ve written before about the concept of Metis (Μῆτις), which I first encountered in the book Seeing like a State. The way I’d explain Metis is that it’s the delta between riding a bike, and reading the owner’s manual for a bike.
Too many organizations assume they can document the knowledge into markdown files, wiki articles, or email. That documented knowledge is understanding.
If this is true, then explain a library?
Navigating Uncertainty
There’s far too much in the “what’s next” category than fits in one post, but I’m glad you’re taking a moment to entertain these dangerous concepts. I have two additional ideas I’ll share here as heretical. These are heretical because they fly in the face of traditional enterprise, how people are graded in firms, and the flow of organizational power. I’ll note them now as a short list of heresy.
Disposability
This really deserves its own article-but I’m capping myself to one long form article, followed by daily fun project here on this substack. If you enjoyed the long-form article, be sure to use the like button so I can calibrate my writing behavior accordingly and map out how often the long-form article should be released.
Disposable software becomes more common-and should be embraced as a software architectural maxim. The relationship between an app and your average social media post is a lot closer than you might be comfortable to admit.
There’s a tremendous power in just letting go of things. Does it challenge the entire notion of SaaS? You bet it does.
Assume failure by design
What happens if several junior teams are tasked with building an important enterprise widget to solve a problem at scale? There’s failure. The task is to figure out how to de-risk the failure out of the system. When no one understands what the software is doing, this is even more true. The unsexy parts of the software become the most important parts. The essentials are the most important part of any endeavor.
Question that can’t be asked
This masks a question that a firm needs to ask-but just asking the question is heretical at the highest of levels. A question that’s challenging and difficult, whose answer if gotten wrong terminates the economic mandate of the firm itself. I enjoy these questions. They’re like magic spells in a game-the potent, high-end ones.
Would executives accept slower visible output in exchange for deeper invisible capability?
Somewhere a cynic readies an answer. In a world where speed of delivery is viewed as “the answer.” the question is dead on arrival. What if, however, that invisible capability could be made more visible?
But wait, one more thing
Is this an essay on understanding, comprehension, and learning, or is it one about trust? As a subscriber, consumer or customer of a system, we implicitly trust that the business we transact with maintains the promise of its brand. There’s a relationship of trust to learning. This is true even as an individual. My own brand is a high output builder that explores topics with a veteran’s lens, and a fun showcase (if not relentless) daily post of what technology can (or can’t) do.
More than one AI system has told me I tend to have “Eric shaped” writing that blends mischief and seriousness. Expect more of the same as a subscriber.
One last thing…
Here’s an infographic that summarizes the entire essay as an easy to learn infographic.
Connected work
- Cognition: Anki Flashcards: See how spaced repetition changes the forgetting curve for one learner.
- Frontier Atlas: Browse the large public project catalog discussed in the essay.
- Leadership OS: Continue into the leadership practices that shape how teams learn and operate.