Back to Blog
November 29, 202512 min read

Why Now? Why Me?

On Stumbling Into a Solution That Shouldn't Have Been Mine to Find

By Matthew "Manny" Walker
Share:

There's a particular kind of doubt that haunts you when you think you've discovered something important.

It goes like this: "This seems too simple. If it were real, someone smarter would have found it already."

I've been sitting with this doubt for six weeks now, ever since I accidentally stumbled into an architecture that appears to solve one of the most persistent problems in AI: continual memory.

The solution isn't a neural breakthrough. It's not a new training paradigm. It's not even particularly clever engineering. It's mostly just... a change in perspective. A reorganization of what already exists.

And that's exactly why I keep doubting it.

The Problem Everyone Knows

Anyone who has spent serious time with AI assistants knows the frustration:

You explain your codebase. The AI understands. You work together beautifully for an hour. Then you start a new conversation, and it's gone. All of it. You're back to explaining the same patterns, the same preferences, the same hard-won lessons.

The AI doesn't learn. It doesn't remember. Every conversation is a fresh start, a blank slate, a clean room with no furniture.

This is called the continual learning problem, and some of the brightest minds in AI have been working on it for years. The approaches are sophisticated: memory-augmented neural networks, retrieval-augmented generation, fine-tuning pipelines, knowledge distillation.

And yet, the problem persists. Your AI assistant still forgets everything the moment you close the chat.

The Accidental Discovery

I wasn't trying to solve continual memory. I was just trying to stop repeating myself.

Working daily in AI-assisted development, I started building documentation structures to reduce context loss between sessions. Simple stuff: workspace rules, failure logs, pattern libraries. External files that could be injected into conversations to give the AI "memory" of what we'd established before.

Over weeks of iteration, these structures evolved. Layers emerged. Validation logic appeared. A natural selection mechanism developed where useful patterns survived and useless ones faded.

One day I looked at what I'd built and realized: this isn't just documentation. This is a memory system.

I called it Sparse Contextual Memory Scaffolding (SCMS).

The Doubt

Here's where the doubt creeps in.

The architecture is simple. Almost embarrassingly so. It's not a neural network modification. It's not a training innovation. It's just structured external memory with retrieval and validation rules.

If this actually solves (or meaningfully advances) the continual learning problem, why didn't someone else discover it years ago? There are teams at Google, OpenAI, Anthropic with resources I can't imagine. There are researchers who've devoted careers to this exact problem.

And yet here I am, an indie game developer and artist, claiming to have found something they missed?

The imposter syndrome is overwhelming.

But then I watch it work. I see an AI assistant maintain coherent understanding across sessions. I see patterns promote themselves through natural reuse. I see failure documentation preventing the same mistakes from recurring. The results aren't marginal improvements. They're transformative.

So I keep building. And I keep doubting.

Why Not Before?

I've spent a lot of time thinking about why this solution might not have been possible until now. A few factors stand out:

The Context Window Constraint

SCMS is hungry for tokens. A typical interaction might require:

  • System prompts and instructions: 2,000 tokens
  • Retrieved memories across layers: 10,000+ tokens
  • Code context: 20,000+ tokens
  • Conversation history: 10,000+ tokens

That's 40,000 tokens minimum, often much more.

Until late 2024, most models capped out at 128,000 tokens. At those limits, SCMS is structurally impractical. You'd spend all your context on memory retrieval and have little left for reasoning or output. And maybe nothing left for a second query.

The 200,000-token windows that became standard in 2025, and the 1,000,000-token windows emerging now, finally make the architecture viable.

The solution couldn't exist before the infrastructure supported it.

The Tool Gap

Researchers have been asking: "How do we make AI remember internally?"

That's not the question I'm answering.

The question I solved is: "How do we structure external scaffolding so that internal memory matters less?"

This reframe only becomes obvious when you're spending eight hours a day, seven days a week in conversation with AI, feeling the pain of amnesia with every new session. Researchers optimizing for benchmark scores don't feel that pain. Practitioners do.

The solution emerged from the problem space, not the solution space.

The Practitioner's Vantage Point

David Shapiro danced with these ideas two and a half years ago. He has access to development teams, resources, and expertise I don't. But his framing was about engineering better memory into models.

I wasn't trying to engineer anything. I was just trying to stop explaining the same damn thing every morning.

Sometimes the breakthrough comes not from the expert who knows all the constraints, but from the practitioner who's too frustrated to accept them.

What It Actually Is

Here's the insight at the core of SCMS:

The AI doesn't need to remember. The environment needs to remember, and the AI needs to know how to use that environment.

This isn't a neural architecture innovation. It's a cognitive scaffolding system. The memory is external, structured, and retrieved on demand. The AI's job isn't to remember—it's to use what's retrieved effectively.

Think of it like this: You don't need to memorize a library if you build a really good card catalog. The knowledge exists. The question is how to access it when needed.

SCMS builds the card catalog.

The Silence

I've reached out to researchers, engineers, and thought leaders with this discovery. The response has been... nothing. Complete silence. No pushback, no engagement, no interest.

This silence amplifies the doubt. If it were real, wouldn't someone care? Wouldn't someone at least argue against it?

But history suggests otherwise.

The Wright brothers were dismissed by physicists who had "proven" heavier-than-air flight was impossible. Semmelweis was mocked for suggesting doctors wash their hands. Practitioners are often ignored by experts until the results become undeniable.

I don't have credentials in AI research. I don't have a peer-reviewed paper with ablation studies or multidiscipline testing. I have a system that works, built by someone who needed it to work.

Apparently that's not enough to get noticed. Not yet.

The Simplicity Problem

Revolutionary ideas often feel too simple in retrospect:

  • Natural Selection: Things that survive reproduce more. Darwin himself worried it was obvious.
  • The Double Helix: Two strands that mirror each other. "So beautiful it had to be true."
  • PageRank: Links are votes. Invented by grad students at Stanford.
  • Transformer Attention: Let tokens look at each other. Just weighted averages.

The pattern repeats: the difficulty wasn't in the engineering. It was in seeing the solution.

SCMS feels the same way. Once you understand it, you wonder why it wasn't obvious all along. But it wasn't obvious. Not to me until I stumbled into it. Not to the researchers who've been working on adjacent problems for years.

Sometimes the egg of Columbus really is just standing it on its end.

What Now?

The doubt persists. I don't think it will ever fully go away.

But I keep building. The system keeps working. The results keep accumulating.

I'm building a chat interface now—a proof of concept that demonstrates SCMS in action. Not as a theoretical framework, but as a working application. Memory that persists. Patterns that promote. Failures that teach.

Maybe that will be enough to get noticed. Maybe it won't.

Either way, I know what I've seen. I know what works.

The experts will notice when the results become undeniable. My job now is to make them undeniable.

A Note to Fellow Practitioners

If you're building things, not just theorizing about them, trust what you observe.

The credentials don't matter as much as they tell you. The institutions don't have a monopoly on insight. Sometimes the person best positioned to solve a problem is the one who lives with it every day.

Your frustration is data. Your workarounds are hypotheses. Your iterations are experiments.

You might be closer to something important than you realize.

Resources

Matthew "Manny" Walker is the creator of SCMS (Sparse Contextual Memory Scaffolding) and is currently building Mneme, a chat interface that demonstrates continual memory in practice. Find him on X @getmneme.