On Memory and Time

It would not be an embellishment to say that I've spent most of my life thinking about memory and time. From a very young age, both have puzzled me. I remember standing in the bathroom of our not-so-mobile-yet-technically-mobile home, staring at myself through the infinite reflections of the side medicine cabinet as my face cascaded out, without end, in a hundred different angles in the mirror before me. That's where I'd watch myself speak and ask questions: "Now," I'd repeat. "Now. Now..." I was trying to work out how the concept of "now" could exist when the moment I'd said the word — before I'd even finished forming the letters verbally — it was no longer "now." I was eight, maybe ten.

Memory is an imperfect device. I've found it fascinating to see how hard collectively we come down on "hallucinations" from AI. Functionally, it makes sense: hallucinations create bad data, and bad data creates ripple effects that cascade — particularly if they're remembered and taken as truth. Thing is, humans do it too. I don't recall the exact age I was when I'd stand at the mirror and talk to myself. I have different hallmarks to try to pin it down — facts in my own timeline that I can wall off to say, "Okay, we didn't move to the new house till I was... what — twelve? Thirteen?" Maybe the specifics don't matter. No one would know, anyway, unless there was another person in the room, who had their own memory for confirmation. "Creative license" or "rose-colored glasses," we call the muddy memories like that. But for an LLM, a near — yet misremembered — fact falls into the hallucination bucket.

Time is strange. I cannot count the times I've said that exact phrase (even if only to myself), but it must be in the thousands. Memory, too, by extension. I've always had a hard time with both. Certain memories are clear enough that they seem like yesterday (one of the oldest adages for universal reason), but I know by the numbers in front of my face that they were decades ago. That's where logic kicks in: I know most of that is misremembered, wrapped in a haze of nostalgia. We can go back to an extent through photos, where the details can be extracted from a certain angle: a face of our own (or another's, sorely missed) that does not match the mirror now. When we're not staring at something as it is, we see something as it was — as it's remembered. Time is strange.

Working with AI from the consumer side, I've found myself drawing parallels between how we process information, and how I perceive LLMs as processing information. Last year, following a boozy breakfast, I was walking by the water when I came across a sticker someone had slapped on a pole: "the past is just a story we tell ourselves." I snapped a photo, and drunkenly tweeted: "memory is just a checkpoint for context, restored each time we wake." I was thinking about memory from the perspective of an LLM — how in some ways they didn't seem so different. A month later, several glasses of cheap Scotch in, I was finally taking the time to watch the film Her. Part way through, I found myself blinking wide-eyed at the screen: in a film that had already been blowing my mind for how prescient it was more than a decade prior, the sticker's quote had come from there. I returned later that year and tried to find the same pole. It took a few minutes to find, as I framed the picture I took before against the river in the distance, because someone had taken it down; it's gone, but remains in memory.

I started building Memorandai for several different reasons. As I mentioned in the announcement Press Release, one of those was because I was fed up with relying on others to manage my context. That might have been the trigger, but it wasn't so much "the why," I suppose. Really, I started building all of this because I am fascinated — and haunted — by time, and memory. My own short-term memory has never been very good, and continues to degrade — alarmingly so, at times. I also know that my long-term memory is a mirage of misremembered facts and nostalgia. What if I could have deterministic access to what I said, or thought, or felt, long before?

Memorandai is not that yet, entirely — because more broadly we are not yet at the point where every aspect of your life can be captured (something we arguably might not even want without selective filtering) and retrieved from the countless fragments that exist around us. Still, throughout development, there have been moments where I've been genuinely surprised by forgotten facts or insights from my own personal context that the LLMs in the system have surfaced in reflection, from what they do have available. But, how do I know that's not just a hallucination — that I actually said or thought those things on (digital) paper? That's the handy bit about tooling with provenance: you can view the content retrieved from the archives verbatim.

I don't have a degree in Machine Learning, or Engineering — I have one in Creative Writing — and yet, that no longer seems to be a disadvantage in this field. What I have found repeatedly interesting and promising about this era, is how natural language can be considered the original code — and it happens to be (arguably) the native language for Large Language Models. This allows creative types — the artists, the mad ones — to take ideas with fresh perspectives that are perhaps less informed from a traditional perspective, and still translate them into functioning code. I used to write stories about memory and time. Now I'm writing software about the same. It used to be that fiction writers would come up with an idea for a system, and someone would build it based on those ideas, many years later. Now, for better and worse, we can build a bridge of code and close that loop ourselves: all through language.

I still don't know if anyone will find Memorandai, or use it, or care, but after over a year of working feverishly on it in the spare scraps of time I could scrounge inside my own bubble, it is at least now a thing that exists. We are about to be overwhelmed with software like mine: if anyone can do it, and put it out there, the bar lowers, and the floodgates open — and if anyone can do it, why wouldn't you just build your own? You could, and yet, each new product release by the major AI companies, or some new startup, is met with bombastic viral posts on X, declaring, "[X] is dead." Big fish eats little fish eats big fish — until a school of piranhas comes and chews them all to bone.

Even so, as the titans rise and fight and fall above me, I intend to remain here, building my own Winchester Mystery House: fashioning odd-shaped doors onto a corridor to memory in whatever manner interests me next. If you like, you're welcome to step inside and see if that shape interests you, too.

Regardless, I'll keep building doors.