In the build log so far I had two problems. Photo chaos and paper chaos.
The photos got sorted out. The Mac Studio runs Immich now, my wife uploads from her phone without me asking, and the fifteen years of scattered videos are finally in one place. Mostly solved.
The paper never got fixed. It just sat there.
#The binder
Every family has the binder. Or the drawer, or the shoebox. Ours is a binder, plus a couple of shoeboxes. Insurance policies, the mortgage agreement, the kids’ vaccination records, the car papers, tax letters, contracts, drawings the kids made, the warranty for a dishwasher we got rid of years ago.
I know roughly what’s in it. (That’s a lie.) I know roughly where things are. (That one too.) The one letter with the ID number that arrived months ago and that we now suddenly need for an important inquiry? My wife has to help me hunt for it. Notice period on a contract, a policy number, the date the car insurance renews: the process is half a Sunday gone, digging from the inbox tray through the shoeboxes to the binders my wife keeps in order.
I have automated business processes for a living my entire career. Yet the most important paperwork in my life was filed under "I'll deal with it later," in a binder, by hand.
Scanning it doesn’t fix that. It just adds more work. I tried, with a regular scanner, and I won’t even get started on what a chore that was. And back then there was no OCR, no full-text search across everything. A scan was just a picture of a page, in a folder, next to all the other pictures of pages.
What I wanted was radically easier. A letter arrives, I take a photo or use the scanner app, send it to a messenger, and something takes care of the rest, so that I can find it again months later without losing a Sunday over it.
Local, no question. Our documents are basically our lives. The marriage certificate, the kids’ birth certificates, tax records. There was no way I was sending that over a wire to someone else’s computer. Whatever the something turned out to be, it had to run inside the house.
Fast forward a bit: a local Matrix server became the messenger, the central chat interface to the whole machine in the corner, which I’d started calling Merlin.
#The smallest possible thing
This was right after I got local models running on the Mac and couldn’t stop grinning about it. I wasn’t thinking about a wiki, or a system, or anything with a name. I had a pile of documents and one question: could a local model be the something? The thing that takes the mess off my hands, that looks at a document and just knows what it is?
So I started building a classifier one weekend. Point a model at a PDF, get back a category and a couple of tags. Tax, insurance, school, medical. Good enough to keep going.
Then I wired the two ends together. I photographed a letter with my phone, dropped it into the family chat, and waited. A few seconds later the bot wrote back: filed under Insurance, summary attached. The document was already sitting in Paperless, tagged, its text extracted and searchable. I could find it later by typing a keyword.
I was ecstatic. Genuinely. I showed my wife, then photographed another one just to watch it happen again. A real letter, from our real mailbox, photographed and filed and announced in the chat, by a model running on the machine in the corner of the room. Nobody typed a tag. Nothing left the house.
That little loop, photograph, wait, filed, is the thing that hooked me. It is what started the whole famstack journey. And here we are.
#It kept growing
Once the loop worked, the next step always looked obvious. If the model can tag a document, it can file it properly. If it can file it, it can pull the few facts that matter out of it. If it can pull facts out, it can write them down somewhere the whole family can read, instead of leaving them buried in a PDF.
None of this felt like a master plan. Each piece was a small weekend thing that made the next small weekend thing look reachable. I was a few weekends deep before I noticed I’d built a system.
Then agents arrived and the goal moved. What if I gave a locally running agent access to all of this knowledge? Filing was never really the point. Accumulating and extracting the facts buried in the documents was. That turned out to be a lot harder, software-engineering background or not. A lot of research went into it: experiments, finding the right models, more experiments, benchmarks. It was a LOT of work, and most of this series is about that part.
#The name showed up later
Months in, Andrej Karpathy posted a design: point an LLM at your documents, have it compile a wiki, keep it current. People started calling these things second brains.
I read it and recognized the half-built thing on my Mac. The plumbing had been growing toward that exact shape for weeks, one obvious weekend at a time, and here was someone writing it up as a deliberate design. The note didn’t start my project. It gave the thing a name I didn’t have for it.
So the series title is borrowed, after the fact. I didn’t set out to build a second brain. I set out to stop the Sunday afternoons we both lost hunting for a single piece of paper. Everything else grew from that first photographed letter landing in the chat.
The rest of this series is the road from that prototype to a wiki that maintains itself, with the failures left in. There were plenty. The whole arc, and where to start building your own, is in the overview: How to Build a Local Second Brain for Families.