The distance became a weekend

This little guy runs my house.

Local LLMs are going to be a normal part of life. Not a replacement for the big cloud models, but running quietly alongside them, on our phones, our laptops, eventually in our homes. The kind of AI that never sends your data anywhere because it never had to.

So a few weekends ago I decided to stop theorizing about it and implement one.

I put a voice assistant in my house that runs entirely on a laptop in my office. No Alexa, no Google, no cloud. You say a wake word, it listens, a local large language model thinks, and it answers out loud. Your voice never leaves the house. The model doing the thinking is Gemma 4, Google’s small open model, running on a single gaming laptop.

I am not an engineer, and almost none of this was in my toolkit when I started, so I had to learn.

I had to learn Docker. I had to learn how to run Gemma locally with llama-server instead of just calling an API. Connect speech-to-text (Whisper), text-to-speech (Piper). Make all of those systems flow into each other. And finally, how to get those systems to interact with hardware by flashing firmware onto a tiny microphone board. And predictably, debug why it couldn’t hear me, which turned out to be a networking headache that took longer than the AI itself. Each of those was its own small mountain.

Two years ago I wouldn’t have attempted any of it. Not because it was impossible, but because the distance between “I want to understand this” and “I built a working version” felt too far to cross in my free time.

That distance has become a weekend. That’s the part I don’t think we’ve fully absorbed yet.

I’m not claiming the little model in my office competes with the frontier ones. It doesn’t, and it isn’t supposed to. The future everyone who works on this seems to agree on is hybrid: small private models handling the everyday, personal, sensitive things locally, and the big cloud models for the heavy reasoning. What surprised me is how much of daily life a 4-billion-parameter model already handles just fine.

And this isn’t some fringe experiment. Apple and Google already run small models directly on your phone, Apple Intelligence and Gemini Nano, doing quiet on-device work you probably never think about. Analysts expect AI-capable phones to make up nearly half of all shipments this year. The hardware to run this stuff privately is already in most people’s pockets. Most of us just haven’t noticed yet.

I lead education for a living, so I think about this through one lens: what do people actually need to learn now. And I keep landing in the same place. The frontier isn’t only being built by the researchers training the giant models. It’s also going to be built by regular, curious people who decide to run this stuff themselves, on their own hardware, on their own terms.

I wanted to be one of them before I talked about it.