The story behind it
Rhys is our name for him. What I’m actually building is an assistant any household could have, and the name is just whatever you give it.
The idea is simple. There’s one brain, which knows our family and our home and can take action. Then there are satellites, which are just ways to reach it. You can type to it, talk to it, or one day call it on the phone. It doesn’t matter which, because they all lead to the same brain. The brain runs on Hermes, an open agent I host at home. What I’ve built is everything around it: the voice engine, the satellites, and a phone line that’s paused for now while I rebuild it.
The hardest part has been voice. I want to talk to Rhys the way I’d talk to a person, with no awkward pauses, and have him follow along when more than one of us is in the conversation. Speech-to-speech voice models sound wonderfully human, but they come with their own brain, separate from the one that actually knows our household. Getting the two to work together without one of them confidently making something up is a problem I haven’t solved yet. For now Rhys transcribes what you say, thinks, and then speaks. It’s accurate, but you can hear the wait.
My girls helped me test his speaker recognition. My oldest asked, “Who am I?” and Rhys confidently named our two-year-old. They thought it was hilarious, and so did I, but it’s also exactly why who’s talking matters. Rhys keeps personal information separate from what anyone in the house can hear, privileged actions need the right person present, and since voice recognition can still get it wrong, some features are limited when you’re talking rather than typing.
Today I’m his main user, and mostly by typing. A lot of my side projects get started that way: I describe what I want from the road and Rhys builds it, including the site for our family’s fantasy football league.
My wife doesn’t use him yet. She’s a little wary of how much he knows, which is fair. I think what will change that isn’t Rhys knowing more. It’s Rhys solving a real problem for her, like groceries, reminders or keeping track of spending, and being on her phone when she isn’t home. That’s the next step: earning a place in her day.