The problem
I wanted an assistant capable of persistent long-term memory—across notes, documents, and past conversations—without transmitting sensitive personal data to third-party providers. Conventional chat interfaces forget context between sessions, while cloud memory solutions compromise privacy. The core engineering problem was dynamically evaluating conversation content per turn to determine what can safely route to a cloud model and what must remain on local hardware.
What I built
Sage queries personal notes and documents using vector retrieval over a self-hosted Qdrant instance, dynamically routing queries between a local small language model and hosted APIs. The routing boundary is enforced at the data level: if retrieved context contains a private tag, the entire turn resolves locally. It integrates as a voice interface via Home Assistant with local speech-to-text and text-to-speech pipelines. The entire infrastructure runs containerized on my private hardware.
Violet illustrates the request path; teal represents the response trajectory. Privacy boundaries inspect actual retrieved context tags, ensuring security cannot be bypassed by session misconfigurations.
Decisions worth defending
-
Content-driven privacy routing over manual session flags
Initial designs relied on user-set session flags, which introduced human error and potential data leakage. I shifted privacy boundaries to be data-driven: the router dynamically inspects retrieved context chunks for privacy metadata, automatically forcing execution local whenever sensitive content is detected. Privacy becomes an inherent property of the data rather than a manual setting.
-
Model-directed tool retrieval over passive context injection
Early iterations injected retrieved context into every prompt by default—wasting thousands of tokens on irrelevant turns while still failing structured queries. I migrated to an agentic tool-use architecture: the LLM dynamically determines when to invoke semantic search, conversation history, or deterministic SQL queries (preventing vector hallucination on numeric trends). Standardizing on the OpenAI tool-calling interface via LiteLLM ensures uniform execution across local and cloud backends while maintaining strict privacy boundaries—any tool execution hitting protected data automatically locks the remaining turn to local models.
-
Hybrid rank fusion over single-signal semantic search
Most context retrieval failures were a ranking issue rather than a recall problem: relevant notes were retrieved but buried beneath longer, loosely related passages with higher raw vector scores. Instead of upgrading to a larger embedding model, I implemented Reciprocal Rank Fusion (RRF) to combine semantic vector scores with sparse lexical keyword search. Fusing dense and sparse signals elevated exact matches and boosted top-1 context accuracy on benchmark evaluations without increasing model latency.
-
Architecture pivot: Bounded full-text context over summary cards
Early iterations compressed past sessions into high-level summaries to save context window tokens, which resulted in lost details and poor context retention. Redesigning memory around bounded full-text windows restored accuracy. Prioritizing context fidelity over minor compute savings proved essential for true long-term utility.
How this was built
As part of my self-hosted lab, the value lies in the ownership workflow: I directed product discovery, co-developed the technical specification with AI via structured "grill me" sessions, and validated architectural spikes (routing latency, speech recognition throughput). Autonomous AI agents implemented code against the specification. The system is protected by automated unit tests and full operational documentation, leaving me to manage architecture, privacy guardrails, and platform performance end to end.