House Brownie — Joshua Coleman

A local-first household assistant built on one bet: that orchestration can buy back what small local models give up. Tokens are cheap and context is scarce, so Brownie trades a single large call for many small, tightly-framed ones. It was prototyped on a 32 GB desktop GPU — then, after a laptop GPU died to sustained inference load, retooled to run CPU-only on an ordinary machine, with the discrete GPU kernel-caged so the inference path can’t reach the hardware that killed the last one.

The bet

No single call tries to do everything. A complex request is decomposed into independent sub-problems, each researched in its own fresh context window with purpose-built framing, then synthesized and critiqued. Twenty 5k-token calls beat one 100k-token call: each slice gets the whole budget devoted to it, and reasoning that never needed to coexist never has to. Orchestration architecture substitutes for context size — exactly the trade you want when the model is small and the window is tight.

Why it runs on a potato now

The first working version ran a 32B model on a 32 GB desktop GPU: streaming chat, a tool-use loop, single-GPU image generation with a load/unload swap. It proved the substrate end to end. It also confirmed the obvious — consumer laptops do not survive sustained GPU inference — so Brownie retooled.

The model now runs as a separate, CPU-only, kernel-caged llama-server process: pinned to efficiency cores under a hard CPU-time quota, behind a thermal watchdog that pauses inference before the package gets warm. The application talks to it only over an HTTP seam. Build against the seam, not the model — swapping today’s caged laptop worker for a ventilated LAN box later is a one-URL change, not a rewrite. The retool made the original thesis matter more, not less: a 4B model in an 8K window needs multipass orchestration to punch above its weight far more than a 32B model ever did.

The substrate

Framing files. Role, constraint, domain, and output-format files, concatenated per call. Every token the model spends orienting itself is a token it isn’t spending thinking; framing files pay that tax up front. Unfamiliar domains get drafted by the system itself and refined over use.

Versioned-summary compression. Long-running investigations keep a versioned summary chain on disk; each new finding recompresses the state to about 500 tokens. Because lossy compression compounds, every third version is drift-checked against an earlier checkpoint — what was in the old summary that’s missing or contradicted in the new one? — a bounded search for what fell out, before it fossilizes.

Files are the interface. Everything is plaintext on disk: framing, memory, logs, decisions. The filesystem is the audit trail, the memory, and eventually the personality. Continuity is architectural, not model-native — a brain/nervous-system split, not a bigger prompt.

What’s built and runs today

The application layer, model-agnostic behind the seam:

Flask + WebSocket chat with streaming responses and history persisted to SQLite (WAL).
The model seam — an OpenAI-compatible HTTP client to the caged worker; no weights loaded in-process.
A tool-use loop over a sandboxed tool set: scoped file I/O, shell with a blocklist, Python execution, RAG query.
ChromaDB wired for vector memory; per-user login for a household.

In progress: standing up the caged CPU worker (Qwen3-4B) as the live backend behind the seam. The GPU prototype already proved the full stack, image generation included; the CPU version is the honest, hardware-safe successor.

Designed, not built — roadmap, listed as intent, not capability:

the multi-stage pipeline as an automatic router (today’s chat is single-pass);
RAG ingestion — the store is wired, but nothing populates it yet;
the reflective thinking loop — an overnight compute budget spent on self-directed research, memory maintenance, and pattern synthesis, with no immediate deliverable;
the research membrane — a separate, stateless, whitelisted agent that is the only thing allowed to touch the internet.

The next milestone is narrower and concrete: the caged 4B answering real household questions over the seam, on a laptop, with nothing phoning home. That “runs on an ordinary machine” demo is a strict upgrade to this page, grafted on when it’s true.

Status & access

Each household’s Brownie is private — the family’s data never leaves the house; that is the point of local-first. The architecture and the code are shareable.