Osric Labs

The Challenge

Running AI-driven operations at scale means constant interaction with cloud LLM APIs. For an autonomous system making hundreds of inference calls per hour, this creates three compounding problems:

• Escalating costs -- per-token pricing on cloud APIs turns every agent action into a line item, making autonomous operation economically unsustainable at scale
• Latency constraints -- round-trip times to cloud endpoints introduce unpredictable delays in agent decision loops, degrading real-time orchestration
• Vendor lock-in -- dependency on external API providers means rate limits, policy changes, and outages can halt the entire system without warning

The system needed to run autonomously 24/7, self-heal when nodes fail, upgrade its own capabilities, and do all of this without a human in the loop for routine operations.

Our Approach

We designed and built a fully local-first, multi-node physical architecture that keeps all inference, orchestration, and data on-premise.

• Local-first inference -- llama.cpp on the Big Brain node runs Qwen 2.5 32B for complex reasoning; vLLM across the Worker Factory handles high-throughput 7B/14B tasks in parallel
• Physical distribution -- a Raspberry Pi Control Plane manages the WebSocket gateway, orchestration, Telegram interface, and boot sequencing across all nodes
• Self-improving agents -- the Skill Registry tracks agent capabilities and training outcomes; the Planner Trainer refines orchestration strategies based on task performance data
• Docker-isolated workers -- each agent runs in its own container with scoped credentials from the Credential Vault, enabling safe parallel execution
• Frontier Router -- when a task exceeds local model capability, the router selectively escalates to cloud APIs, keeping cloud usage under 2% of total inference
• Self-upgrade system -- the Upgrade Manager can pull, test, and deploy new service versions across the cluster without human intervention