We burned $12 in an unbounded agent loop before we noticed. Here's what we changed.
We just published our agent harness write-up and I wanted to share the lessons here because they cost us real money to learn.
The incident that started it: a runaway session quietly ate ~$12 of API usage before anyone caught it. The model wasn't broken — it was stuck in an unbounded loop with bloated context and silent retries, looking productive the whole time.
After debugging enough production failures, we stopped thinking about "better prompts" and started thinking about the harness around the agent. Three problems kept showing up:
1. Controlling execution. The agent loop only matters when it fails. Your tool registry should enforce rules, not just expose callables — bounded inputs, clear failure modes, visible side effects. Long-running agents need checkpointing, pause/resume, and cleanup. Not optional.
2. Controlling context. One memory store is a trap. We ended up with four: conversation (native), long-term (DB), agent notes (markdown + tags), preferences (vector). Large tool outputs spill to disk and get replaced with compact digests — except errors, which stay verbatim so recovery works.
3. Controlling trust. Every tool carries ALLOW / ASK / DENY. No "probably safe" auto-classification — that always breaks under chained actions. Code execution runs in sandboxed temp dirs with timeouts and resource caps. Hooks log every AGENT_START / LLM_REQUEST / TOOL_CALL_END so you can actually see what happened.
TL;DR: The model isn't the hard part. The harness is. If we rebuilt from scratch, we'd pick control over convenience every time: bounded execution, recoverable errors, explicit permissions, spill-to-disk context, inspectable runtime.
Full write-up: https://medium.com/p/5ec2c649bbb9
Curious what others are doing - how are you surfacing tool calls to users without killing flow?