Aaron Williams ยท Merkle & Bloom / Barnabas Coaching
A real hour of chaos on my agent setup. What broke, what the logs actually said, and why the boring file-on-disk decision from my 200-day post is what saved it.
This past Tuesday at 7:14pm I sent my agent a message and then watched a spinner for twenty minutes. By the time it stopped, the gateway behind it had restarted twice, four subagents had refused to spawn, and the main session was gone. Everything that mattered survived. This is the post-mortem, with the error strings left exactly as I found them.
The short version: my agent asked me for a credential, and I never got the question.
A background task called the secrets tool, which prompts the human to type a value into a masked entry. The prompt never arrived. Over iMessage the delivery fails outright; in the web UI it just stays quiet. The gateway held the request open, timed out at 120 seconds, twice, then logged requester settle wake failed every 60 seconds for the next twenty minutes. That loop was the spinner. Nothing was thinking. The machine was waiting on an input nobody could see.
At 19:18:45 the system reset the main session on its own. It fired 49 milliseconds after I sent a fresh message, logged the reason as reset, and recorded no actor. I didn't do it. The agent didn't do it.
A session reset is supposed to be the recovery. This one started the messiest stretch of the evening.
The model came back up and began emitting malformed tool calls, over forty in a row, each rejected with message required raw_params. The transcript layer threw transcript projection rebuilding errors while it tried to make sense of its own state. And every attempt to spawn a subagent came back with the same refusal:
embedded tool authority registration does not match its attemptThat last string is the interesting one. After a restart, the gateway's record of which tools each agent may use no longer matched what the spawn attempt presented, so it failed closed. It did that four times: two on Sonnet, one on Opus, one more during the incident itself. I can't fully explain it yet and the fault sits upstream in the harness, so it lives on the watch list, not the fixed list.
Restarts bookended the evening. One hit mid-turn around 19:39 and killed three review subagents and a tool batch in flight. A real one landed at 19:50:12, SIGTERM, back up at 19:50:38, twenty-six seconds down, with the status page reading OpenClaw update in progress: requested. Meanwhile the UI had shown me [System] gateway restart interrupted your turn at 19:37, thirteen minutes before any restart happened. The error message lied. Worth remembering the next time one sounds confident.
The casualty list is short because almost nothing lived in the blast radius.
The main session died heavy: 193,528 tokens, too big for its own guardrails, unsalvageable mid-incident. I don't mourn it. Two hundred days into this project I wrote that the days belong to the agent, not the session, because everything durable lives in files on disk and the session is disposable. Tuesday night was the thesis test I hadn't asked for.
Recovery was reading, not archaeology. The agent's daily log held the timeline. Context-offload notes in memory/context-offloads/ captured every compaction as it happened; the two days before the crash had already been noisy, nine of them on Monday alone. Transcript corpora from earlier sessions sat on disk too. Rebuilt from those files, the session went through a proper compaction, 54,064 tokens down to 8,672, and picked up where the logs said it left off. The subagent work that was in flight when things broke, a draft of the 200-day post and a spend tracker, landed later the same evening, because its inputs and outputs were files.
One more routing note: the local Q8 model on the M5 Max, the one benchmarked at 90.4 tokens a second in that post, doesn't touch OpenRouter, so the model mix-up below couldn't have happened to it. Different failure modes. That's the point of having both.
Cleanup turned up two more problems. Both quiet, both real.
First, the secret store. API keys live in SQLite, and live rows carry a deleted_at_ms of NULL. One of the key-lookup queries expected 0 for "not deleted", and NULL is not 0 in SQL, so a live key read as deleted. One value, NULL instead of 0, and a stored credential goes invisible while sitting exactly where it should be. The fix was one query. The unease is bigger: this store only gets read during outages, so a latent NULL bug survives until the worst possible night to find it.
Second, the money. Checking spend during cleanup, that day came in at $4.81 against a $1.56 daily average. A 308% spike. The cause was a model misconfiguration: the default model had been flipped to GLM Flash V4 Turbo instead of GLM 5.3 Flash, and nothing complained. Wrong model, same shape of answers, three times the cost, discovered by accident during cleanup.
The credits tracker built that same evening is what made the spike visible: a daily OpenRouter balance line on a 7am cron. It now doubles as a silent-model-swap alarm, because spend is the one channel a misconfiguration reliably speaks through. (Four dollars eighty-one isn't a crisis. A default model that changes without telling me is.)
Secrets requests no longer wedge the main session. The rule is now: confirm the credential exists in the store before any agent asks for it, and never let the interactive session sit on a masked-entry prompt. That wedge cost twenty minutes and triggered the whole cascade.
The credits cron stays. One line every morning, and a model swap or a runaway loop shows up as a number instead of a surprise weeks later.
And the key that leaked got rotated. Full honesty here: while debugging the secret store, a fragment of a management key ended up in a session transcript. That's a logging mistake on my side, it went into ERRORS.md like every other entry, and the key was rotated the same night.
The tool-authority mismatch is unexplained and upstream; if it recurs after the update, it gets reported, not guessed at. Sessions can still reset with no actor recorded. UI boilerplate can assert a restart that didn't happen. A restart mid-turn still kills anything a subagent is holding. And memory recall is a habit enforced by the same agent that can skip it, which I flagged in the 200-day post and flag again here.
I'd rather list these than edit the post until it sounds finished.
The 200-day post ended with a line: the agent isn't the thing you fix, the rules are. Tuesday added a corollary. The files are the disaster recovery plan.
The crash took the session, the spinner, and most of an evening. It couldn't touch a directory of markdown files that knew exactly what had happened, and those files were enough to put everything back. The next crash is a matter of when. The setup is ready for it in exactly one way: whatever matters gets written down while it's happening.