fix(mothership): bound stream recovery with a per-run restart budget

[waleedlatif1]

Summary

Problem. A recovered Chat controller that failed to persist a replay event for any reason other than a budget refusal threw StreamControllerSupersededError and left its run recoverable without finalizing. Every reconnect poll (tail, batch GET, replay hold, every open tab, every pod) then claimed the run again and re-POSTed it to the worker. Nothing bounded it: no cooldown, no attempt cap, and since the tail no longer ends at 60 minutes, a storm ran as long as a tab stayed open. Each re-POST made the worker re-hand the run, so a single stuck run could drive sustained worker load.

Root cause. Recovery treated "this controller failed" as "another controller took over", and readChatStream claimed any non-terminal run with a free chat lock on every read. The only guards were the chat lock (one live controller at a time) and the controller-token compare-and-swap, neither of which limits how often a dying controller is replaced.

Fix (restart limit with exponential backoff, the CrashLoopBackOff / supervisor pattern):

• planRecovery + claimRunController: each run carries a recoveryBackoff (attempts, claimedAt, notBefore) in its request context, written in the same token-compared claim UPDATE, so it is as atomic as the claim and covers every pod, tab and caller. The first takeover is immediate; later ones wait backoffWithJitter (1 s base, 60 s cap). A takeover more than 5 minutes after the previous one starts a fresh budget.
• The claim is the last step before the stream starts, after the replay read, billing and access checks. A takeover that fails before it can start leaves the run untouched: it spends no attempt and does not refresh the run, so the orphan sweep can still settle a run that can never start.
• After MAX_RECOVERY_ATTEMPTS (5) takeovers, the next claim ends the run as stream_recovery_exhausted through the existing terminal path: the run is settled as an error and the worker gets the same explicit abort a replay refusal sends. The worker is not re-POSTed for that claim.
• Only a lease another controller holds is a hand-off. An unreadable lease proves nothing, so the fenced writes that follow (leased append, token-checked run updates) decide ownership instead of the controller leaving the run to a successor.
• A failed leased append that is not a budget refusal still hands off, as before, so a short Redis blip (failover, connection reset) is still completed by a successor; the budget bounds the persistent case.
• StreamTurnFailure is the shared base for replay refusals and recovery exhaustion, so the lifecycle classifies both as errors, never as a cancellation or a hand-off.
• One Claimed stream run for recovery log line per claim, with the attempt number. Before this, recovery claims were invisible on the Sim side.

Behaviour changes

• A run whose controllers keep dying now ends as an error after 5 takeovers (about 30 s of backoff in total) instead of being re-POSTed indefinitely. The user sees "This response was stopped because it was interrupted repeatedly…" and the transcript up to that point is kept.
• A second takeover of the same run within 5 minutes waits 1–2 s; later ones wait longer, capped at 60 s.
• Unchanged: the first recovery after a deploy or crash (immediate, still gated by the lock TTL), parked / permission-wait / client-tool runs (their controller keeps the chat lock, so they are never claimed), replay-budget refusals, worker 5xx and socket-drop retries.

Deploy / compat

• No migration. recoveryBackoff is a new key in the existing requestContext jsonb; runs without it parse as a fresh budget, so in-flight runs are unaffected on rollout and an older pod simply ignores the key (jsonb || keeps the other keys).
• No worker change is required. Closes the Sim side of the storm; the worker-side re-hand coalescing ships separately.

Type of Change

• Bug fix

Testing

• New recovery-storm.integration.ts (real Postgres + Redis, the real chat stream GET route and chat lifecycle, a stub worker): immediate first takeover, one failed append handed off and completed by the next controller, persistent append failure / thrown tool / lost lease exhausting after MAX_RECOVERY_ATTEMPTS with exponential gaps and exactly one worker stop (1 and 3 tails), waiting out a recent takeover's backoff, a takeover that fails before it starts leaving the run untouched and the next one ending the exhausted run, budget reset after the window, and a parked run with a live controller never being taken over.
• Red-first: each guard was reverted on its own and its tests went red (never-exhausted, no reset, claim drops the backoff, no wait gate, failure not pre-aborted, unreadable lease treated as superseded).
• Storm harness against the real chat stream GET route, before vs after, same scenarios:
  • Persist failure on the first frame: before, about 0.6 worker POSTs/s with the run left active indefinitely; after, 6 POSTs in total, then the run is error and the worker is stopped.
  • Append failure after a persisted frame (the shape that looped): before, about 3.5 POSTs/s with one tab and about 7/s with three, growing the replay ring on every cycle; after, 6 POSTs in total with one or three tabs, then terminal. Roughly a 35–70x reduction over the first minute, and bounded instead of unbounded.
  • Healthy recovery, worker 5xx and socket drops: unchanged (1, 4 and 4 POSTs).
• bun run test:integration for recovery-storm, replay-budget, replay-gap, stream-recovery, buffer-ttl, workflow-client-settlement, orphaned-runs: 90/90 passed (77/77 for the mothership suites after the review round).
• bun run --cwd apps/sim test lib/mothership/request lib/mothership/chat: 1198 passed.
• bun run lint, bun run check:audits (54 audits), docs-manifest:check, block-registry check, apps/sim type-check: all pass.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing (new tests pass the test-audit authoring gate)
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 1, 2026 10:10pm UTC

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 11 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds restart limits to stream recovery logic.

The PR appears safe to merge; no outstanding blocking finding was identified.

Summary

The PR adds a durable, per-run recovery budget and backoff so repeatedly failing chat-stream controllers eventually settle the run instead of re-POSTing the worker indefinitely. The follow-up moves the controller claim after recovery prerequisites, leaving runs untouched when recovery cannot start.

• Recovery exhaustion uses the existing terminal and worker-abort path.
• Integration tests cover takeover, backoff, exhaustion, and pre-claim failure.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Reconnect reads run] --> B{Recovery plan}
  B -->|Wait| C[Leave run unchanged]
  B -->|Claim or exhausted| D[Acquire chat lease]
  D --> E[Read replay and check admission and access]
  E -->|Failure| F[Release lease; leave run unchanged]
  E -->|Ready| G[Claim controller and persist budget]
  G -->|Claim| H[Resume worker stream]
  G -->|Exhausted| I[Settle error and abort worker]

Loading

_{Reviews (2) · Last reviewed commit: "fix(mothership): claim a recovering run ..."}

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 11 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>