Attach bridge: support response-stream resumption across same-instance disconnects

[Clubhouse1661]

Problem

The godot-ai attach bridge deliberately has no post-dispatch read deadline. A bridge-side timeout could turn a successfully completed mutation into an ambiguous result, and retrying it could duplicate the mutation.

This preserves the core safety invariant, but leaves one recovery gap: if the downstream Streamable HTTP response stream is lost while the backend process remains alive with the same instance_id, the bridge can wait indefinitely. Its identity monitor continues to see a healthy backend, while the MCP SDK may have no response stream from which to receive the result.

This follow-up comes from #822 and the PR A implementation.

What PR A established

Phase 0 testing found:

• FastMCP 3.0 and the current 3.4 release both expose an event_store argument on FastMCP.http_app().
• With the current FastMCP 3.4 stack, a TCP cut-and-resume test successfully reconnected and recovered the response when event IDs were present.
• The equivalent end-to-end flow hangs on the supported FastMCP 3.0 floor, even though the reconnecting GET request occurs.
• Enabling the event store only on newer dependency versions would give the attach bridge version-dependent recovery semantics.

For that reason, PR A did not enable an event store.

Two distinct failure cases

These should remain separate in the implementation and documentation:

1. The response stream is lost while the server continues processing.

   This should be recoverable through MCP event IDs and response-stream resumption.

2. The server-side request task dies without producing a response.

   An event store cannot recover a response that was never produced. Without introducing an unsafe post-dispatch deadline, recovery remains upstream cancellation. Cancellation must clean up the bridge’s local tasks and must never replay the call.

Proposed work

1. Reduce the FastMCP 3.0 hang to a minimal reproducible case and determine whether the cause is in FastMCP, the MCP SDK, or the bridge’s transport construction.
2. Implement a compatibility shim, upstream fix, or explicitly justified dependency-floor change so resumption behaves consistently across every supported FastMCP version.
3. Add a backend-level, in-memory event store with bounded retention:
   • TTL bounded;
   • size bounded;
   • shared by the backend application;
   • no persistent state across backend restarts.
4. Wire it through every Streamable HTTP construction path:
   • normal CLI startup in src/godot_ai/__init__.py;
   • reload/ASGI startup in src/godot_ai/asgi.py.
5. Keep read=None for dispatched operations. Do not implement bridge-side read deadlines or replay after ambiguous dispatch.
6. Preserve and document the server-task-death limitation.

Required tests

• A real TCP proxy severs the downstream response stream after a tools/call has been dispatched.
• The backend completes the operation exactly once.
• The client reconnects using the event ID and receives the original result.
• The bridge does not invoke the tool a second time.
• The test passes at both the FastMCP dependency floor and the current pinned/maximum-supported stack.
• Both normal CLI startup and ASGI/reload startup use the event store.
• Cancelling a call whose server task will never produce a response cleans up all bridge tasks and does not replay the operation.

Acceptance criteria

• Same-instance network stream loss is transparently resumable across every supported dependency version.
• No mutation is executed more than once.
• No post-dispatch read deadline is introduced.
• A request task that dies without producing a response remains cancellation-recoverable and is explicitly documented as distinct from stream resumption.

[dsarno]

Maintainer direction: keep this open, but do not spend effort building a compatibility shim for FastMCP 3.0.

Please first rerun the real TCP cut-and-resume test against the current supported 3.4.x releases and identify the earliest version on which the full flow is reliable. If that matrix is green, the preferred implementation is:

1. raise the FastMCP floor to that verified version;
2. add one backend-shared, in-memory event store with both TTL and size bounds;
3. wire the same store semantics through normal CLI startup and ASGI/reload startup;
4. retain read=None, never replay an ambiguously dispatched mutation, and keep server-task death cancellation-only;
5. retain the real TCP exactly-once tests across the new floor and current maximum-supported version.

If current 3.4.x is not consistently resumable, leave the feature deferred behind the upstream transport problem rather than adding a read deadline or home-grown replay path. The no-duplicate-mutation invariant remains more important than transparent recovery.

[dsarno]

Progress note: the v4 branch (#943) pins fastmcp==3.4.7, which satisfies step 1 of the direction above (floor raised to a version where the TCP cut-and-resume flow is reliable). attach/proxy.py still leaves the event store disabled with the 3.0 caveat comment, so steps 2–5 (bounded, backend-shared in-memory event store wired through CLI and ASGI startup, read=None retained, cancellation-only on task death) remain open and can now be built against v4 without a compatibility shim.

[dsarno]

Scheduled for 4.1 as an enhancement; the fastmcp==3.4.7 pin already satisfies step 1 of the maintainer plan. Steps 2–5 remain: a backend-shared in-memory event store bounded by TTL and size, consistent CLI/ASGI wiring, read=None with no ambiguous-dispatch replay and cancellation-only recovery for server-task death, plus real TCP cut/resume exactly-once and cancellation tests across the supported dependency versions. — Codex

[dsarno]

Retested on the currently pinned fastmcp==4.0.5 and mcp==2.2.0 through real TCP and the actual stdio attach bridge.

Ordinary response-stream cuts resume with one tool invocation. Independent review found a remaining SDK race, reproduced twice with a deterministic scheduling hook. If the tool finishes after the replay snapshot but before the resumed live stream is registered, its final response is retained in the event store but never reaches the waiting attach call. The trace records one tool POST, one side effect, a cursor-bearing resume GET, the retained original result, and a call still waiting until upstream cancellation. Cancellation completes without redispatching the mutation.

Following the existing maintainer direction, this remains deferred behind the upstream transport issue. The bounded-store implementation has not been shipped. No read deadline or custom mutation replay will be added to work around this gap. The next gate is the same real TCP race test passing on a supported SDK version, followed by the bounded-store qualification.

Reproduction sequence for the upstream boundary (the hook is test-only, not a production workaround):

1. Start a legacy Streamable HTTP session through the actual stdio attach bridge and a controllable TCP proxy; dispatch one blocked tool and retain its priming cursor.
2. Cut the response connection and observe the Last-Event-ID resume GET.
3. At the resumed _mint_priming_event call, after replay_events_after returns but before _request_streams[stream_id] is registered, release the tool and wait until its final response is stored.
4. Let stream registration proceed. The caller remains pending two seconds after the final response was stored; direct event-store replay retrieves that same final response. Cancel the caller and verify teardown.

Both independent runs recorded post_count=1, one side effect, resume_get=true, pending_after_final_stored=true, and successful cancellation. This proves the forced ordering can lose delivery; it does not estimate its frequency in ordinary use. The relevant SDK ordering is in mcp/server/streamable_http.py, replay followed by resumed priming and then live-stream registration. The final result can arrive in the intervening gap.