Proposal: Transport-agnostic resumable streams #543
Replies: 6 comments 7 replies
-
Universal subscription mechanismAnother thought I had is that transport-agnostic resumable streams provide an alternative way of modeling subscriptions. For example, currently, in order to subscribe to resource updates, the client sends a The transport is left to handle the case where a client might disconnect while the server continues to emit An alternative way of modeling resource subscriptions would be for the client to send a The stream would be resumable regardless of transport, guaranteeing that the client sees all update notifications. There would also be a clear retention policy for update notifications, allowing servers to reclaim resources as appropriate. |
Beta Was this translation helpful? Give feedback.
-
|
Tools could also use this strategy to act as event publishers. Related discussions: |
Beta Was this translation helpful? Give feedback.
-
|
This strategy also allows the server to notify the client when a subscription is no longer valid. For example, currently, if a client subscribes to a file resource, and that file is subsequently deleted, the server can either quietly do nothing or send a When using a transport-agnostic resumable stream for the subscription, the server could instead send a Relatedly, it might be worth adding a |
Beta Was this translation helpful? Give feedback.
-
StatefulnessDuring the last Hosting WG meeting, there were some questions about statefulness and how this proposal would work with the Streamable HTTP transport, so I wanted to share a high-level sketch. The sketch is not prescriptive; it's just one possible implementation. The steps are as follows:
Streaming HTTP transport compatibilityBecause the Streamable HTTP transport is unidrectional, it does not use explicit ACKs. Instead,
Footnotes
|
Beta Was this translation helpful? Give feedback.
-
|
I might have missed something but I didn't gather whether this is going to be an approach that applies to all tools (all tools stream regardless of whether the response is created at once or over time) or only to select tools where streaming responses comes more naturally. If this idea allows for the latter, it could also benefit from tool annotations that allow the client to identify streaming vs non-streaming tools: #489 |
Beta Was this translation helpful? Give feedback.
-
|
Ideally, this would apply to all tools, and the SDKs would transparently handle stream management on both the server and the client. That way tool authors don't have to make a guess about how long their tool will run for all inputs and environments. So, with regards to a tool annotation, I don't think it would make sense. |
Beta Was this translation helpful? Give feedback.
-
Webhooks exampleWebhooks aren't part of this proposal, but during the last Hosting WG meeting, I mentioned an example of how this proposal could integrate with webhooks:
Steps 3 and 5-8 can be handled transparently by the SDKs, so the tool author does not need to do anything extra to enable this flow. |
Beta Was this translation helpful? Give feedback.
-
Backward compatibility and opting inThis proposal does not require that tools be written differently than they are now. The SDKs should be able to handle beginning, resuming, and ending streams in a transparent way (and only when permitted by the negotiated protocol version). The only required change to server code should be configuring If there are no configured values for |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jonathanhefner we had an offline conversation, I'm bringing the remaining points here for visibility. I think my residual concerns with this proposal are that it is a breaking change for clients (I agree prototected by protocol negotation) and that clients only know AFTER they've sent a request that the stream is setup. I'll elaborate both points below and propose a variation of the above that I think solves the latter problem and potentially the former by making this opt in if so desired on the client side, whilst also keeping the essense of this proposal intact.
sequenceDiagram
actor Client
actor Server
Client->>Server: InitializeRequest(stream={resumeInterval={min=10, max=60}}}
Server-->>Client: InitializeResult(capabilities(stream={streamId=1,resumeInterval={min=10, max=60}})
Client->>Server: CallToolRequest()
Server-)Client: ProgressNotification (1/4)
Server-)Client: ProgressNotification (2/4)
Note over Client,Server: Client disconnects
Client->>Server: InitializeRequest(stream={streamId=1,resumeInterval={min=10, max=60}}}
Server-->>Client: InitializeResult(capabilities(stream={streamId=1,resumeInterval={min=10, max=60}})
Server-)Client: ProgressNotification (3/4)
Server-)Client: ProgressNotification (4/4)
Server-->>Client: CallToolResult
This changes the stream setup to be an active request as part of session initialisation from the client (possibly with a default to min=0, max=inf so the client defaults to requesting infinite stream durability) the server then responds with the actual negotiated stream durability and the client can then chose whether or not they can honour that reconnect time period and therefore whether to call the expensive or long running request. Another variant of the above would be adopt the CallToolAsyncRequest from #617 which would allow the stream setup to be done per call rather than as a default for the session as a whole with the initialise approach above. This is a larger change though so if the consensus is go with this option I think this variant is a reasonable compromise that allows for the client to have some level of certainty BEFORE it submits the request. |
Beta Was this translation helpful? Give feedback.
-
I'm not sure I understand the concern. I think most clients and servers will be using an SDK, and the SDKs will likely maintain support for older protocol versions for a while. During < 8000 a href="https://modelcontextprotocol.io/specification/2025-03-26/basic/lifecycle#version-negotiation" rel="nofollow">version negotiation, the client SDK will send the latest protocol version that it supports. If the server SDK is newer, it will downgrade to that protocol version. If the server is older, it will respond with the latest protocol version that it supports. If, for some reason, the client does not support that version, it will disconnect. Otherwise, things proceed. It's analogous to HTTP servers supporting multiple versions of the HTTP protocol. Also, a semantic version number might be worthwhile once the protocol is more mature, but, for now, you can think of the date as a major version number.
There is no guarantee of a result regardless of whether there is a resumable stream. Paraphrasing from our previous conversation: if the client were able to request a The real problem is that a client can't know whether it will be disconnected nor how long it will be disconnected for, so it is futile for a client to specify these parameters. Instead, the server should make a best effort to accommodate offline clients, within reasonable limits. Anything more than that should be handled outside of the protocol (via HA client proxies, service agreements, etc).
In my opinion, breaking up the sequence into two separate requests (one for "begin", one for the actual tool call) would be a non-starter. It would be less efficient due to the extra round trip, and it would complicate horizontal scaling because the servers have to maintain continuity between requests. And, again, how would the client know whether it can honor the reconnect period in order to make the decision? (Also, for that matter, how would the client know whether the tool call will actually be expensive or long-running?) |
Beta Was this translation helpful? Give feedback.
-
Mainly that there is quite a burden on builders to keep up with changes and being intentional about breaking changes I think is positive for the community building and using this protocol, by using semantic versioning it's at least clear when something has changed and who is affected.
Rather than going round the loop on this in this point I'll raise a discussion point on protocol versioning and relationship to mcp protocol maturity. Note https://semver.org/ generally handles fast moving early iteration by using a 0.X.Y version number the 0 indicates to the parties involve the API is not yet stable and breaking changes might occur at any time. However after 1.0.0 this is not the case and clients/servers should be able to infer without having to read all the docs whether they support particular combinations of versions.
I agree this is a risk, the use case I'm focussing on is calling a long running task (the example I've been using is training a model) I envisage this to be done on a network that is generally available so I'm expecting the client to be able to reconnect within a matter of minutes at worst. I agree that in the case the client can't reconnect, i.e. the initialise(streamId=previousStreamId) fails then the client has some extra work to do to figure out what state the server is in, but at least it can do this knowing the previous stream is gone. This is not obviously possible if it relies on notifications with no return value.
I do agree these should all be best endevours behaviours, I agree other mechnanisms can be layered on top if full durability is required.
The initialize step is already a part of the protocol so I /think/ this is simplifying the implementation of this, i.e. no new protocol messages but some extra optional params on the already existing requests. I agree horizontal scaling is non-trivial however I don't think this changes things as initialize is already a part of the protocol and happens once on the client irrespective of whether the backend server is scalled across multiple processes. I suggest that a server should setup an external event store (pick a caching technology of choice, e.g. redis etal) and use the session id to respond to look up the current state if a request is sent to a process that was not the original receiver of the initialise request.
I had envisage some other changes to the Tool along the lines of idempotentHint, in the simple case it might just state this as text in a tool description and an agent (either human or AI could then adjust its behaviour accordingly) |
Beta Was this translation helpful? Give feedback.
-
I forgot to mention in my previous comment that if there is a significant need for clients to specify a
The client will know the stream is gone because
Ah, I was getting confused with our previous discussion. So are you suggesting that the client specifies a global One thing I like about it, though, is that it jibes with the idea of sending capabilities on each tool call in order to support session-less servers. So if we had something like that, the client could still specify
Those might help a bit, but also consider that a tool itself may not know whether it will be expensive or long-running, especially without foreknowledge of its inputs or runtime environment. |
Beta Was this translation helpful? Give feedback.
-
SDK API for multi-turn tool interactionsThe following SDK API suggestion is not part of this proposal; this proposal will work with the current SDK APIs. However, transport-agnostic resumable streams offer a way to scale multi-turn tool interactions: when the server sends a request to the client, such as a sampling request, the server can disconnect and stop execution while waiting for a response. The current TypeScript SDK (and perhaps other SDKs) makes this difficult because server-to-client requests are handled as promises. The server makes a request and awaits the promise, holding server resources until the client responds. Below is an example of a suggested API for handling server-to-client requests using discrete functions. To make a request, the server returns a request object from a function, allowing the server to stop execution while waiting for a response. The request object server.tool(
"my_long_running_tool",
inputSchema,
streaming((inputParams) => {
// When this function returns, the SDK sends a `stream/begin` notification,
// then the SDK runs the `perform` callback further below.
return { resumeInterval: { min: 10, max: 24 * 60 * 60 } };
}).perform((inputParams, stream) => {
// This function can close the connection at any time and continue to emit
// JSON-RPC notifications.
stream.closeConnection();
stream.emitProgress(progress);
// When a stream will be interrupted due to a server-to-client request, the
// server should explicitly manage its state.
myStore.set(stream.id, myState);
// The server can make a request to the client by returning a request
// object. If the connection is still open, the request will be sent over
// the connection, and the SDK could then close the connection so that
// resources aren't consumed while waiting for a response. If the
// connection was already closed, the request will be stored along with
// emitted notifications.
return stream.samplingRequest(prompt);
}).resume((samplingResult, stream) => {
// Restore state from the previous callback:
let previousState = myStore.get(stream.id);
// Emit result:
stream.emitResult(result);
// This function does not return a value, so the SDK sends a `stream/end`
// notification.
}).finalize((stream) => {
// This function is called after the stream ends or is abandoned.
// Clear state from external store to free resources:
myStore.delete(stream.id);
})
); |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Pre-submission Checklist
Your Idea
Motivation
Stream resumability as currently defined faces multiple issues:
Last-Event-ID.Proposal
To address these issues, I'd like to propose a transport-agnostic resumable streams. The basic elements are:
stream/beginJSON-RPC notificationstream/beginJSON-RPC notification with arequestIdandstreamId.requestIdis the request'sidproperty.streamIdis a generated unique ID for the stream.stream/beginalso includes aresumeIntervalparam withminandmaxproperties.resumeInterval.minindicates the minimum amount of time the client should wait before resuming the stream after a disconnect.resumeInterval.maxindicates the maximum amount of time the client may wait before resuming the stream after a disconnect. If this limit is exceeded, the stream is considered "abandoned", and the server may delete all data and cancel all work related to the stream. A value of 0 indicates that the stream is not resumable, and that the work will be cancelled upon disconnect.stream/beginis sent, both the server and client may disconnect at will. (We will need to amend this part of the spec.)stream/endJSON-RPC notificationstream/endJSON-RPC notification with thestreamIdto mark the end of the stream.stream/resumeJSON-RPC notification (or request?)stream/resumeJSON-RPC notification with astreamIdto resume that stream on the current connection. The server should then send all JSON-RPC messages for that stream that were not previously sent to the client.stream/resumea JSON-RPC request instead of a JSON-RPC notification. As such, the server would immediately send a JSON-RPC response either confirming the resume or reporting an error for an invalidstreamId. However, this would inject an extra message into the stream, which might slightly complicate replay mechanisms.streamIdis invalid, the server should immediately send astream/endnotification.stream/pollJSON-RPC requeststream/pollJSON-RPC request with astreamId, and the server should respond with the status of the stream, including whether there are pending JSON-RPC messages and whether the stream has ended.stream/pollrequest, the server should reset the "abandoned" timer for the stream. However, the server may choose not to do so based on internal limits for stream age.Server-to-client requests
Handling server-to-client requests (e.g. sampling requests) can be tricky, depending on the transport. There are two points that I think are important:
Following from those points, I think unidirectional transports such as HTTP should disconnect when the server sends a request to the client during a stream, including during a stream initiated by
stream/resume.Also, following from the 2nd point, I think server-to-client requests should embed the stream ID in the request ID. That way, if a response is received on a new connection, the stream ID can be extracted from the response (which will have the same ID as the request), and the stream can resume on that connection.
Though not part of this proposal, I have a suggestion for an SDK API that leverages resumable streams to improve scalability in the face of server-to-client requests.
Batching and multiplexing
Support for JSON-RPC batching was recently removed from the spec; however, when using a unidirectional transport such as HTTP, there are a couple of compelling use cases related to streams:
stream/pollrequests on a single connection.stream/resumenotifications on a single connection to multiplex streams.Webhooks
I see webhooks as more of a transport-level concern, so webhooks are not part of this proposal. However, webhook support could be based on resumable streams. For a detailed example, see #543 (comment).
Asynchronous long-running tasks
This proposal is a generalization of my proposed approach to asynchronous long-running tasks, and supersedes that proposal. I think the basic elements in this proposal provide a solid base for handling asynchronous long-running tasks and can be extended as necessary with features like batching and webhooks.
Scope
Beta Was this translation helpful? Give feedback.
All reactions