Refactor Token ID handling

Token ID handling is a bit of a burden for Responses API Agent server developers, and we want to avoid them having to handle this sort of thing.

Also, there are generation engines that require additional information beyond that of just token IDs that should also be passed back to the model endpoint