[router] conversation item API: create, retrieve and delete #11369

key4ng · 2025-10-09T06:26:17Z

Motivation

This PR enables feature:

create conversation item.
- logic:
  - if the id matches record in items db, it fetches the content from db,
  - if not, it will add the record based on user's import
  - return error if the item id is already appear in the conversation table
- supported type:
  - "message",
  - "reasoning",
  - "mcp_list_tools",
  - "mcp_call",
  - "item_reference"
    - can use the item created by response api even without conversation id provided
Retrieve conversation item
Delete conversation item
- it only delete the link between conversation id and item id, other conversation can still retrieve the item

Modifications

Accuracy Tests

create item using item reference

curl http://localhost:30000/v1/conversations/conv_bca3ba2e3cad819576b58e09f898114f9ffac202b2b3993c/items \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "items": [
      {
      "id": "mcp_541cd78c3aee05e0d6fb7f41cd8e0c1c169180ecc59b5f0b30",
      "type": "item_reference"
      }
    ]
  }'
{"object":"list","data":[{"id":"mcp_541cd78c3aee05e0d6fb7f41cd8e0c1c169180ecc59b5f0b30","type":"mcp_call","name":"brave_web_search","arguments":"{\"query\":\"new restaurants opened recently in Seattle 2025 Seattle restaurants opened recently\", \"count\": 10, \"offset\": 0}","output":"{\"content\":[{\"type\":\"text\",\"text\":\"Title: Seattle’s New Restaurant Openings - Seattle - The Infatuation\\nDescription: Expect seasonal flavors like banana bread with strawberry jam, plus plenty of mainstays, too (looking at you, Arbequina olive oil). ... Featuring possibly the most colorful dining room in Kirkland, <strong>Dilli 6</strong> is a new Indian restaurant that clearly had no budget limit for paint.\\nURL: https://www.theinfatuation.com/seattle/guides/seattle...\"}],\"isError\":false}","server_label":"brave","approval_request_id":null,"error":null,"status":"completed"}],"first_id":"mcp_541cd78c3aee05e0d6fb7f41cd8e0c1c169180ecc59b5f0b30","last_id":"mcp_541cd78c3aee05e0d6fb7f41cd8e0c1c169180ecc59b5f0b30","has_more":false}
<!-- If this pull request affects model outputs (e.g., changes to the kernel or model forward code), provide accuracy test results. -->

create item which has record in db

curl http://localhost:30000/v1/conversations/conv_bca3ba2e3cad819576b58e09f898114f9ffac202b2b3993c/items \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "items": [
     {
      "id": "msg_097d772e007086690168e7fb081f0c8196acca3314bad42dd0",
      "type": "message",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "annotations": [],
          "logprobs": [],
          "text": "Here are two good options to add to your toolkit:\n\n- Rust: Systems programming language focused on safety and performance (memory safety without a garbage collector). Great for low-level code, game engines, CLI tools, and high-performance apps.\n\n- Go (Golang): Simple syntax, fast compilation, strong standard library, excellent for concurrent servers, microservices, and cloud infrastructure.\n\nWant recommendations tailored to your goals (web, data, mobile, etc.)?"
        }
      ],
      "status": "completed",
      "created_at": "2025-10-09T18:12:24.865791Z"
    }
    ]
  }' 
{"object":"list","data":[{"id":"msg_097d772e007086690168e7fb081f0c8196acca3314bad42dd0","type":"message","role":"assistant","content":[{"type":"output_text","annotations":[],"logprobs":[],"text":"Here are two good options to add to your toolkit:\n\n- Rust: Systems programming language focused on safety and performance (memory safety without a garbage collector). Great for low-level code, game engines, CLI tools, and high-performance apps.\n\n- Go (Golang): Simple syntax, fast compilation, strong standard library, excellent for concurrent servers, microservices, and cloud infrastructure.\n\nWant recommendations tailored to your goals (web, data, mobile, etc.)?"}],"status":"completed"}],"first_id":"msg_097d772e007086690168e7fb081f0c8196acca3314bad42dd0","last_id":"msg_097d772e007086690168e7fb081f0c8196acca3314bad42dd0","has_more":false}

Create item that not in DB

curl http://localhost:30000/v1/conversations/conv_bca3ba2e3cad819576b58e09f898114f9ffac202b2b3993c/items \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "items": [
     {             
        "id": "mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd",
        "type": "mcp_list_tools",
        "server_label": "deepwiki",
        "tools": [
          {
            "annotations": {
              "read_only": false
            },
            "description": "Get a list of documentation topics for a GitHub repository",
            "input_schema": {
              "type": "object",
              "properties": {
                "repoName": {
                  "type": "string",
                  "description": "GitHub repository"
                }
              },
              "required": [
                "repoName"
              ],
              "additionalProperties": false,
              "$schema": "http://json-schema.org/draft-07/schema#"
            },
            "name": "read_wiki_structure"
          }]}
    ]
  }' 
{"object":"list","data":[{"id":"mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd","type":"mcp_list_tools","tools":[{"annotations":{"read_only":false},"description":"Get a list of documentation topics for a GitHub repository","input_schema":{"type":"object","properties":{"repoName":{"type":"string","description":"GitHub repository"}},"required":["repoName"],"additionalProperties":false,"$schema":"http://json-schema.org/draft-07/schema#"},"name":"read_wiki_structure"}],"server_label":"deepwiki","status":"completed"}],"first_id":"mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd","last_id":"mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd","has_more":false}

Retrive a conversation item

curl http://localhost:30000/v1/conversations/conv_bca3ba2e3cad819576b58e09f898114f9ffac202b2b3993c/items/mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd
{"id":"mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd","type":"mcp_list_tools","tools":[{"annotations":{"read_only":false},"description":"Get a list of documentation topics for a GitHub repository","input_schema":{"type":"object","properties":{"repoName":{"type":"string","description":"GitHub repository"}},"required":["repoName"],"additionalProperties":false,"$schema":"http://json-schema.org/draft-07/schema#"},"name":"read_wiki_structure"}],"server_label":"deepwiki","status":"completed"}

Delete a conversation item

curl -X DELETE http://localhost:30000/v1/conversations/conv_bca3ba2e3cad819576b58e09f898114f9ffac202b2b3993c/items/mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd
{"id":"conv_bca3ba2e3cad819576b58e09f898114f9ffac202b2b3993c","object":"conversation","created_at":1760039180,"metadata":{"topic":"test"}}

// this conversation can not retrieve this item any more
curl http://localhost:30000/v1/conversations/conv_bca3ba2e3cad819576b58e09f898114f9ffac202b2b3993c/items/mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd
{"error":"Item not found in this conversation"}

// another conversation can still retrieve
curl http://localhost:30000/v1/conversations/conv_fb1779083c60692b0642e0d1dee5504dfe29b6c4c07f02ad/items/mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd
{"id":"mcpl_03d5467b35bf0c830068e7f0223a0881968c2771ea1aedsfsfd","type":"mcp_list_tools","tools":[{"annotations":{"read_only":false},"description":"Get a list of documentation topics for a GitHub repository","input_schema":{"type":"object","properties":{"repoName":{"type":"string","description":"GitHub repository"}},"required":["repoName"],"additionalProperties":false,"$schema":"http://json-schema.org/draft-07/schema#"},"name":"read_wiki_structure"}],"server_label":"deepwiki","status":"completed"}

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

key4ng · 2025-10-09T06:26:43Z

/gemini review

gemini-code-assist · 2025-10-09T06:26:46Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

… improve logic to create item

key4ng · 2025-10-09T19:59:46Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces significant new functionality for creating, retrieving, and deleting conversation items, along with the necessary data storage and routing logic. The changes are extensive and well-structured, particularly the logic for handling item creation with or without provided IDs and the ability to reference existing items. I've identified a few areas for improvement, mainly related to performance in the in-memory store, code maintainability in the main conversation handler, and a small bug in one of the new tests. My comments provide specific suggestions to address these points.

sgl-router/tests/responses_api_test.rs

sgl-router/src/data_connector/conversation_item_memory_store.rs

sgl-router/src/routers/openai/conversations.rs

… conv_item_crd

key4ng · 2025-10-09T21:33:11Z

Hi @slin1237 this pr is ready for review.
following items I want to do:

robust integration tests that I can confidently rely on without manually e2e tests
centralized place to handle the response api related spec validation

key4ng added 6 commits October 9, 2025 05:34

initial implementation

318f0de

update router_manager

b82e658

fix item reference

494f635

fix retrieve mcp message

11198fa

Check if link exists before retrieve

e03c90b

fix format

093a1f7

sglang-bot added the run-ci label Oct 9, 2025

key4ng added 4 commits October 9, 2025 18:25

store items when create response even with no conversation specified;…

8d16b08

… improve logic to create item

clean code

da890b2

standardize id character length

94052f8

remove debug log

b9d1f7e

gemini-code-assist bot reviewed Oct 9, 2025

View reviewed changes

resolve comments

0ab4d55

key4ng marked this pull request as ready for review October 9, 2025 20:51

key4ng requested review from ByronHsu, CatherineSue and slin1237 as code owners October 9, 2025 20:51

key4ng changed the title ~~conversation item crd[wip]~~ [router] conversation item API: create, retrieve and delete Oct 9, 2025

key4ng added 3 commits October 9, 2025 13:52

Merge branch 'main' into conv_item_crd

a7ac377

fix clippy

94c4ca0

Merge branch 'conv_item_crd' of https://github.com/key4ng/sglang into…

46874e8

… conv_item_crd

slin1237 approved these changes Oct 9, 2025

View reviewed changes

slin1237 merged commit eb7d926 into sgl-project:main Oct 9, 2025
35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[router] conversation item API: create, retrieve and delete #11369

[router] conversation item API: create, retrieve and delete #11369

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Motivation

Modifications

Accuracy Tests

create item using item reference

create item which has record in db

Create item that not in DB

Retrive a conversation item

Delete a conversation item

Benchmarking and Profiling

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Development

Uh oh!