Closed
Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new bug or useful enhancement to share.
Expected Behavior
I hope to fix this bug as soon as possible.
Current Behavior
In #1311, the logprobs field was not handled correctly.
Environment and Context
- Python 3.11.8
- llama-cpp-python[all]==v0.2.59
- GNU Make 4.4.1
- g++ (GCC) 13.2.1 20230801
Failure Information (for bugs)
In #1311, the logprobs field was introduced. Due to the lack of logprobs returned when using function calls, it resulted in the malfunction of function calls.
Steps to Reproduce
- Install llama-cpp-python[all]==v0.2.57
- edit config.json and run server
python3 -m llama_cpp.server --config_file config.json{ "model": "models/Qwen1.5/qwen1_5-4b-chat-q4_k_m.gguf", "model_alias": "qwen1_5-4b-chat-q4_k_m", "chat_format": "chatml-function-calling", "n_gpu_layers": -1, "offload_kqv": true, "n_threads": 12, "n_batch": 512, "n_ctx": 2048 }, - Run the first function call example in this notebook.
- The execution is normal.
- Install llama-cpp-python[all]==v0.2.59
- Run the first function call example in this notebook.
- Internal Server Error
Exception: 2 validation errors:
{'type': 'missing', 'loc': ('response', 'typed-dict', 'choices', 0, 'logprobs'), 'msg': 'Field required', 'input': {'finish_reason': 'tool_calls', 'index': 0, 'message': {'role': 'assistant', 'content': None, 'tool_calls': [{'id': 'call__0_get_current_weather_cmpl-58520529-a626-4a1e-8b4b-1fca9dd2d68a', 'type': 'function', 'function': {'name': 'get_current_weather', 'arguments': '{ "location": "San Francisco, Tokyo, Paris" , "unit": "fahrenheit"}'}}], 'function_call': {'name': 'get_current_weather:', 'arguments': '{ "location": "San Francisco, Tokyo, Paris" , "unit": "fahrenheit"}'}}}, 'url': 'https://errors.pydantic.dev/2.6/v/missing'}
{'type': 'string_type', 'loc': ('response', 'str'), 'msg': 'Input should be a valid string', 'input': {'id': 'chatcmpl-5ce5ae67-c028-427f-a8bb-fe3ff94eb934', 'object': 'chat.completion', 'created': 1712263433, 'model': 'qwen1_5-4b-chat-q4_k_m', 'choices': [{'finish_reason': 'tool_calls', 'index': 0, 'message': {'role': 'assistant', 'content': None, 'tool_calls': [{'id': 'call__0_get_current_weather_cmpl-58520529-a626-4a1e-8b4b-1fca9dd2d68a', 'type': 'function', 'function': {'name': 'get_current_weather', 'arguments': '{ "location": "San Francisco, Tokyo, Paris" , "unit": "fahrenheit"}'}}], 'function_call': {'name': 'get_current_weather:', 'arguments': '{ "location": "San Francisco, Tokyo, Paris" , "unit": "fahrenheit"}'}}}], 'usage': {'completion_tokens': 22, 'prompt_tokens': 31, 'total_tokens': 53}}, 'url': 'https://errors.pydantic.dev/2.6/v/string_type'}
Traceback (most recent call last):
File "/home/devcxl/download/llama-server/.evm/lib/python3.11/site-packages/llama_cpp/server/errors.py", line 171, in custom_route_handler
response = await original_route_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/devcxl/download/llama-server/.evm/lib/python3.11/site-packages/fastapi/routing.py", line 296, in app
content = await serialize_response(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/devcxl/download/llama-server/.evm/lib/python3.11/site-packages/fastapi/routing.py", line 155, in serialize_response
raise ResponseValidationError(
fastapi.exceptions.ResponseValidationError: 2 validation errors:
{'type': 'missing', 'loc': ('response', 'typed-dict', 'choices', 0, 'logprobs'), 'msg': 'Field required', 'input': {'finish_reason': 'tool_calls', 'index': 0, 'message': {'role': 'assistant', 'content': None, 'tool_calls': [{'id': 'call__0_get_current_weather_cmpl-58520529-a626-4a1e-8b4b-1fca9dd2d68a', 'type': 'function', 'function': {'name': 'get_current_weather', 'arguments': '{ "location": "San Francisco, Tokyo, Paris" , "unit": "fahrenheit"}'}}], 'function_call': {'name': 'get_current_weather:', 'arguments': '{ "location": "San Francisco, Tokyo, Paris" , "unit": "fahrenheit"}'}}}, 'url': 'https://errors.pydantic.dev/2.6/v/missing'}
{'type': 'string_type', 'loc': ('response', 'str'), 'msg': 'Input should be a valid string', 'input': {'id': 'chatcmpl-5ce5ae67-c028-427f-a8bb-fe3ff94eb934', 'object': 'chat.completion', 'created': 1712263433, 'model': 'qwen1_5-4b-chat-q4_k_m', 'choices': [{'finish_reason': 'tool_calls', 'index': 0, 'message': {'role': 'assistant', 'content': None, 'tool_calls': [{'id': 'call__0_get_current_weather_cmpl-58520529-a626-4a1e-8b4b-1fca9dd2d68a', 'type': 'function', 'function': {'name': 'get_current_weather', 'arguments': '{ "location": "San Francisco, Tokyo, Paris" , "unit": "fahrenheit"}'}}], 'function_call': {'name': 'get_current_weather:', 'arguments': '{ "location": "San Francisco, Tokyo, Paris" , "unit": "fahrenheit"}'}}}], 'usage': {'completion_tokens': 22, 'prompt_tokens': 31, 'total_tokens': 53}}, 'url': 'https://errors.pydantic.dev/2.6/v/string_type'}
INFO: 127.0.0.1:36362 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error