10000 Add json schema mode by abetlen · Pull Request #1122 · abetlen/llama-cpp-python · GitHub
[go: up one dir, main page]
Skip to content

Add json schema mode #1122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jan 27, 2024
Merged

Add json schema mode #1122

merged 3 commits into from
Jan 27, 2024

Conversation

abetlen
Copy link
Owner
@abetlen abetlen commented Jan 23, 2024 •
edited
Loading

Adds Anyscale-style json schema mode for all simple chat formats (ie chatml, llama2, alpaca, vicuna, etc) as well as llava multimodal models. Source https://docs.endpoints.anyscale.com/guides/json_mode/

Server Usage

This is using the python openai API but any REST client will work.

import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1", api_key="esecret_YOUR_API_KEY"
)

# Note: not all arguments are currently supported and will be ignored by the backend.
chat_completion = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that outputs in JSON.",
        },
        {"role": "user", "content": "Who won the world series in 2020"},
    ],
    response_format={
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {"team_name": {"type": "string"}},
            "required": ["team_name"],
        },
    },
    temperature=0.7,
)
print(chat_completion.model_dump())
{'id': 'chatcmpl-60994326-99f9-41f6-8fc0-ae9b08026521', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '{ "team_name
8000
": "Los Angeles Dodgers" } ', 'role': 'assistant', 'function_call': None, 'tool_calls': None}}], 'created': 1706391561, 'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1', 'object': 'chat.completion', 'system_fingerprint': None, 'usage': {'completion_tokens': 16, 'prompt_tokens': 65, 'total_tokens': 81}}

Python Usage

>>> from llama_cpp import Llama
>>> llm = Llama(model_path="path/to/model.gguf", chat_format="chatml")
>>> llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that outputs in JSON.",
        },
        {"role": "user", "content": "Who won the world series in 2020"},
    ],
    response_format={
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {"team_name": {"type": "string"}},
            "required": ["team_name"],
        },
    },
    temperature=0.7,
)
{'id': 'chatcmpl-60994326-99f9-41f6-8fc0-ae9b08026521', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '{ "team_name": "Los Angeles Dodgers" } ', 'role': 'assistant', 'function_call': None, 'tool_calls': None}}], 'created': 1706391561, 'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1', 'object': 'chat.completion', 'system_fingerprint': None, 'usage': {'completion_tokens': 16, 'prompt_tokens': 65, 'total_tokens': 81}}

@abetlen abetlen merged commit d8f6914 into main Jan 27, 2024
@abetlen abetlen deleted the add-json-schema-mode branch January 31, 2024 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@abetlen
0