10000 Integrate Functionary v2.5 + Refactor Functionary Code by jeffrey-fong · Pull Request #1509 · abetlen/llama-cpp-python · GitHub
[go: up one dir, main page]
Skip to content

Integrate Functionary v2.5 + Refactor Functionary Code #1509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Integrate Functionary v2.5 + Refactor Functionary Code #1509

Show file tree
Hide file tree
Changes fr 8000 om all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b342f39
set up abstract classes for non-streaming
jeffrey-fong Jun 5, 2024
115079b
set up abstract classes for streaming
jeffrey-fong Jun 5, 2024
4672f2a
Merge branch 'abetlen:main' into functionary-v2.5
jeffrey-fong Jun 5, 2024
7a3af05
update README
jeffrey-fong Jun 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -486,20 +486,20 @@ llm.create_chat_completion(
```

<details>
<summary>Functionary v2</summary>
<summary>Functionary</summary>

The various gguf-converted files for this set of models can be found [here](https://huggingface.co/meetkai). Functionary is able to intelligently call functions and also analyze any provided function outputs to generate coherent responses. All v2 models of functionary supports **parallel function calling**. You can provide either `functionary-v1` or `functionary-v2` for the `chat_format` when initializing the Llama class.
The various gguf-converted files for this set of models can be found [here](https://huggingface.co/meetkai). Functionary is able to intelligently call functions and also analyze any provided function outputs to generate coherent responses. All v2 models of functionary supports **parallel function calling**. You can provide `functionary` for the `chat_format` when initializing the Llama class.

Due to discrepancies between llama.cpp and HuggingFace's tokenizers, it is required to provide HF Tokenizer for functionary. The `LlamaHFTokenizer` class can be initialized and passed into the Llama class. This will override the default llama.cpp tokenizer used in Llama class. The tokenizer files are already included in the respective HF repositories hosting the gguf files.

```python
from llama_cpp import Llama
from llama_cpp.llama_tokenizer import LlamaHFTokenizer
llm = Llama.from_pretrained(
repo_id="meetkai/functionary-small-v2.2-GGUF",
filename="functionary-small-v2.2.q4_0.gguf",
chat_format="functionary-v2",
tokenizer=LlamaHFTokenizer.from_pretrained("meetkai/functionary-small-v2.2-GGUF")
repo_id="meetkai/functionary-small-v2.5-GGUF",
filename="functionary-small-v2.5.Q4_0.gguf",
chat_format="functionary",
tokenizer=LlamaHFTokenizer.from_pretrained("meetkai/functionary-small-v2.5-GGUF")
)
```

Expand Down
4 changes: 2 additions & 2 deletions docs/server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,12 +78,12 @@ You'll first need to download one of the available function calling models in GG

- [functionary](https://huggingface.co/meetkai)

Then when you run the server you'll need to also specify either `functionary-v1` or `functionary-v2` chat_format.
Then when you run the server you'll need to also specify `functionary` chat_format.

Note that since functionary requires a HF Tokenizer due to discrepancies between llama.cpp and HuggingFace's tokenizers as mentioned [here](https://github.com/abetlen/llama-cpp-python/blob/main?tab=readme-ov-file#function-calling), you will need to pass in the path to the tokenizer too. The tokenizer files are already included in the respective HF repositories hosting the gguf files.

```bash
python3 -m llama_cpp.server --model <model_path_to_functionary_v2_model> --chat_format functionary-v2 --hf_pretrained_model_name_or_path <model_path_to_functionary_v2_tokenizer>
python3 -m llama_cpp.server --model <model_path_to_functionary_model> --chat_format functionary --hf_pretrained_model_name_or_path <model_path_to_functionary_tokenizer>
```

Check out this [example notebook](https://github.com/abetlen/llama-cpp-python/blob/main/examples/notebooks/Functions.ipynb) for a walkthrough of some interesting use cases for function calling.
Expand Down
Loading
0