8000 Implement GGUF metadata KV overrides by phiharri · Pull Request #1011 · abetlen/llama-cpp-python · GitHub
[go: up one dir, main page]
Skip to content

Implement GGUF metadata KV overrides #1011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Jan 15, 2024
Merged

Implement GGUF metadata KV overrides #1011

merged 9 commits into from
Jan 15, 2024

Conversation

phiharri
Copy link
Contributor
@phiharri phiharri commented Dec 14, 2023 •
edited by abetlen
Loading

Quick implementation of KV overrides which accepts strings in llama.cpp --kv_overrides KEY=TYPE:VALUE .. format. For example llama.expert_used_count=int:3 Multiple overrides may be space separated.

Not that familiar with ctypes or the preferred types to use here.

Closes #1084

@phiharri phiharri mentioned this pull request Dec 14, 2023
Closed
abetlen added a commit that referenced this pull request Dec 22, 2023
@abetlen
fix: inccorect bindings for kv override. Based on #1011
6d8bc09
@abetlen
Copy link
Owner
abetlen commented Jan 14, 2024

Hey @phiharri thanks for the contribution, I have a few changes to make but the overall api seems correct. I think the kv overrides argument to the Llama class should just be a dictionary and the types should be based on the python types 8000 but the string based approach for the cli args looks good.

@abetlen
Copy link
Owner
abetlen commented Jan 15, 2024

Looks good now, @phiharri thank you for the contribution!

@abetlen abetlen merged commit 76aafa6 into abetlen:main Jan 15, 2024
abetlen pushed a commit that referenced this pull request Jan 24, 2024
@phiharri
fix: GGUF metadata KV overrides, re #1011 (#1116)
fe5d6ea 
* kv overrides another attempt

* add sentinel element, simplify array population

* ensure sentinel element is zeroed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add KV overrides support to Llama class and server
2 participants
@phiharri @abetlen
0