8000 Support SPM infill by CISC · Pull Request #1492 · abetlen/llama-cpp-python · GitHub
[go: up one dir, main page]
Skip to content

Support SPM infill #1492

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
Jun 13, 2024
Merged

Support SPM infill #1492

merged 16 commits into from
Jun 13, 2024

Conversation

CISC
Copy link
Contributor
@CISC CISC commented May 29, 2024 •
edited
Loading

Add spm_infill option to perform infill in the Suffix/Prefix/Middle pattern instead of Prefix/Suffix/Middle as some models (like the new Codestral) prefer this.

Also added a tokenizer hack to remove leading space in suffix to improve inference, tested on several different tokenizers/vocabs with success.

@CISC
Copy link
Contributor Author
CISC commented May 29, 2024

I also see that mistral-inference adds BOS at the beginning of infill, so I guess we should too, but this requires the add_bos_token function to check if BOS should be added or not. I added this is in #1439 so if that gets merged first we can use that in this PR.

@CISC
Copy link
Contributor Author
CISC commented May 29, 2024

Actually, since I ended up not using them there let's make it simple and remove them from that PR and add them here.

@CISC
Copy link
Contributor Author
CISC commented May 30, 2024

My Codestral GGUFs are up for those who wish to test with this option.

NOTE: It requires ggml-org/llama.cpp#7644 to not insert garbage middle token!

@abetlen
Copy link
Owner
abetlen commented Jun 4, 2024

@CISC looks good overall, do you mind adding something minimal to /examples that would help me review this and demonstrat the FIM API?

@CISC
Copy link
Contributor Author
CISC commented Jun 4, 2024

Sure, no problem.

@CISC
Copy link
Contributor Author
CISC commented Jun 4, 2024

@abetlen Example added.

@abetlen abetlen closed this Jun 13, 2024
@abetlen abetlen deleted the spm_infill branch June 13, 2024 07:38
@abetlen abetlen restored the spm_infill branch June 13, 2024 07:38
@abetlen abetlen reopened this Jun 13, 2024
@abetlen abetlen merged commit dbcf64c into abetlen:main Jun 13, 2024
16 checks passed
@CISC CISC deleted the spm_infill branch June 13, 2024 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@CISC @abetlen
0