8000 Migrate inference to llama_batch and llama_decode api (#795) · LOGp/llama-cpp-python@ab028cb · GitHub
[go: up one dir, main page]
Skip to content

Commit ab028cb

Browse files
abetlenantoine-lizee
abetlen
and
antoine-lizee
authored
Migrate inference to llama_batch and llama_decode api (abetlen#795)
* Add low-level batching notebook * fix: tokenization of special characters: (abetlen#850) It should behave like llama.cpp, where most out of the box usages treat special characters accordingly * Update CHANGELOG * Cleanup * Fix runner label * Update notebook * Use llama_decode and batch api * Support logits_all parameter --------- Co-authored-by: Antoine Lizee <antoine.lizee@gmail.com>
1 parent f436e0c commit ab028cb

File tree

3 files changed

+753
-8
lines changed

3 files changed

+753
-8
lines changed

0 commit comments

Comments
 (0)
0